Commented above pt., 13 sie 2021 o 03:48 Jarek Potiuk <[email protected]> napisał(a): >
> * List (and link) available logging options at > https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html?highlight=remote%20log#advanced-configuration > .You will not find list of implemented integrations in this page - you > should look for details of advanced logging in providers (but it's not at all > obvious where and that they exist at all). There are no links to S3/GCS > logging configuration/handling and it's not easy to find out where you should > look for them. Better examples would also be useful. > > * Secret Backends page is a bit better - > https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html. > At least it mentions GCP/Hashicorp as "examples" but it misses AWS one and > when you go to "Supported Backends" you see only the "Local Filesystem"one. I > think it is really misleading that you do not have a full list of secret > backends in the community-managed providers. > I am concerned about adding information about the content of provider packages in the core documentation as it is very easy to get obsolete as Airflow and the packages have a different release cycle and the new packages are compatible with the old Airflow versions so it may not be obvious that you should be looking at the latest documentation for Airflow to know the full list of providers even if you are using a non-latest version of Airflow. I think it's worth taking an approach similar to operators, where the core documentation does not contain the full list of operators from the provider packages, but only contains a list of operators in the core, and includes references to the documentation for providers that includes this list of operators in provider packages. Here is a reference of all core operators: https://airflow.apache.org/docs/apache-airflow/stable/operators-and-hooks-ref.html Here is a reference of all operators in providers packages: https://airflow.apache.org/docs/apache-airflow-providers/operators-and-hooks-ref/index.html The list of operators in the providers' package is automatically generated on the basis of provider.yaml files and the correctness of the file are automatically verified, so we can be sure that the reference is up-to-date and complete. This also reduces the maintenance burden of this documentation. Adding the backend and task handler secret to providers.yaml also means that information about them will be available on the main page of the project in the "Integrations" section.
