> I am concerned about adding information about the content of provider packages in the core documentation as it is very easy to get obsolete
I agree we should not put any provider details in "core" . But we should at the very least (I think) put links to all the "community" providers that implement certain features. This is really a "discoverability" problem, nothing more. I think we - long term committers who know all about airflow, providers, etc. are overestimating user's knowledge about airflow internals - and the documentation should be there to guide them to learn. There was this - very relevant - comic from XKCD day before yesterday https://xkcd.com/2501/ that shows the mechanism very well. I tried to put myself in the shoes of a new user. Try to do it Kamil as well. When you look at the "logging" or "secrets" section, you are completely unaware that you can get AWS, GCP and other integrations provided by the community. And there is NOTHING to tell you otherwise. You need to know that you should start looking elsewhere - and I want to help the people who are looking at the page to give the links where they can find itt. Essentially when you do not airflow, do not realise that there are providers, and do not realise that those providers implement those issues, You leave with the impression that a lot of stuff is missing. With the current documentation structure, I am afraid People simply do not even know that there are community-managed implementations out there. What I am thinking really is to this kind of formula (It shows how secrets should look like but it should be applied across the board in similar cases): ############################################ Secret Backends Page: Paragraph Describe Secret backends in general # Available Secret backends ## Core Airflow Secret backends: * <File backend> - link pointing to it ## Backends Provided by community-managed providers: * <VaultBackend> * <AWSSecretBackend> * <KMSBackend> ########################################## I think about just links to the appropriate documentation available in providers. No more, no less. This could be applied (automatically) to all functionalities provided by providers. I think this is safe, can be automated and solves the discoverability problem. It does not require extra maintenance. J. On Sat, Aug 14, 2021 at 6:40 PM .... <[email protected]> wrote: > Commented above > > pt., 13 sie 2021 o 03:48 Jarek Potiuk <[email protected]> napisał(a): > > > > > * List (and link) available logging options at > https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html?highlight=remote%20log#advanced-configuration > .You will not find list of implemented integrations in this page - you > should look for details of advanced logging in providers (but it's not at > all obvious where and that they exist at all). There are no links to S3/GCS > logging configuration/handling and it's not easy to find out where you > should look for them. Better examples would also be useful. > > > > * Secret Backends page is a bit better - > https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html. > At least it mentions GCP/Hashicorp as "examples" but it misses AWS one and > when you go to "Supported Backends" you see only the "Local Filesystem"one. > I think it is really misleading that you do not have a full list of secret > backends in the community-managed providers. > > > > I am concerned about adding information about the content of provider > packages in the core documentation as it is very easy to get obsolete > as Airflow and the packages have a different release cycle and the new > packages are compatible with the old Airflow versions so it may not be > obvious that you should be looking at the latest documentation for > Airflow to know the full list of providers even if you are using a > non-latest version of Airflow. > > I think it's worth taking an approach similar to operators, where the > core documentation does not contain the full list of operators from > the provider packages, but only contains a list of operators in the > core, and includes references to the documentation for providers that > includes this list of operators in provider packages. > Here is a reference of all core operators: > > https://airflow.apache.org/docs/apache-airflow/stable/operators-and-hooks-ref.html > Here is a reference of all operators in providers packages: > > https://airflow.apache.org/docs/apache-airflow-providers/operators-and-hooks-ref/index.html > > The list of operators in the providers' package is automatically > generated on the basis of provider.yaml files and the correctness of > the file are automatically verified, so we can be sure that the > reference is up-to-date and complete. This also reduces the > maintenance burden of this documentation. > > Adding the backend and task handler secret to providers.yaml also > means that information about them will be available on the main page > of the project in the "Integrations" section. > -- +48 660 796 129
