Hello everyone, I think in Airflow 2 it's hard to find what kind of options you have for logs, secrets, authentication.
For all those, we do not have a single place where people could gather all the options and where people can find them easily. I think complete separation out of particular logs, secrets to providers and defferring most authentication information to FAB (without being explicit about it) made it quite difficult to find how to do it all. I've learned that people are finding better answers in Stack Overflow questions than in our documentation and this is not a good sign. Example slack discussion for LDAP authentication https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1628782373128600 which ended up with recommendation to https://stackoverflow.com/questions/65946118/how-to-setup-ldap-authentication-in-airflow-2-0 (but there were many more of those) Also it's not really easy to figure out configuring Oauth - I struggled with it myself few weeks ago and figured it out eventually, but still not fully (role mapping is not explained anywhere for one - at least not that I am aware of). When I searched for anything useful, more often than not I was redirected to Airflow 1.10 documentation that had some kind of overview of possible options. But it's not really existing in Airflow. I cannot find pages summarising the available (Airflow community managed) deployment options such as remote log configuration. Some details (but without comprehensive examples) are available in providers (which is a good place for example the remote loggers belong to their providers), but you need to know you should look for them there - and it is not at all obvious. I think it would be great to extend our documentation pages with: * Available Authentication options with examples (even if most of the documentation is in FAB, it is rather difficult to map Airflow configuration into FAB one - at least it's not obvious which part is FAB, which part is Airflow, where to put what, how to configure roles etc.): LDAP, Oauth, Google, ....... * List (and link) available logging options at https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html?highlight=remote%20log#advanced-configuration .You will not find list of implemented integrations in this page - you should look for details of advanced logging in providers (but it's not at all obvious where and that they exist at all). There are no links to S3/GCS logging configuration/handling and it's not easy to find out where you should look for them. Better examples would also be useful. * Secret Backends page is a bit better - https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html. At least it mentions GCP/Hashicorp as "examples" but it misses AWS one and when you go to "Supported Backends" you see only the "Local Filesystem"one. I think it is really misleading that you do not have a full list of secret backends in the community-managed providers. I think in all those cases there was a good intention - not to repeat what is written somewhere else and not to add "all community providers specific implementation", because they are separated out and we should treat the providers as "separate" from core and put documentation there. However I think the usefulness of the documentation suffered by this change- some level of redundancy and making useful information in the place where people are looking for it rather than somewhere else where it is "the single source of truth" is not the best idea for the useful documentation. While this is OK for "operators/sensors" etc. as this is pretty "obvious" they are in "providers", when it comes to common features such as authentication, logging, secrets, I think having single page with comprehensive overview what is available for those is rather useful I'd love to hear other's opinion -maybe it's just me, but I would prefer to see in one place all the options I have with authentication, logging configuration, secrets - at least nicely indexed with comprehensive list of options I have for the community and links to the exact places where more details are provided. Let me know what you think. J. -- +48 660 796 129
