Hello, I agree with you. I think if you are new to airflow, you won't find those options for the providers unless you specifically search for them, which is unlikely because you may not even know that they exist.
On Fri, 13 Aug 2021 at 03:48, Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > I think in Airflow 2 it's hard to find what kind of options you have for > logs, secrets, authentication. > > For all those, we do not have a single place where people could gather all > the options and where people can find them easily. I think complete > separation out of particular logs, secrets to providers and defferring most > authentication information to FAB (without being explicit about it) made it > quite difficult to find how to do it all. > > I've learned that people are finding better answers in Stack Overflow > questions than in our documentation and this is not a good sign. Example > slack discussion for LDAP authentication > https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1628782373128600 > which ended up with recommendation to > https://stackoverflow.com/questions/65946118/how-to-setup-ldap-authentication-in-airflow-2-0 > (but there were many more of those) > > Also it's not really easy to figure out configuring Oauth - I struggled > with it myself few weeks ago and figured it out eventually, but still not > fully (role mapping is not explained anywhere for one - at least not that I > am aware of). > > When I searched for anything useful, more often than not I was redirected > to Airflow 1.10 documentation that had some kind of overview of possible > options. But it's not really existing in Airflow. > > I cannot find pages summarising the available (Airflow community managed) > deployment options such as remote log configuration. Some details (but > without comprehensive examples) are available in providers (which is a good > place for example the remote loggers belong to their providers), but you > need to know you should look for them there - and it is not at all obvious. > > I think it would be great to extend our documentation pages with: > > * Available Authentication options with examples (even if most of the > documentation is in FAB, it is rather difficult to map Airflow > configuration into FAB one - at least it's not obvious which part is FAB, > which part is Airflow, where to put what, how to configure roles etc.): > LDAP, Oauth, Google, ....... > > * List (and link) available logging options at > https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html?highlight=remote%20log#advanced-configuration > .You will not find list of implemented integrations in this page - you > should look for details of advanced logging in providers (but it's not at > all obvious where and that they exist at all). There are no links to S3/GCS > logging configuration/handling and it's not easy to find out where you > should look for them. Better examples would also be useful. > > * Secret Backends page is a bit better - > https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html. > At least it mentions GCP/Hashicorp as "examples" but it misses AWS one and > when you go to "Supported Backends" you see only the "Local Filesystem"one. > I think it is really misleading that you do not have a full list of secret > backends in the community-managed providers. > > I think in all those cases there was a good intention - not to repeat what > is written somewhere else and not to add "all community providers specific > implementation", because they are separated out and we should treat the > providers as "separate" from core and put documentation there. > > However I think the usefulness of the documentation suffered by this > change- some level of redundancy and making useful information in the place > where people are looking for it rather than somewhere else where it is "the > single source of truth" is not the best idea for the useful documentation. > While this is OK for "operators/sensors" etc. as this is pretty "obvious" > they are in "providers", when it comes to common features such as > authentication, logging, secrets, I think having single page with > comprehensive overview what is available for those is rather useful > > I'd love to hear other's opinion -maybe it's just me, but I would prefer > to see in one place all the options I have with authentication, logging > configuration, secrets - at least nicely indexed with comprehensive list of > options I have for the community and links to the exact places where more > details are provided. > > Let me know what you think. > > J. > > -- > +48 660 796 129 >
