Precisely my thinking - it's not really "we do not have the information" but "It's hard to discover that we have it".
On Fri, Aug 13, 2021 at 10:59 AM Javier Lopez <[email protected]> wrote: > Hello, > > I agree with you. I think if you are new to airflow, you won't find those > options for the providers unless you specifically search for them, which is > unlikely because you may not even know that they exist. > > On Fri, 13 Aug 2021 at 03:48, Jarek Potiuk <[email protected]> wrote: > >> Hello everyone, >> >> I think in Airflow 2 it's hard to find what kind of options you have for >> logs, secrets, authentication. >> >> For all those, we do not have a single place where people could gather >> all the options and where people can find them easily. I think complete >> separation out of particular logs, secrets to providers and defferring most >> authentication information to FAB (without being explicit about it) made it >> quite difficult to find how to do it all. >> >> I've learned that people are finding better answers in Stack Overflow >> questions than in our documentation and this is not a good sign. Example >> slack discussion for LDAP authentication >> https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1628782373128600 >> which ended up with recommendation to >> https://stackoverflow.com/questions/65946118/how-to-setup-ldap-authentication-in-airflow-2-0 >> (but there were many more of those) >> >> Also it's not really easy to figure out configuring Oauth - I struggled >> with it myself few weeks ago and figured it out eventually, but still not >> fully (role mapping is not explained anywhere for one - at least not that I >> am aware of). >> >> When I searched for anything useful, more often than not I was redirected >> to Airflow 1.10 documentation that had some kind of overview of possible >> options. But it's not really existing in Airflow. >> >> I cannot find pages summarising the available (Airflow community managed) >> deployment options such as remote log configuration. Some details (but >> without comprehensive examples) are available in providers (which is a good >> place for example the remote loggers belong to their providers), but you >> need to know you should look for them there - and it is not at all obvious. >> >> I think it would be great to extend our documentation pages with: >> >> * Available Authentication options with examples (even if most of the >> documentation is in FAB, it is rather difficult to map Airflow >> configuration into FAB one - at least it's not obvious which part is FAB, >> which part is Airflow, where to put what, how to configure roles etc.): >> LDAP, Oauth, Google, ....... >> >> * List (and link) available logging options at >> https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html?highlight=remote%20log#advanced-configuration >> .You will not find list of implemented integrations in this page - you >> should look for details of advanced logging in providers (but it's not at >> all obvious where and that they exist at all). There are no links to S3/GCS >> logging configuration/handling and it's not easy to find out where you >> should look for them. Better examples would also be useful. >> >> * Secret Backends page is a bit better - >> https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html. >> At least it mentions GCP/Hashicorp as "examples" but it misses AWS one and >> when you go to "Supported Backends" you see only the "Local Filesystem"one. >> I think it is really misleading that you do not have a full list of secret >> backends in the community-managed providers. >> >> I think in all those cases there was a good intention - not to repeat >> what is written somewhere else and not to add "all community providers >> specific implementation", because they are separated out and we should >> treat the providers as "separate" from core and put documentation there. >> >> However I think the usefulness of the documentation suffered by this >> change- some level of redundancy and making useful information in the place >> where people are looking for it rather than somewhere else where it is "the >> single source of truth" is not the best idea for the useful documentation. >> While this is OK for "operators/sensors" etc. as this is pretty "obvious" >> they are in "providers", when it comes to common features such as >> authentication, logging, secrets, I think having single page with >> comprehensive overview what is available for those is rather useful >> >> I'd love to hear other's opinion -maybe it's just me, but I would prefer >> to see in one place all the options I have with authentication, logging >> configuration, secrets - at least nicely indexed with comprehensive list of >> options I have for the community and links to the exact places where more >> details are provided. >> >> Let me know what you think. >> >> J. >> >> -- >> +48 660 796 129 >> > -- +48 660 796 129
