Thanks A LOT Jarek for this :) I am extremely pleased to see other auth
managers coming from the community. This is definitely the end result we
were aiming for ... 2 years ago. It was a long road but I agree, I really
like the end result. I would not have made it without you so thank you as
well for having enabled this :)
As you mentioned, that'd be great, after some time, to incorporate this
LDAP auth manager into LDAP provider so that it makes it easier for users
to use/install it.
This is also a very good example that implementing an auth manager is not
so hard :) so if you would like to implement an auth manager based on your
favorite authN/authZ tool, please do so and I would love to help you out if
needed. Who knows, maybe it will so cool it will be the next default auth
manager in Airflow when Fab auth manager will be gone in the future?
On 2025/12/03 11:20:24 Jarek Potiuk wrote:
Hello here,
New day - new prefix of the email.
I wanted to actually BOAST about the Extensible User Management AIP-56 (
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
- we completed the AIP a long time ago, but I think the actual
completion
of the work is really now.
Big shoutout to Vincent leading it and everyone else who helped with it.
The soundness of the approach we took has just been finally confirmed by
a
3rd-party implementation of LDAP Auth Manager - by Emre Can (@emredjan) -
https://github.com/emredjan/airflow-ldap-auth-manager just published and
made available at our Ecosystem page.
I find it super exciting, and I feel somewhat proud.y f - mostl,
personally
even if I was just mostly gently pulling and pushing things here and
there
and came up with the initial direction. Not everyone followed it, but I
wanted to share the small success story of some of the more "hairy"
things
we have done. Or rather mostly Vincent did mostly with a lot of people
helping.
I think that one is a really a good example to follow of a long-term
feature that was done the right way with a big-hairy long term goal that
has been relentlessly followed - that was done really in parallel of
Airflow 3 work and had started a long time ago (2 years almost to the
date).
We are doing it now with Amogh leadership now started by Ash - with
isolation of Task.sdk - even more "hairy" thing - and even more
impactful.
And I hope we can take it as a good example for things like state
management - where we should take time and deliberate effort to make some
longer-term vision and decisions and relentlessly follow it to
completion.
The story is pretty long, but I wanted to describe it here - even if not
for you to read today, but to capture some of the things that made it
happen, so that we can refer to it in the future. I thought about the day
that I will write the email for a looong time. And I knew what I wanted
to
write - and the day finally came.
Brace yourselves, those how will venture into reading it.
----
It's a long journey since we had just mostly *whined* about FAB being
such
a pain. In Airflow 1 FAB RBAC was already a step-up compared to the
earlier
approach (everyone can do everything approach) - yes that was a default I
still remember from Airflow 1.
This one somehow disappears in memories even of those who were here 6
years
ago. But it was a great improvement back then to have RBAC and user
management. And one could think that we could only get it even more
feature-full as something that Airflow has "built-in" - but it turned out
to be a big dependency hog, FAB for a long time was not really evolving
as
fast as Airflow was with its enterprise integration needs. Groups were
only
recently added to FAB (and we are not even using groups even if we
switched
to FAB that supports it). It was a pain for people to configure it, they
had to switch between Airflow and FAB docs, it was confusing whether we
have Airflow or FAB issue, we also at some point of time vendored-in
Security Manager partially in Airflow (and since then we need to manually
synchronise changes implemented in Fab to the manager we have in
Airflow).
A lot of security issues came through FA and often we had to wait for FAB
or even contribute back the fixes. We worked closely with Daniel, that is
great cooperation, we even had calls when some security issues were
raised
that affected FAB, Airflow and Superset and we discussed our responses
together and synced the remediations. But still - this was a lot, a lot,
a
lot of pain, and we had this recurring whining *"Oh we wish we did not
have
to depend on FAB".*
When we initially discussed it with Vincent we had many back-forth on how
to design the Auth Manager. We all came up with the idea to do it
differently - and rather than complicate the way how RBAC is done with
groups or other features, we decided that it's not really an Airflow job
to
do Authentication and Authorisation management - and to delegate it to
those who do it better.
We made some choices that I think stand the pass of time with simple, yet
powerful API design where instead of thinking what "more" we can do in
Airflow, is what we can do to make Airflow more of an "extensible
platform"
- that will be super-easy to extend. We believe that people will come and
do their own Auth Manager implementation some day. This day happened.
Then AWS team with AWS Auth Manager - mostly Vincent, Nick and AWS team
dogfooded and polished the rough edges in - initially experimental AWS
Auth
Manager. That was still way back in Airflow 2. Great POC of the idea. One
that was absolutely necessary to move forward.
At the same time we (Vincent again!) implemented FAB Auth Manager
interface
and moved all functionality to the provider and step-by-step we moved out
all FAB-y things to the provider, piece-by-piece, release by release.
Keeping the long-term vision in mind and getting ever closer to it. That
was **not an easy task** and it had MANY bumps, but with Vinc
relentlessly
leading it and a number of people helping, (I had a very little part in
it
- mostly lurking from behind Vincent's back - but I was in fact following
very closely, ready to step-in and help any time with some difficult
choices and decisions. There were many things around with dependencies,
issues, back-compatibility, etc. etc. And the work continues there still
to
complete some issues.
In the meantime of course Airflow 3 happened with all the FastApi
conversion (which turns out to be a great decision) and new UI where we
got
rid of the Fab ties from the UI part - Widgets, Connections, Plugins some
internal tooling we got this out as huge part of Airflow 3 migration.
Jens,
Pierre. Karthikeyan and many others did that. Then we got
SimpleAuthManager
as default - which is "good" as demo, development and shows the basic
capabilities of Auth Managers - with different types of (hard-coded)
users
- which shows what Auth Managers can do, without even pretending, it can
be
used in production (actually even actively shouting you in the face it
should not - stll some people wants to do it of course).
Eventually - we are extremely closely to cut the final last ties of FAB
and
make the provider truly, truly optional and with its dependencies not
holding us back - there are literally few more APIs to convert to Fast
API
for Fab Auth management left (and Yun-Ting Chiu - @chiuinggum has done
and
continues doing a great job there). Some last remaining bridge between UI
and API for custom - old AuthBackends from Airflow 2 is to be added. Once
we do it, the day where we whine "Oh I wish we did not have to rely on
FAB"
will be finally gone. It was great choice when we started, but it turned
out too much of a "gravity center" with a number of choices that held us
back,
Then KeyCloak implementation that was really "far fetched idea" - (also
Vincent's doing) - started. Tt is really nice and simple and extremely
powerful. Not everyone realizes but thanks to the way we designed Auth
Manager, the enterprise users of our can do a LOOOOOOT more than the
"constrained" FAB RBAC allows. You can configure all the complexity that
KeyCloak provides - mix and match authentication policies and rules - and
you are only limited to what KeyCloak provides. One of the examples is
that
for example y)ou could configure that a given Dag or Dag group can be
triggered by a group of users between 9am and 5pm only - and at other
times
it will be rejected. And generally any complexity of deciding what users
should be capable of can be implemented in a simple way by defining and
combining javascript policies of KeyCloak. I really like how KeyCloak
does
one thing BEST -> ID management integrated with enterprise ID systems.
And
we now have a really nice (and simple, yet powerful) provider which
people
can use. If you already used Keycloak - you could integrate it via FAB,
but
with KeyCloak provider, it's so much more powerful having native
integration.
We already managed to implement nice batch optimizations and evolved tha
Auth Manager API. We can have a single call for all Dags you are
attempting to display in dag view (avoiding n+1 issue) and see only those
that you can have access to. We have that in KeyCloak (or maybe is about
to
be completed - but we know how to do it). And any implementation of Auth
Manager can use it as well. And we are discussing some discovery of
permission API that will allow for example to know if you should gray out
the "Trigger Run" button before you attempt it in the new UI if you do
not
have permissions to get better UX.
For many of our users KeyCloak is too heavy to rely on. What if you have
LDAP and you don't care about all the features of KeyCloak and do not
want
to bank on it? Fret not - as of a few days ago there is this 3rd-party
LDAP
provider - so you can use it instead, without having to run and install
the
KeyCloak for your company. The day might come if Emre would like that and
it proves to be useful we might get it contributed and we might be even
super happy to take maintenance of it - following our - just voted - new
provider acceptance policy. And it's the first fully reusable, generic
Auth
Manager by "someone else" that is functional and published.
And ... you can also roll your own - and Emrecan has promised to review
and
update our docs to make it even easier to do your own Auth Manager, so I
hope we will soon have some other "simpler-purpose" Auth Managers.
Fingers crossed.
That was a journey I wanted to share.
Vinc, you rock
J.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
