Hi all,

I'd like to discuss an idea for letting Airflow act as an OpenID
Connect issuer for its own tasks.
Before starting any implementation, I'd like to get some feedback on
the proposed design.

The idea is to let a running task request a short-lived ID token for
an audience of its choice, then use it with any service supporting
OIDC federation
(AWS AssumeRoleWithWebIdentity, GCP Workload Identity Federation,
Azure federated credentials, Vault JWT auth, Kubernetes, internal
APIs, etc.).

This would allow tasks to authenticate without storing long-lived
credentials in Airflow Connections.
GitHub Actions, GitLab CI and Kubernetes service account tokens
already follow a similar approach.

Here's what I have in mind:

1/ The API server exposes two public endpoints under a configured
issuer URL: /.well-known/openid-configuration and
/.well-known/jwks.json.
These endpoints expose public keys and OpenID issuer metadata.

2/ A new Execution API endpoint issues the token. The request is
authenticated using the task's execution token and the existing
ti:self check.
The worker only provides the audience. The API server derives the
subject and other claims (dag_id, task_id, run_id, bundle_name,
team_name, etc.) from the task instance in the database before signing
the token. Task code cannot choose its own identity.

3/ The Task SDK exposes a get_id_token(audience=...) method, routed
through the supervisor like other Execution API calls.

Providers can then use the token with their existing authentication mechanisms:
Amazon: assume_role_with_web_identity_federation
Google: SubjectTokenSupplier
Microsoft Azure: ClientAssertionCredential
HashiCorp: JWT auth type

Most of the building blocks are already there: JWTGenerator and the
JWKS helpers in api_fastapi/auth/tokens.py, task-instance-scoped
execution tokens, and the supervisor message channel.

There are a few things I'd like to get the community's input on:

a) Signing key

I'd suggest using a dedicated asymmetric key configured in a new
[workload_identity] section, separate from jwt_private_key_path in
[api_auth].
Many deployments use jwt_secret (HS512), which cannot be published as
a JWKS key. Having a separate key would also ensure that workload
identity tokens cannot be used to authenticate against Airflow's own
APIs. Internal audiences would be rejected as well.

b) Subject format

I'd suggest a Dag-level default subject format:
bundle:<bundle>:dag:<dag_id> with a configurable template.
Azure requires an exact match on sub and doesn't support wildcards.
With a per-task subject, users would need to create one federated
credential per task.
Including the bundle (and team) makes sense to me because these values
come from deployment configuration, while dag_id is chosen by the Dag
author.

c) Audiences

We should allow any audience except Airflow's internal ones.

d) Security model

The identity would essentially mean "code running in this Dag, from
this bundle".
This is consistent with the current security model, where Dag authors
are trusted to run code on workers. Still, I think this should be
explicitly documented in security model docs.
Tokens would be short-lived (5 minutes by default), masked in logs,
and issuance would be audited.
Do you see any security concerns or conflicts with the existing model,
particularly in multi-team deployments?

e) Process

I think this would require an AIP. It introduces a public endpoint, a
new Execution API route, and a new Task SDK API.

I am new here so I'm looking forward to hearing your thoughts!
I'd be happy to work on the implementation if there's interest in the proposal.

Thanks,
Baptiste

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to