rahul-madaan opened a new pull request, #72835: URL: https://github.com/apache/airflow/pull/72835
Follow-up to #72711, which is superseded by this PR and by OpenLineage/OpenLineage#4935. **Blocked until `openlineage-python` 1.54.0 is released.** CI will fail until then, because the pin in `providers/openlineage/pyproject.toml` points at a version that is not on PyPI yet and the code imports a class that only exists there. `uv.lock` also needs regenerating once the release lands; `uv` cannot resolve the workspace before that. ### Background OpenLineage backends protected by an OAuth 2.0 authorization server issue short-lived access tokens to a client ID and client secret. The token provider for that grant belongs in the OpenLineage client rather than in this provider, so that every integration and every Airflow version can use it, and it is being added there as the `oauth2_client_credentials` auth type in OpenLineage/OpenLineage#4935. Once that is released, an Airflow user can already use it by putting the credentials in `[openlineage] transport`. This PR adds the part that is Airflow specific. ### What this adds - `auth.type: airflow_connection_oauth2_client_credentials`, resolved from an Airflow connection the same way `airflow_connection_api_key` is (#66342): login is the client ID, password is the client secret, host is the token endpoint. It keeps the client secret in whichever secrets backend is already configured instead of in `airflow.cfg`. Works for `http` and `async_http` transports, including nested `composite` ones. - Spark transport injection forwards the current access token as `api_key` auth, and logs that Spark cannot refresh it. Forwarding the client credentials instead would let the Java client refresh them, but it would put a long-lived secret into Spark properties, so the shorter-lived token seemed the better default. Happy to switch if you disagree. - Bumps `openlineage-python` to `>=1.54.0`. ### Testing Unit tests for connection resolution, including nested composite transports with mixed auth types, and for both Spark injection paths. Verified end to end against a Keycloak-protected backend with a real Dag run on Airflow 3.3.0, using both the inline and the connection-backed configuration, plus a soak run confirming the token is refreshed before it expires. closes: #72711 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QBfMWG5AG33JMqGGQgAV6v -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
