dabla opened a new pull request, #77007: URL: https://github.com/apache/airflow/pull/77007
Listing Dags with the Keycloak auth manager costs one request to Keycloak per Dag, and since Airflow 3.4.0 the Dag list of the UI adds one request per asset. This PR adds an opt-in setting, `[keycloak_auth_manager] team_level_authorization`, for realms whose policies do not look at the individual resource. With it, the list filters ask Keycloak once per team and apply the answer to all resources of that type. related: #71499 ## Why The auth manager models a resource type as one Keycloak resource (`Dag`, or `Dag:<team>` in multi-team mode) and sends the id of the Dag as a claim, so that a policy *can* decide per Dag. To list Dags it therefore has to ask once per Dag. That is the latency reported in #71499 (about 900 Dags, 20 s and more for the Dag list). Many realms never use that possibility. The permissions the CLI creates (`create-permissions`, `create-team`) are granted by role and group only, and no policy reads the `resource_id` claim. Keycloak then returns the same answer for every Dag of a team, a thousand times. The reporter of #71499 describes exactly this setup. Since Airflow 3.4.0 this got worse for the UI: the Dag list endpoint (`/ui/dags`) also depends on the readable assets, so each uncached request authorises every asset one by one as well. ## What was tried before - #56433 implemented batch filtering with a cache warmed in a background thread. It was closed as too complex, to wait for the multi-team model. - #63184 added the TTL cache with single-flight, and #70647 the thread pools. They make the cost bearable but not small: the cache is per API server process and lasts 30 s, so with several API server processes most requests still start cold. - #71507 added a core hook for auth managers whose decision does not depend on the Dag id. Keycloak did not opt in, and the PR was closed. - [keycloak/keycloak#45311](https://github.com/keycloak/keycloak/issues/45311) asks Keycloak to accept several claim sets in one request, which would allow real batching while keeping per-resource policies. It is still in triage. - The remaining route discussed in #71499 is to model every Dag as its own Keycloak resource, which needs a remodel, a migration and a sync on every parse. This PR is a simpler offer for the deployments that do not need per-resource policies: they declare it, and the auth manager stops asking the same question per resource. ## What changes With `team_level_authorization = True`: - `filter_authorized_dag_ids`, `filter_authorized_connections`, `filter_authorized_pools` and `filter_authorized_variables` send one request per call. Core calls them once per team, so that is one request per team. - `filter_authorized_assets` sends one request for all assets, which have no team resource. - The request is the one sent today for a single resource: it names one resource of the list and its team. The same permission is evaluated in Keycloak. - The result is cached exactly as before (same key, same TTL). Nothing changes when the setting is off, which is the default. Checks of a single resource and the `batch_is_authorized_*` methods are not touched. The setting must stay off when a policy grants or denies access per resource (for example a JavaScript or regex policy on the `resource_id` claim): such a policy would be evaluated for one resource and its answer applied to the others. The documentation says so. ## Measurements We run this logic in production as a patch over provider 0.11.2 on Airflow 3.4.0, with permissions per team. Timings of `GET /ui/dags?limit=50&paused=false` for one user, three API server pods: | Deployment | Before | After | | --- | --- | --- | | 1,107 Dags, 1,828 assets, 6 teams | 5 to 12 s on six calls in a row | 0.35 to 0.56 s on 36 of 36 calls | | 1,454 Dags, 5 teams | 26 s with a cold cache, 0.64 s when warm | 0.5 to 0.8 s (1.3 to 1.9 s on the first calls after the API servers restarted), 27 calls | The "after" rounds were run 40 s apart, so each started with an expired cache. With Dags per team only and assets still one by one, the first deployment still had one call of 9 to 10 s per pod whenever the cache expired; that is the 3.4.0 asset check. ## Open points - The five filters each get the same three-line branch. They already share their whole structure, so they could be folded into one helper; I kept that out to leave the existing code untouched in a change about authorization. - Whether the cache is still worth having in team-level mode (a handful of requests per call) is a fair question. This PR keeps it, to change one thing only. - `version_added` is set to 0.13.0; please correct it if the next release is numbered differently. ## Tests - New tests in `test_keycloak_auth_manager.py`: one decision for each of the five filters (granted and denied), the request naming a listed Dag and its team, no request for an empty list, and the cached decision. - `uv run --project providers/keycloak pytest providers/keycloak/tests/unit/keycloak`: 340 passed, 1 skipped, 3 errors. The 3 errors are in tests this PR does not touch and come from my local environment (`ModuleNotFoundError: No module named 'airflow_shared'` in the `caplog` fixture). - `prek run --files <the six changed files>`: 61 hooks passed. `update-providers-build-files` could not run locally (no route to my package index from the Breeze container), so I wrote the `get_provider_info.py` entry by hand and checked that it equals the `provider.yaml` entry. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Fable 5.2) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
