ephraimbuddy opened a new pull request, #74233: URL: https://github.com/apache/airflow/pull/74233
Dag processors currently use an in-process Execution API for parse-time requests. This adds an opt-in authenticated HTTP path through the normal `airflow dag-processor` command, with separate credentials for processor management and individual file-parsing attempts. Set `[dag_processor] execution_api_token_file` to a token file provisioned by `airflow dag-processor-token`. The processor then registers its Job, sends heartbeats, and records completion through the Execution API. Bundle secret lookups and parse-time requests use the same HTTP client. The authentication flow includes: - Externally provisioned, bundle-scoped session credentials, so the processor does not need to mint its own tokens. - Idempotent registration and completion, token renewal, and explicit handling of retired or replaced Jobs. - Short-lived parsing credentials bound to a Job, session, bundle, file, and attempt. They cannot manage Jobs or exchange tokens; resource permissions remain bundle/team based. - Startup synchronization of bundle ownership before secret lookups, bounded heartbeat-failure handling, and child-process cleanup on restart or shutdown. The migration adds nullable `session_id` and `registration_id` columns to the existing Job table; it adds no tables. The new endpoints are versioned, with typed contracts and regenerated SDK models. This does **not** make the processor DB-less: result persistence and orchestration still use the metadata database. The existing behavior remains the default when the token-file setting is unset. Authenticated processors do not use the shared SDK secret cache, whose lookups lack the bundle identity needed for this authorization path. Validation: - Breeze regression suite: 504 passed, five PostgreSQL-only tests skipped on SQLite. - Final lifecycle and API-manager checks: 26 passed. - Live-HTTP tests run the normal CLI in a separate process, with and without multi-team mode, and verify secret reads, serialized Dag persistence, and API-managed Job completion. - Ruff and core mypy passed. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
