ephraimbuddy opened a new pull request, #73750: URL: https://github.com/apache/airflow/pull/73750
This PoC runs Dag parsing through dedicated executor capacity and tests whether the scheduler can host parse orchestration while importing and provider calls remain outside its process. It is M0 feasibility work for a proposed AIP. This draft is for design discussion and experimentation, not for merging as a production feature. The implementation covers: - A bounded `ParseDagDefinitions` workload for LocalExecutor and Celery, with separate parsing capacity and explicit routing. - SDK importing for Python files and ZIP members, with serialized results crossing an authenticated experimental API boundary. - Durable claims, per-definition receipts, admission accounting and conservative recovery when delivery or termination is uncertain. - An opt-in `airflow dag-processor --executor-parsing` command and a scheduler-hosting experiment using the same `ParseOrchestrator` against registered inventory. - Breeze tests and reproducible drivers for isolated Celery workers, recovery, metadata publication, scheduler responsiveness and local measurements. Normal deployments still require the Dag processor. The prototype uses auxiliary SQLite tables and a separate development API; production migrations, multi-team authorization, remote discovery, HA ownership/adoption, Kubernetes, callback and priority handling, source removal and automatic recovery of uncertain remote execution remain open. The local route shares its host's trust; the Celery experiments use a separately provisioned worker without the metadata database or signing key. The recorded local benchmark is slower than the existing manager, so this does not claim a performance improvement. Two small scheduler-hosting experiments kept scheduling during slow imports, a paused broker and database contention, and drained all reservations. These are feasibility results, not production latency guarantees. Validation and reproduction: - Before the upstream rebase: 346 focused regression tests passed, alongside two live scheduler/Celery experiments. Earlier checkpoints and their overlapping test counts are documented in the work log. - The branch has been rebased onto `upstream/main`; full post-rebase validation has not been confirmed. - Run `breeze run pytest airflow-core/tests/integration/dag_processing/test_executor_parsing.py -v` for the self-contained Dag processor command test. - [Usage and scheduler experiment](https://github.com/astronomer/airflow/blob/dag-parsing-executor-poc/airflow-core/docs/administration-and-deployment/dagfile-processing.rst), [work log](https://github.com/astronomer/airflow/blob/dag-parsing-executor-poc/files/dag-parsing-aip/WORK_LOG.md), [scheduler measurements](https://github.com/astronomer/airflow/blob/dag-parsing-executor-poc/files/dag-parsing-aip/scheduler-hosting-20260926.md), and [local benchmark and recovery findings](https://github.com/astronomer/airflow/blob/dag-parsing-executor-poc/files/dag-parsing-aip/recovery-and-scan-20260926.md). Raw run artifacts referenced by those reports are local and are not included in this PR. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
