namanjain24-sudo opened a new pull request, #73172: URL: https://github.com/apache/airflow/pull/73172
A task with `executor="KubernetesExecutor"` fails to start when the worker it lands on has a different executor configured, with the error from #56271: ``` UnknownExecutorException: Task 'a' specifies executor 'KubernetesExecutor', which is not available ... ``` `parse()` in the task runner loads the Dag through `BundleDagBag`, and `DagBag.process_file` runs `_validate_executor_fields` on every Dag it bags. A worker usually has only its own executor configured. The cncf-kubernetes pod templates, for example `kubernetes_executor_templates/basic_template.yaml` and the files in `pod_template_file_examples/`, set `AIRFLOW__CORE__EXECUTOR=LocalExecutor`. So the check fails inside the pod, the Dag is dropped from the bag, and the task cannot start. The Dag processor has already run this check when it parsed the file, so the worker does not need to repeat it. This adds a `validate_executors` argument to `DagBag` that defaults to `True`, and the task runner passes `False`. The Dag processor, the CLI and `dag.test()` keep validating as before. Tests: - `test_parse_does_not_validate_task_executors` parses a Dag whose task uses an executor the worker does not know. It fails on main with the error above and passes with this change. - `test_validate_executors` in `test_dagbag.py` covers both values of the new argument. - `test_task_runner.py` (536 passed with `TZ=UTC`; in a non-UTC local timezone eight `TestTriggerDagRunOperator` tests fail on main as well), `test_dagbag.py` (76 passed, 1 skipped) and prek, including mypy for airflow-core and task-sdk, pass locally. closes: #56271 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: a Gen-AI coding assistant, following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions). I reviewed the change and ran the checks above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
