The GitHub Actions job "Tests" on airflow.git/fix/multiple-executors-queue-routing has failed. Run started by GitHub user Arunodoy18 (triggered by Arunodoy18).
Head commit for run: ff8ef814977519f40fe5cde388c37cc8759088b3 / Arunodoy18 <[email protected]> Fix: Enable queue-based executor routing for multiple executors in Airflow 3.0+ Problem: In Airflow 3.0+, when using multiple executors with comma-separated config (e.g., executor='CeleryExecutor,KubernetesExecutor'), tasks were not being routed to the correct executor based on their queue parameter. All tasks were being sent to the first (default) executor regardless of their queue. For example, tasks with queue='kubernetes' were executed by CeleryExecutor instead of being routed to KubernetesExecutor. Root Cause: The scheduler's _try_to_load_executor() method only checked for explicitly set ti.executor values but did not consider the task's queue parameter for routing decisions. This differed from Airflow 2.x hybrid executors like CeleryKubernetesExecutor which supported queue-based routing. Solution: 1. Enhanced _try_to_load_executor() in scheduler_job_runner.py to check if a task's queue matches any executor's kubernetes_queue configuration before falling back to the default executor. 2. Modified KubernetesExecutor.__init__() to read kubernetes_queue from config [kubernetes_executor] section (default: 'kubernetes'). 3. Added kubernetes_queue configuration option to: - airflow/config_templates/provider_config_fallback_defaults.cfg - providers/cncf/kubernetes/get_provider_info.py 4. Added comprehensive test test_try_to_load_executor_queue_based_routing() to verify queue-based routing works correctly. This fix restores the queue-based routing behavior from Airflow 2.x hybrid executors (CeleryKubernetesExecutor, LocalKubernetesExecutor) to work with Airflow 3.0's true multi-executor architecture. Testing: - Tasks with queue='kubernetes' are routed to KubernetesExecutor - Tasks with other queues use the default CeleryExecutor - Existing explicit executor assignments still work (backward compatible) - Multi-team executor selection is not affected Fixes #<issue-number> Report URL: https://github.com/apache/airflow/actions/runs/20694292792 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
