anmolxlight opened a new pull request, #67127: URL: https://github.com/apache/airflow/pull/67127
## What\n\n- cache the Celery app used by inside each publisher subprocess\n- key the cache by so multi-team configurations still get isolated app instances\n- add regression coverage proving repeated publishes for the same team reuse the app while different teams remain separated\n\n## Why\n\n currently creates a fresh app for every publish. Each fresh app loses Celery's lazy backend cache, so repeatedly performs backend resolution and entry point scanning. On large deployments this can push task publishing past .\n\nCaching the app per subprocess preserves the post-AIP-67 behavior of constructing apps inside publisher subprocesses while restoring per-process amortization.\n\nfixes #67123\n\n## Tests\n\n- ============================= test session starts ============================== platform linux -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0 -- /home/ubuntu/work/airflow/.venv/bin/python cachedir: .pytest_cache rootdir: /home/ubuntu/work/airflow configfile: pyproject.toml plugins: cov-7.1.0, anyio-4.13.0, unordered-0.7.0, custom-exit-code-0.3.0, rerunfailures-16.1, xdist-3.8.0, instafail-0.5.0, time-machine-3.2.0, timeouts-1.2.1, requests-mock-1.12.1, asyncio-1.3.0, mock-3.15.1, kgb-7.3, icdiff-0.9 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s collecting ... collected 54 items providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_sentry_integration PASSED [ 1%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_supports_sentry SKIPPED [ 3%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_cli_commands_vended PASSED [ 5%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_celery_executor_init_with_args_kwargs PASSED [ 7%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_exception_propagation SKIPPED [ 9%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_gauge_executor_metrics PASSED [ 11%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_command_validation[wrong-command] SKIPPED [ 12%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_command_validation[incomplete-command] SKIPPED [ 14%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_command_validation[complete-command] SKIPPED [ 16%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_try_adopt_task_instances_none SKIPPED [ 18%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_try_adopt_task_instances SKIPPED [ 20%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_try_adopt_with_and_without_external_executor_id SKIPPED [ 22%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_cleanup_stuck_queued_tasks SKIPPED [ 24%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_revoke_task SKIPPED [ 25%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_result_backend_sqlalchemy_engine_options PASSED [ 27%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_operation_timeout_config PASSED [ 29%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_send_workloads_to_celery_hang SKIPPED [ 31%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_send_workload_uses_external_executor_id_as_celery_task_id PASSED [ 33%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_send_workload_reuses_celery_app_for_same_team PASSED [ 35%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_send_workload_keeps_celery_app_cache_team_scoped PASSED [ 37%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_celery_executor_with_no_recommended_result_backend PASSED [ 38%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_sentinel_kwargs_loaded_from_string PASSED [ 40%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_celery_task_acks_late_loaded_from_string PASSED [ 42%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_visibility_timeout_default_warns_when_not_configured PASSED [ 44%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_visibility_timeout_no_warning_when_configured PASSED [ 46%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_visibility_timeout_not_set_for_unsupported_broker PASSED [ 48%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_celery_extra_celery_config_loaded_from_string PASSED [ 50%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_result_backend_sentinel_kwargs_loaded_from_string PASSED [ 51%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_result_backend_master_name_loaded PASSED [ 53%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_result_backend_transport_options_with_multiple_options PASSED [ 55%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_result_backend_sentinel_kwargs_invalid_json PASSED [ 57%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_result_backend_sentinel_kwargs_not_dict PASSED [ 59%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_result_backend_sentinel_full_config PASSED [ 61%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestMultiTeamCeleryExecutor::test_multi_team_isolation_and_workload_routing PASSED [ 62%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestMultiTeamCeleryExecutor::test_workload_routing_through_team_specific_app PASSED [ 64%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_celery_tasks_registered_on_import PASSED [ 66%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_process_workloads_routes_execute_callback[default_queue] PASSED [ 68%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_process_workloads_routes_execute_callback[callback_queue] PASSED [ 70%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_execute_workload_runs_execute_task_before_airflow_3_3 SKIPPED [ 72%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_execute_workload_runs_base_executor_workload_on_airflow_3_3_plus PASSED [ 74%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::test_execute_workload_ignores_already_running_task PASSED [ 75%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_amqps_broker_url_builds_ssl_config PASSED [ 77%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_amqp_broker_url_still_builds_ssl_config PASSED [ 79%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_redis_mutual_tls_builds_ssl_config PASSED [ 81%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_amqps_mutual_tls_missing_key_cert_raises PASSED [ 83%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_ssl_active_without_cacert_uses_system_cas PASSED [ 85%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_amqps_broker_url_no_ssl_when_inactive PASSED [ 87%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_amqps_one_way_tls PASSED [ 88%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_redis_one_way_tls PASSED [ 90%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestAmqpsSslConfig::test_one_way_tls_ignores_key_cert PASSED [ 92%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCreateCeleryAppTeamIsolation::test_custom_celery_config_options_applied PASSED [ 94%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCreateCeleryAppTeamIsolation::test_default_celery_config_options_skipped_via_identity_check PASSED [ 96%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCreateCeleryAppTeamIsolation::test_team_specific_broker_not_overwritten PASSED [ 98%] providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCreateCeleryAppTeamIsolation::test_team_app_name_includes_team_name PASSED [100%] =========================== short test summary info ============================ SKIPPED [1] providers/celery/tests/unit/celery/executors/test_celery_executor.py:157: Test only for Airflow < 3.2 SKIPPED [1] devel-common/src/tests_common/pytest_plugin.py:702: The test 'providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_exception_propagation' requires one of 'mysql', 'postgres' backend started and 'BACKEND' environment variable to be set (currently it set to None). It can be set by specifying backend at breeze startup. SKIPPED [3] providers/celery/tests/unit/celery/executors/test_celery_executor.py:219: Airflow 3 doesn't have execute_command anymore SKIPPED [1] devel-common/src/tests_common/pytest_plugin.py:702: The test 'providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_try_adopt_task_instances_none' requires one of 'mysql', 'postgres' backend started and 'BACKEND' environment variable to be set (currently it set to None). It can be set by specifying backend at breeze startup. SKIPPED [1] devel-common/src/tests_common/pytest_plugin.py:702: The test 'providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_try_adopt_task_instances' requires one of 'mysql', 'postgres' backend started and 'BACKEND' environment variable to be set (currently it set to None). It can be set by specifying backend at breeze startup. SKIPPED [1] devel-common/src/tests_common/pytest_plugin.py:702: The test 'providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_try_adopt_with_and_without_external_executor_id' requires one of 'mysql', 'postgres' backend started and 'BACKEND' environment variable to be set (currently it set to None). It can be set by specifying backend at breeze startup. SKIPPED [1] devel-common/src/tests_common/pytest_plugin.py:702: The test 'providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_cleanup_stuck_queued_tasks' requires one of 'mysql', 'postgres' backend started and 'BACKEND' environment variable to be set (currently it set to None). It can be set by specifying backend at breeze startup. SKIPPED [1] devel-common/src/tests_common/pytest_plugin.py:702: The test 'providers/celery/tests/unit/celery/executors/test_celery_executor.py::TestCeleryExecutor::test_revoke_task' requires one of 'mysql', 'postgres' backend started and 'BACKEND' environment variable to be set (currently it set to None). It can be set by specifying backend at breeze startup. SKIPPED [1] devel-common/src/tests_common/pytest_plugin.py:661: The test is skipped because it has quarantined marker. And --include-quarantined flag is not passed to pytest. <Function test_send_workloads_to_celery_hang> SKIPPED [1] providers/celery/tests/unit/celery/executors/test_celery_executor.py:997: pre-3.3 compatibility path only applies before Airflow 3.3 ================== 42 passed, 12 skipped, 1 warning in 3.78s ===================\n- All checks passed!\n- 2 files already formatted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
