zoomingrocket opened a new issue, #73583: URL: https://github.com/apache/airflow/issues/73583
### Under which category would you file this issue? Providers ### Apache Airflow version 3.3.2 ### What happened and how to reproduce it? Airflow Version Airflow: 3.3.2 Python: 3.12.13 OS: RHEL 9 (x86_64) Executor: LocalExecutor Metadata DB: PostgreSQL OpenTelemetry enabled via Airflow configuration Azure Key Vault secrets backend enabled Description After upgrading from Airflow 3.1.8 to 3.3.2, the environment became unusable when OpenTelemetry was enabled. Two independent symptoms occur: DAG Processor DAG files are discovered normally. DAG Processor starts parser children. Child processes never complete parsing. DAG File Processing Stats repeatedly show: # DAGs = 0 # Errors = 0 Current Duration = increasing until hitting: dag_file_processor_timeout and being killed. After timeout: # Errors = 1 is recorded. No actual import errors are written to: import_error and: serialized_dag is not updated by the live DAG processor. Manual serialization via: airflow dags reserialize works correctly. LocalExecutor Scheduler successfully moves tasks: scheduled -> queued Example scheduler log: Trying to enqueue tasks ... for executor: LocalExecutor(parallelism=4) However: tasks remain permanently queued LocalExecutor workers never appear tasks never transition to running Key Observation Disabling OpenTelemetry entirely immediately resolves both issues. When OTEL is enabled DAG Processor parser children hang LocalExecutor workers never start Tasks remain queued indefinitely When OTEL is disabled DAG Processor functions normally serialized_dag updates LocalExecutor workers start queued tasks execute normally This behavior is fully reproducible. py-spy Evidence DAG Processor child Repeatedly captured in: submit get_aggregated_resources _get_process_dependent_resource _handle_fork _after_in_child Example stack: submit (concurrent/futures/thread.py) get_aggregated_resources (resources/__init__.py) _get_process_dependent_resource (resources/__init__.py) _handle_fork (trace/__init__.py) _after_in_child (trace/__init__.py) Scheduler LocalExecutor worker creation Captured while trying to spawn LocalExecutor workers: _spawn_worker _spawn_workers_with_gc_freeze LocalExecutor.start multiprocessing.Process.start _launch _after_in_child _handle_fork Example: _spawn_worker (airflow/executors/local_executor.py) _spawn_workers_with_gc_freeze (airflow/executors/local_executor.py) start (airflow/executors/local_executor.py) _launch (multiprocessing/popen_fork.py) _after_in_child (trace/__init__.py) _handle_fork (trace/__init__.py) The LocalExecutor worker never completes startup. Additional Findings Works Metadata DB connectivity Scheduler heartbeat Triggerer heartbeat API Server Azure Key Vault secret resolution Manual DAG import Manual DAG serialization Does not work DAG Processor parser child completion LocalExecutor worker startup OpenTelemetry Versions opentelemetry-api==1.44.0 opentelemetry-sdk==1.44.0 opentelemetry-exporter-otlp==1.44.0 opentelemetry-exporter-otlp-proto-grpc==1.44.0 opentelemetry-exporter-otlp-proto-http==1.44.0 opentelemetry-semantic-conventions==0.65b0 Installed via official Airflow 3.3.2 constraints. Comparison with Airflow 3.1.8 On the same infrastructure: Same OS Same Python major version Same PostgreSQL Same Key Vault integration Airflow 3.1.8 works correctly. Notably, LocalExecutor worker processes are created normally in 3.1.8, while they never appear in 3.3.2 when OpenTelemetry is enabled. Questions Is this a known issue with Airflow 3.3.2 + OpenTelemetry? Is there a known interaction between OpenTelemetry fork handlers and: DAG Processor worker creation? LocalExecutor worker creation? Is this related to any work merged after 3.3.2 around: Task SDK process creation fork/exec handling OpenTelemetry resource initialization inherited lock handling? Are there recommended OpenTelemetry settings for Airflow 3.3.x when using LocalExecutor? *Copilot help taken to format as part of the debugging session ### What you think should happen instead? DAG Processor, Scheduler like components should startup and operate normally ### Operating System RHEL 9.x ### Deployment Virtualenv installation ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers _No response_ ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
