Andrushika opened a new pull request, #75400: URL: https://github.com/apache/airflow/pull/75400
## Why Every Python Dag file import goes through `PythonDagImporter._load_modules()`. It runs in the Dag processor parse process, and again in the task process on the worker before the task starts. Since #51171, this import sets a `SIGSEGV` handler and never restores it. A Python signal handler cannot run for a real segfault: the C level handler returns, the same instruction faults again, and the process spins at 100% CPU. The [Python `signal` docs](https://docs.python.org/3/library/signal.html#execution-of-python-signal-handlers) warn about this: > It makes little sense to catch synchronous errors like SIGFPE or SIGSEGV that are caused by an invalid operation in C code. Python will return from the signal handler to the C code, which is likely to raise the same signal again, causing Python to apparently hang. On main, with a Dag that calls `ctypes.string_at(0)`: - In `execute()`, the task instance is still `running` after 60s. With `execution_timeout=10s`, the supervisor kills it after about 132s. - At the top of the Dag file, the task process is still `running` after 200s. - In the Dag processor, every parse waits for `dag_file_processor_timeout` (50s). No import error is shown, and the Dags go stale after about 70s without a reason. The handler only runs for a signal sent from outside, like `os.kill(os.getpid(), signal.SIGSEGV)` in #50303. It never runs for a real crash. ## What Remove the handler. A crashing process now dies with `SIGSEGV` right away, and the supervisor writes its usual SIGSEGV message to the task log or the Dag processor log. `DagFileProcessorProcess.is_ready` now turns a parse killed by a crash signal (`SIGSEGV`, `SIGBUS`, `SIGILL`, `SIGFPE`, `SIGABRT`) into an import error for the file. The Lang-SDK processor already does this when its runtime exits without a result. `SIGKILL`, for example from the OOM killer, keeps today's behavior and does not become an import error. The two removed `test_dagbag.py` tests covered the handler itself. One of them sends `SIGSEGV` to the pytest process. related: #50303 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
