GitHub user seanmuth added a comment to the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)
**Second, independent case for the same gap: OpenSSL fork-orphaned lock (ZD98087/ZD98601, [#71707](https://github.com/apache/airflow/issues/71707))** Different native library, same structural failure mode as the JVM case above, and it sharpens why an in-process remedy can't generalize. **Mechanism:** OpenSSL 3.x's provider-store `pthread_rwlock` has no fork handler. Task SDK's supervisor spawns the task process via a bare `os.fork()` (`WatchedSubprocess.start()`); if any thread in the supervisor (OTel/Sentry export, google-auth token refresh, secrets-backend polling — all confirmed independently across three separate customer incidents) is mid-critical-section on that lock at the instant of fork, the lock is orphaned into the child in a permanently locked state — no thread survives fork to ever release it. The next OpenSSL call the task makes (typically its first outbound HTTPS call) blocks forever. No exception, no OOM, no crash. **Why this can't be caught in-process, and why that generalizes beyond this one bug:** `execution_timeout` is implemented as `SIGALRM` + `signal.setitimer` (`airflow.sdk.execution_time.timeout.TimeoutPosix`) — an in-process signal-based timeout. Two independent reasons it cannot fire against this class of hang: CPython only dispatches a pending signal handler when control returns to the bytecode eval loop, and a thread blocked inside a native call (`SSL_CTX_new_ex` → `pthread_rwlock_wrlock`) never returns there; and even if the signal does interrupt the underlying `futex_wait` at the kernel level, glibc's rwlock wait loop treats that as a spurious wakeup and retries rather than propagating a failure. Heartbeating doesn't help either — it's a separate channel that stays healthy regardless of whether the task thread is making progress, same as the zombie-detection gap described above for the JVM case. Real-world result: task pods sat alive, heartbeating, and completely frozen for as long as 3+ days before manual intervention. **Why I think this belongs at the supervisor level, not as two more per-provider patches:** the JDBC/JVM case and this one have nothing in common at the library level, but they fail identically from the supervisor's point of view — a child process that is alive, heartbeating, and making zero forward progress, for a reason no in-process signal can observe or interrupt. A supervisor-level check (wall-clock time since the last real progress signal from the child, cross-checked against near-zero child CPU sampled externally, terminating unconditionally via `SIGKILL` past a threshold) doesn't need to know anything about JVMs or OpenSSL to catch either case. Notably, the supervisor is a safe place to add this: the fork hazard here is asymmetric — the *parent's* copy of the lock isn't stuck, only the child's is, since the thread holding it in the parent survives and releases it normally moments later — so the supervisor can never be wedged by the same event that wedges its child. To be clear, this isn't a substitute for fixing either root cause (fork+exec for the task process on Linux is already merged, [#72164](https://github.com/apache/airflow/pull/72164), shipping in 3.3.2 and 3.4.0; JVM isolation is the right fix for the JDBC case above) — it's a backstop that bounds the blast radius to a bounded retry instead of a silent multi-day hang, regardless of which native library is at fault next time. --- Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
