GitHub user seanmuth edited a comment on the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)
Second, independent case for the same blind spot: OpenSSL fork-orphaned lock ([#71707](https://github.com/apache/airflow/issues/71707), fix merged as [#72164](https://github.com/apache/airflow/pull/72164), shipping 3.3.2/3.4.0). Task SDK's supervisor forks the task process via a bare `os.fork()`; if any supervisor thread (OTel/Sentry export, google-auth refresh, secrets-backend polling — confirmed across three separate customer incidents) is mid-critical-section on OpenSSL 3.x's provider-store lock at that instant, the lock is orphaned into the child permanently — no thread survives fork to release it. Same blind spot you both already established for JPype: `execution_timeout`'s SIGALRM never fires (the thread's stuck in native code, never returns to the interpreter loop), and heartbeating stays healthy regardless. Real-world result: task pods alive, heartbeating, and frozen for 3+ days before manual intervention. +1 on making the supervisor enforce `execution_timeout` as a backstop (#53337) — it's the right default-on mechanism and this case is more corroborating evidence for it. Where I'd push for keeping the no-progress/low-CPU angle alive as a second, complementary layer rather than closing it out once #53337 lands: `execution_timeout` only bounds a wedge to whatever the DAG author declared, and two things are true in practice for this specific failure mode. Plenty of deployments don't set it at all. And where it is set, it's commonly hours, not minutes, sized for the task's legitimate runtime rather than for detecting a hang. The OpenSSL deadlock happens at the instant of fork, before the task has done any real work — so a task wedged this way is knowably stuck almost immediately, and a multi-hour `execution_timeout` still leaves it sitting, heartbeating, doing nothing, for most of that window. The two backstops catch different things: `execution_timeout` catches "ran longer than the DAG author's declared bound," which is correct and sufficient for most hangs; a no-progress/low-CPU check catches "this specific process has been doing zero work since shor tly after it started," independent of whatever timeout is or isn't configured. A genuinely wedged process here shows a CPU flatline from very early on, not just "slow" — confirmed directly against a live wedge in this investigation — which is a different and earlier-available signal than elapsed wall-clock time alone. Not proposing this as a replacement for #53337 or as fully specified yet — just think it's worth continued exploration alongside it, since the two are complementary defense-in-depth rather than competing designs. --- Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
