GitHub user kaxil added a comment to the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)
The reframed proposal makes sense to me. I checked the Task SDK: `execution_timeout` is enforced in-process with SIGALRM, and the Python-level handler can only run once the main thread re-enters the interpreter loop. If the main thread is stuck inside a native call that never returns to the interpreter (the JPype case, whether or not the GIL is released), or the extension retries EINTR internally, that never happens and the timeout silently doesn't fire. And since the supervisor heartbeats on the task's behalf for as long as the child process is alive, heartbeat-based detection can't see it either. So a task that's alive but permanently wedged is invisible today. Where I'd simplify the shape: I wouldn't start with a new arm/disarm primitive for hook authors. It needs every provider to be instrumented before it covers anything, and hook authors can't pick good deadlines anyway; a legitimately slow 4-hour query and a wedged one look identical from inside the hook. The DAG author is the one who knows expected runtime, and they already declare it: `execution_timeout`. So the smaller first step is to make the supervisor enforce `execution_timeout` as a backstop, and there's already an open issue for it: #53337. The liveness case here is a good argument for prioritizing it. The task runner knows the timeout once it has parsed the task and already has a message channel to the supervisor (though as Ash notes on that issue, the timeout could also ride the Run request), so this is one new message plus one clock check in the supervisor's existing poll loop, reusing the supervisor's kill escalation. Set the supervisor deadline to `execution_timeout` plus a grace period and well-behaved tasks keep exactly today's semantics (`AirflowTaskTimeout` raised in-process, `on_kill` runs); the supervisor only steps in when the in-process timeout has failed to fire. There is precedent in the SDK: the callback supervisor does the same thing for `callback_execution_timeout`, a wall-clock check in the monitor loop that kills the subprocess. This would be Task SDK / Airflow 3 only, since 2.x is EOL. On AIP-86: deadline alerts are dag-run-level and alerting-only, while this is worker-local enforcement, so they're complementary, but a proposal using the word "deadline" should say so explicitly. Per-call arming could ride on the same mechanism later if finer granularity turns out to be needed. The JVM daemon idea I'd keep provider-local; per-task process isolation already exists, and the liveness backstop is the part that belongs in core. One caveat: `execution_timeout` doesn't span deferrals today (#69678 tracks that separately), and supervisor enforcement doesn't change that, since the process exits at deferral time. GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17856562 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
