seanmuth commented on issue #74063: URL: https://github.com/apache/airflow/issues/74063#issuecomment-6616361228
@jason810496 following up on @luc-pimentel's question above, since the same design question applies to #60858 (task process SIGKILLed while the supervisor survives, so neither `on_failure_callback` nor the OL FAIL event runs). We'd like to push for luc's option 1: the supervisor decides that a callback is needed and flags it on the state update, and the API server only queues the callback request for the Dag processor. That is the same queue the scheduler already uses on the "killed externally" mismatch path. This looks compatible with fixing at the supervisor level. The supervisor can't run the callback itself, because it doesn't parse the Dag file and doesn't write to the callback table, so the queuing has to happen server-side. The difference from #74578 is that the server doesn't infer on its own that a callback is needed. The supervisor makes that call and the server just enqueues it. We don't like option 2 either (the supervisor starts a Python process and runs the callbacks itself). It would make the worker load the Dag file and would duplicate the Dag processor's callback code. It also looks race-prone: the supervisor would be running a new process while it is reporting the final state and exiting, so the callback could be cut off if the pod is terminated. When the original kill was an OOM, the new process would run in the same memory-starved pod. With option 1, #74063 and #60858 could share one mechanism, with a single flag on the state update and different triggers. Does that direction work for you? If so, I'm happy to help with the implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
