Bisk1 commented on issue #33661: URL: https://github.com/apache/airflow/issues/33661#issuecomment-1690201811
I felt like this kind of situation should be handled by adopt_or_reset_orphaned_tasks() part of scheduler, but apparently that piece of code is meant to handle tasks that were orphaned by the scheduler process. In current implementation, the task process is responsible for a task in RESTARTING state, not the scheduler process. I think that the responsibility for handling tasks in RESTARTING state should be shifted to scheduler. Then scheduler would be able to handle orphaned RESTARTING tasks. The task process itself would only handle SIGTERM to interrupt the restarted task (on best-effort basis). My only concern is that it would cause race condition - what if the scheduler attempts to reschedule restarted task before the old task process handled SIGTERM - we could have 2 process with the same task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
