o-nikolas commented on PR #38514: URL: https://github.com/apache/airflow/pull/38514#issuecomment-2033219581
> on executor end... i looked at the way executors have implemented end... i was surprised to see that they wait for all tasks to finish. given that we have re-adoption logic, i wonder if it actually makes sense to do that. why not just stop running and let the new scheduler (presumably there will be one?) pick up the tasks. perhaps it could hasten the process by unassigning i.e. disassociating from itself its tasks. The executor interface has both an `end` which waits synchronously and also `terminate` which ends more aggressively (closer to what you describe) when a SIGTERM is received. It's up to executors to be able to do both. I suppose we could move the scheduler to call terminate in more cases than when just a sigterm is received. But that needs more thinking and is certainly out of scope for this PR which is just to ensure that the current behaviour we have today works with multiple executors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
