thesuperzapper commented on issue #36090: URL: https://github.com/apache/airflow/issues/36090#issuecomment-2228847790
@sunank200 which approach are we planning to take on this issue? 1. Implement the workaround I described in https://github.com/apache/airflow/issues/36090#issuecomment-2094972855 for each deferred operator that needs to clean up external resources (that is, the non-deferred version uses `on_kill`) - _(NOTE: for the ones we already updated, we still need to update them to handle the race condition when clearing that I described in https://github.com/apache/airflow/issues/36090#issuecomment-2164294237)_ 2. Attempting a general fix. I think we should do both, starting with the workaround, as we should be able to get that in for 2.10.1. It's __very important__ we do something soon, because users don't expect this behavior, and it can be quite detrimental to your data pipelines if external jobs/resources keep running when you manually mark tasks as failed/success/clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
