zach-b commented on issue #32289: URL: https://github.com/apache/airflow/issues/32289#issuecomment-1632504873
Thanks @potiuk , I took a look but I'm not very familiar with Airflow internals (this would be my first contribution) and it seems that the fix will not be trivial. I see two options here, none of which sounds perfect: - Ensure that the scheduler schedules the TaskCallbackRequest only once: to do this we would need to maintain a list of the zombie tasks that are already waiting to be cleared, if for some reason clearing is not executed this would also mean that the task would be stuck, not sure this is great… - Check if the task still needs to be cleared when running the TaskCallbackRequest (i.e. that it still meets the Zombie conditions), to do that we would need to distinguish zombie callbacks from others in the TaskCallbackRequest and have dedicated code in the processor… What do you think ? Anyone else who could provide guidance here ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
