heavelock commented on issue #11118: URL: https://github.com/apache/airflow/issues/11118#issuecomment-700060393
> > > Airflow doesn't guarantee when tasks will be executed. It only guarantee when they can start. > I'm still not sure that I fully understand your suggestion. > For your use-case I assume the flow is: > > TimeWindowSensor - > ExecutingOperator > > Your safe time as you describe it is: 1PM - 6PM > Lets assume that TimeWindowSensor was executed at 1:30PM so it succeeded and downstream branch is good to go But then cluster was under heavy load. The ExecutingOperator started running on 06:10PM. That's not the functionality you are after. Well, what can I say. `It works for me` :) I completely agree with drawbacks of this approach that you pointed out. They are completely real and this approach do not guarantee that the time-sensistive task will be executed when it was intended. However, that problem can be mitigated by adjusting priorities of tasks - low priority on the `TimeWindowSensor` and very high priority on `ExecutingOperator`. Of course, this is still not a guarantee but in my case it worked. I have no idea how this would work on big Airflow cluster where there are more than one person fiddling with priorities of tasks. > From what I can tell the solution for you is to write a combined custom operator-sensor that will check the condition and execute the action. This will also solve issues with retries where 1st attempt was in the timeframe but 2nd is not. That would be neat! I did not think about approaching it in this way. The question from my initial post is still valid though. Is it worth spending some time to implement this feature or I am the only person who needs that kind of feature. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
