potiuk commented on issue #26200: URL: https://github.com/apache/airflow/issues/26200#issuecomment-1250260900
Hmmm I think everything is as expected. 1) when you manually set task on 10.00 to "skipped" - it will be skipped without taking "depends_on_past" into account. "Depends_on_past" only prevents the tasks from RUNNING when the same task in previous instance is not "succeeded" or "skipped". But when you manually force it to "skip" this is not done. If you would like it to "wait" for previous task to run, you would have to have a new state "waiting for skipping" or smth. I.e. you would like to have a state when we know that the task should be skipped immediately when it is eligible to run. Airfow has no such state and it's quite doubtful if it will ever have one 2) Then the task at 11.00 will only look at the "last previous", not "all previous" tasks and it will see tha the previous task is skipped, so it will run immediately. All is fine, as expected and perfectly reflects the current state of Airflow implementation, so this is not a bug. What you likely would like to get instead is a new feature for example "depend_on_all_past_not_runnning" or similar, where you should check if there is ANY past task that is currently not "completed" and "queued" or "running". But I think this one is not needed. It is quite a bit more complex to implemented, I think and it might be a bit more "predictable" in terms of execution sequence, but probably much better, simpler and something that you can implement even now, is to assign a separate pool for your task with size = 1. This will make sure that only one task instance can run at a time. This is PROBABLY what you really want to achieve. There is no guarantee on sequence of execution - so if you run 5 of those instances one after the other then, they will execute in a "random" sequence, but In most cases of scheduled run, they should run in a proper sequence as past task will be already running while the future task is scheduled. Converting to a discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
