vandonr-amz commented on issue #21867: URL: https://github.com/apache/airflow/issues/21867#issuecomment-1687032240
Thank you for the insightful comments. I thought previously that a task group would have only one "output path", but it turns out that tasks outside the group can indeed start before the group itself is complete, as a group can cover tasks across multiple independent graphs. <img width="446" alt="image" src="https://github.com/apache/airflow/assets/114772123/db8c4b07-afa1-4c0d-a544-85e353c8a6b5";> Also, I was just thinking, as this discussion is getting more and more complex... The initial goal of the users was to be able to retry an other task as well if one fails, and "retries for taskgroups" was a proposed solution, but maybe we can find an other solution that would have less reach ? What if we could define "dependent tasks" for a task for instance, and a retry of a task would imply a retry of its dependent tasks too ? It's less powerful, but I think it could cover the users' need, and require less brain twisting from our side ? Given the example usages given above, we could even be more specific in naming, calling it for example `validates`, and a validation task failing means that the task it validates needs to re-run. Like `task3 = EmptyOperator(task_id="task3", validates=[task1, task2], retries=3)` And to avoid complicated questions, we could even enforce when parsing the dag that tasks being validated need to be upstream from the validating task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
