shubham22 commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1684691599
Thank you for the very detailed explanation, @dstandish. Your insights are
appreciated, and it took me three days to muster the courage to read through it
😅
> So this seems like a non-issue, except for ... do you allow a retryable
group within a retryable group... oy... I guess in principle there's no obvious
problem. The retry of the outer implies retry of the inner
Yes, the notion of retrying the inner group when retrying the outer, as well
as configuring retrying explicitly for just the inner task group, seems
intuitive and should not cause any problems.
> Certainly though, clearing should always clear all the tasks in the
container, no?
Agree.
> Though we could extend this option to all tasks. And this could be
generalized to "should i be considered for container state" i.e. group or dag.
Conceptually, delegating this decision to a task, so that it can declare
whether it should be considered to determine the container state, appears both
reasonable and clean. I also like that it aligns with the logic used for a
teardown task, making it easier for users to learn and apply.
> It might make sense to simply delegate to the task itself whether it
should be rerun on retries of the container (group or dag).
Here, I think an opinionated approach would be easier to adopt and
configure. Essentially, users would either want to retry failed tasks or the
whole container in most cases, no? Too many task-level configurations might
overly complicate the task model. Should we go down this path, I anticipate a
name like `retry_on_group_retry` :D
> So yeah, I think given the above discussion, delegating to task seems
simpler / more flexible, no?
Yea, I could buy this as users would want more flexibility here and
delegating to the task provides that. `on_failure_trigger_group_retry`.
> You may not be aware, but recently added was a fail_stop feature that
controls this behavior. May not need to mess with that here.
+ 1
> clear, i.e. no status, would seem to be the obvious choice, consistent
with current behavior.
+ 1
> note first that you can have things outside a group which are downstream
of things inside a group. so, what happens when you have tasks that branch off
downstream from tasks in the middle of your retryable task group. in effect, by
retrying your task group, you are clearing "upstream". But do you also "recurse
downstream", i.e. any tasks that are cleared in the group will be recursed
downstream and everything else even outside the group will also be cleared.
Great callout. I hadn't previously considered the recurse downstream
functionality. I also wonder if we really need to? For the sake of simplicity,
we could allow users to choose `recurse downstream` for all involved tasks in
the group and borrow recursing logic from the involved tasks. This would mean
that if a user selects a "retry only the failed tasks" strategy, they wouldn't
need to retry tasks that are downstream of already successful tasks within the
task group.
> Indeed thinking this over it crossed my mind that perhaps what we want is
to reimplement subdag as an inline construct, something different from task
groups.
Personally, I'd be against this, especially given that we have [discouraged
users from using subdags due to performance
issues](https://airflow.apache.org/docs/apache-airflow/2.7.0/core-concepts/dags.html#taskgroups-vs-subdags).
I think we should look into extending task groups and make them more
functional, rather than reverting to extend subdags or reimplement them. Both
are quite similar concepts, we should adopt one and enrich it, so that users
don't need to read yet another documentation page on when to use what.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]