shubham22 commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1684691599

   Thank you for the very detailed explanation, @dstandish. Your insights are 
appreciated, and it took me three days to muster the courage to read through it 
😅
   
   > So this seems like a non-issue, except for ... do you allow a retryable 
group within a retryable group... oy... I guess in principle there's no obvious 
problem. The retry of the outer implies retry of the inner
   
   Yes, the notion of retrying the inner group when retrying the outer, as well 
as configuring retrying explicitly for just the inner task group, seems 
intuitive and should not cause any problems.
   
   > Certainly though, clearing should always clear all the tasks in the 
container, no?
   
   Agree. 
   
   > Though we could extend this option to all tasks. And this could be 
generalized to "should i be considered for container state" i.e. group or dag.
   
   Conceptually, delegating this decision to a task, so that it can declare 
whether it should be considered to determine the container state, appears both 
reasonable and clean. I also like that it aligns with the logic used for a 
teardown task, making it easier for users to learn and apply.
   
   > It might make sense to simply delegate to the task itself whether it 
should be rerun on retries of the container (group or dag).
   
   Here, I think an opinionated approach would be easier to adopt and 
configure. Essentially, users would either want to retry failed tasks or the 
whole container in most cases, no? Too many task-level configurations might 
overly complicate the task model. Should we go down this path, I anticipate a 
name like `retry_on_group_retry` :D
   
   > So yeah, I think given the above discussion, delegating to task seems 
simpler / more flexible, no?
   
   Yea, I could buy this as users would want more flexibility here and 
delegating to the task provides that. `on_failure_trigger_group_retry`.
   
   > You may not be aware, but recently added was a fail_stop feature that 
controls this behavior. May not need to mess with that here.
    
    + 1
   
   > clear, i.e. no status, would seem to be the obvious choice, consistent 
with current behavior. 
   
   + 1
   
   > note first that you can have things outside a group which are downstream 
of things inside a group. so, what happens when you have tasks that branch off 
downstream from tasks in the middle of your retryable task group. in effect, by 
retrying your task group, you are clearing "upstream". But do you also "recurse 
downstream", i.e. any tasks that are cleared in the group will be recursed 
downstream and everything else even outside the group will also be cleared.
   
   Great callout. I hadn't previously considered the recurse downstream 
functionality. I also wonder if we really need to? For the sake of simplicity, 
we could allow users to choose `recurse downstream` for all involved tasks in 
the group and borrow recursing logic from the involved tasks. This would mean 
that if a user selects a "retry only the failed tasks" strategy, they wouldn't 
need to retry tasks that are downstream of already successful tasks within the 
task group.
   
   > Indeed thinking this over it crossed my mind that perhaps what we want is 
to reimplement subdag as an inline construct, something different from task 
groups.
   
   Personally, I'd be against this, especially given that we have [discouraged 
users from using subdags due to performance 
issues](https://airflow.apache.org/docs/apache-airflow/2.7.0/core-concepts/dags.html#taskgroups-vs-subdags).
 I think we should look into extending task groups and make them more 
functional, rather than reverting to extend subdags or reimplement them. Both 
are quite similar concepts, we should adopt one and enrich it, so that users 
don't need to read yet another documentation page on when to use what.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to