shubham22 commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1518474712
Hi everyone! I'm a product manager, and I happened to come across this open
issue. It seems very useful feature, and I thought it would be a great addition
to our backlog. To help move things along, I've put together a set of
requirements for this feature. I'd love for you to take a look and share any
thoughts or concerns you might have. If anyone is available and interested in
implementing this, please feel free to use the requirements as a starting point.
**Requirements:**
* Add a retry parameter at the TaskGroup level, allowing users to specify
the number of retries for the entire TaskGroup (`retries`).
* Implement a mechanism for users to determine when a TaskGroup is
considered failed based on their specific use case (`retry_condition`):
* (default) retrying when any task within the group fails.
* users should be able to define other retry conditions, including retry
when the last task within the group fails.
* Support 2 pre-defined strategies for handling TaskGroup retries
(`retry_strategy`):
* (default) strategy `all_tasks`: clearing the previous task(s) and
retry all of them.
* strategy `failed_tasks`: retrying only the failed task(s) within the
group.
Example 1: Task group with 3 tasks that is retried when any task fails and
retries only failed tasks
```python
@task_group (group_id="tasks_with_retries", retries=3,
retry_strategy="failed_tasks")
def task_group_example():
task_a()
task_b()
task_c()
```
Example 2: Task group with 3 tasks that is retried only when task_b fails
and retries all tasks
```python
@task_group(group_id="tasks_with_retries", retries=3,
retry_condition=custom_condition)
def task_group_example():
task_a()
task_b()
task_c()
def custom_condition(task_group: TaskGroup) -> bool:
task_b_instance = next(task for task in task_group if task.task_id ==
"task_b")
return task_b_instance.state == State.FAILED
```
Note: code snippets are only meant as a rough example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
