VBhojawala opened a new pull request #13479:
URL: https://github.com/apache/airflow/pull/13479
Added taskgroup decorator which can be used to create taskgroup from python
callable. Inside python callable tasks can be grouped by calling tasks (python
callable ) created with task decorator.
- taskgroup decorator takes both optional and keyword argument which are
passed to Constructor of TaskGroup class.
- TaskGroup can be created with one of following syntax
``` python
@taskgroup
@taskgroup()
@taskgroup(group_id='group_name')
```
- TaskGroup class constructor takes one mandatory argument group_id, if not
given in decorator it sets group_id to python callable name.
Following is a simple example demonstrating use of taskgroup decorator
grouping multiple tasks.
``` python
@task
def task_1(value):
return f'[ Task1 {value} ]'
@task
def task_2(value):
print(f'[ Task2 {value} ]')
@taskgroup
def section_1(value):
return task_2(task_1(value))
```
taskgroup decorator utilizes existing TaskGroup context manager currently
used for creating TaskGroup, which means we can create nested taskgroup by
created nested callable. Following is an example demonstrating use of nested
taskgroup.
``` python
@task
def task_start():
return '[Task_start]'
@task
def task_end():
print(f'[ Task_End ]')
@task
def task_1(value):
return f'[ Task1 {value} ]'
@task
def task_2(value):
print(f'[ Task2 {value} ]')
@task
def task_3(value):
return f'[ Task3 {value} ]'
@task
def task_4(value):
print(f'[ Task4 {value} ]')
@taskgroup
def section_1(value):
@taskgroup
def section_2(value2):
return task_4(task_3(value2))
op1 = task_2(task_1(value))
return section_2(op1)
```
Dedicated test cases for taskgroup decorator is created in file
/tests/utils/test_task_group_decorator.py
Recent changes
- Added logic to append suffix to duplicate group_id.
closes: #11870
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
