casassg opened a new issue #11870:
URL: https://github.com/apache/airflow/issues/11870
**Description**
Similarly to `@dag` and `@task` decorators, we can add a decorator to easily
generate `TaskGroup` instances.
**Use case / motivation**
This would be used for reusable pieces of DAGs. An example may be to
hyperparameter tune an ML pipeline. You may want to generate several parallel
subpipelines with different settings/parameters to be executed in parallel. At
the end pull the result and decide the best parametrization.
```py
@task
collect_dataset(...)
pass
@task
def train_model(...)
pass
@task_group
def movielens_model_pipeline(learning_rate: int, feature_list: list,
dataset: XComArg):
dataset = filter_dataset(dataset, feature_list)
train_model(dataset, learning_rate)
@dag
def movielens_hpt(dataset_path: str, feature_list:list=['year', 'director']):
dataset = load_dataset(dataset_path)
for i in range(0.1, 0.9, step=0.1):
movielens_model_pipeline(i, feature_list, dataset)
decide_best_model(...)
```
**Related Issues**
https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3AAIP-31
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
