gaibelg opened a new issue, #53557:
URL: https://github.com/apache/airflow/issues/53557
### Description
Support compacted inputs for dynamic mapping over task groups, thus allowing
a better support for depth-first execution with tasks that require many inputs.
### Use case/motivation
Hi!
Depth-first execution allows to create dependence between mapped instances
of different tasks. It currently has good support for use cases where the
inner-tasks within the task group require very limited amount of inputs. For
example:
```
@task
def add1(x):
return x+1
@task
def multiply(x,y)
return x*y
@task_group
def depth_first_example(xm1, y):
x = add1(xm1)
res = multiply(x,y)
depth_first_example.expand_kwargs([{'x':2, 'y':2}, {'x':3, 'y':2}])
```
However, suppose I want to trigger two tasks that each requires many inputs.
For example - trigger two or more KubernetesPodOperator tasks, where each of
these tasks requires a unique set of params (image, cmds, resources,
service_account, namesapce, etc.). Currently, the only way I found to make this
work was to explicitly provide all of the params to the task group. This then
makes the signature of the task group extremely large, and difficult to handle
and maintain.
for example:
```
@task_group
def depth_first_horror(image0, cmds0, resources0, account0, namespace0,
image1, cmds1, resources1, account1, namespace1):
opr0 = K8Operator(image=image0, cmds=cmds0,...)
opr1 = K8Operator(image=image1, cmds=cmds1,...)
opr0 >> opr1
```
Is there a way to support a more compact way of providing params, like
extending MappedArgument such that it can be expanded with `**` operator?
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
