Hi airflow team,
I have a data pipeline as below
task1 --> task2(parameter) --> task3(parameter)
The parameters of task2 and task3 are based on the outputs from task1.
When it runs, the task1 will create a list of data, say list[a, b, c].
Then, I want to process to run as like below. Is it possible to achieve in
airflow ?
Note the list output from task1 is dynamic.
I am thinking to use SubDAG for task2 and task3, but it seems static.
Thanks.
Jason
task1 --> task2(a) --> task3(a)
\ \---> task2(b) --> task3(b)
\-------> task2(c) --> task3(c)
