freephys opened a new issue #11224:
URL: https://github.com/apache/airflow/issues/11224
Hi,
I am looking for a way to achieve this inside airflow, hopefully some easy
demo code to study with. Start with several jobs with say 50 parameters, then
at some point (After finishing task A->B) , I wanna to run task C in parallel
with different values of say 5 parameters(those 5 parameters will try different
values) , after running task C in parallel, all results are passed onto task D
and E with same parameters. The dag id etc are needed to be generated
dynamically.
Any idea about if this can be done with current airflow or a feature to be
ask?
The advantage will be saving time on task A and B and try task C with
several different parameters so we can pick best one based on the results of
task D
thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
