The new lineage functionality (while in very early states) can do this for you.
B. > On 20 Jun 2018, at 19:46, Andrew Harmon <[email protected]> wrote: > > Hello, > > I wanted to submit a feature suggestion. I have a scenario where I'd like > to create a subdag where the number of tasks in it are dependent on a > previous task. Ideally, this parameter could be passed in via xcom. For > example: > > Task A does something that generates 10 files. Task B is a subdag that at > runtime, creates 10 tasks within the sub dag. Each task will process one of > the 10 files. That way the processing of the 10 files can run in parallel. > Task C will then start when all tasks in the subdag are complete. > > My particular scenario is to process the results of a BigQuery export to > GCS. The export creates n number of files and you don't know that number > until the export is done running. > > Thanks, > Andrew > -- > Andrew Harmon > (202) 615-6433
