It is usually better to create a single pipeline since you will have better load balancing of work across your different tables and I would expect that the pipeline would finish sooner vs waiting for all the pipelines to finish. Also, different runners will be able to support different pipeline sizes. For example, users have submitted pipelines to Google Cloud Dataflow with upto about 1000 steps.
On Tue, Sep 26, 2017 at 6:56 AM, Chaim Turkel <[email protected]> wrote: > Hi, > I am transforming multiple tables from mongo to bigquery (about 20) > currently i have one pipeline for each table. Each table is a > collection. Is there a limitation for how many collections i can have? > Would it be better to create multiple pipelines? > > > chaim >
