You should be able to provide a fixed number of iterations with a for like loop that is "unrolled" constructing a larger pipeline which will allow you to submit the pipeline less often.
On Wed, Jun 22, 2016 at 1:10 PM, Frances Perry <f...@google.com> wrote: > Correct -- Beam pipelines currently have to be DAGs currently and can't > support cycles. > > There's a jira to look into iterations ( > https://issues.apache.org/jira/browse/BEAM-106). The semantics get a > little tricky given the unified model, so there's still quite a lot to > figure out. > > In the meantime, you can maintain the loop outside your pipeline -- > essentially creating a new pipeline for each iteration. > > > > On Wed, Jun 22, 2016 at 11:36 AM, Eswar Reddy <eswarareddyad...@gmail.com> > wrote: > >> Hi Beam Users, >> >> For my future projects, I anticipate having to create pipelines having >> cycle(s).Could see some engines(Flink >> <https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/index.html#iterations> >> and Storm >> <https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8>) >> support this, they also provide flexibility to the user to ensure the loop >> ends for a given message by splitting & filtering streams. Could someone >> share any pointers to analogus of this in Beam Model? I did see that >> documentation >> <https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/Pipeline#class-pipeline> >> says Pipeline is always a DAG though. >> >> Thanks, >> Eswar. >> > >
