Thanks for the insight! It seems very similar to what I have been trying to do :) Wonook
2017-04-13 16:20 GMT+09:00 Robert Bradshaw <[email protected]>: > There is no way (short of inspecting stack traces and bytecodes) for > Beam to distinguish between > > for (int i=0; i<3; i++) { > pc = pc.apply(MyTransform()); > } > > from > > pc.apply(MyTransform()).apply(MyTransform()).apply(MyTransform()); > > However, PTransforms are hierarchal, so you could put the entire loop > inside an enclosing PTransform which would be good practice. > https://cloud.google.com/dataflow/model/composite-transforms (in Beam > one overrides the expand method; but > https://issues.apache.org/jira/browse/BEAM-1452 ). The "tagging" that > is done is that the full name of the inner transforms are all prefixed > with the name of the outer transform. > > > On Wed, Apr 12, 2017 at 11:53 PM, 송원욱 <[email protected]> wrote: > > Hi, > > > > I have been using BEAM to express different workloads to run them using > > different runners. While doing this, I have encountered a question > > regarding expression of workloads with multiple iterations of operators > (ML > > applications like ALS, MLR). As far as I know, it is currently not > > supported to express loops with BEAM programs, so they are simply > > 'unrolled' to be expressed as an acyclic graph. However, would it be > > possible to recognize operators that are inside loops from other > operators > > that are outside of those loops? If not, would it be possible to tag them > > somehow to express them from others? Any other suggestions or plans for > > carrying out the approach are welcome. > > > > Thanks, > > Wonook >
