There is no way (short of inspecting stack traces and bytecodes) for
Beam to distinguish between
for (int i=0; i<3; i++) {
pc = pc.apply(MyTransform());
}
from
pc.apply(MyTransform()).apply(MyTransform()).apply(MyTransform());
However, PTransforms are hierarchal, so you could put the entire loop
inside an enclosing PTransform which would be good practice.
https://cloud.google.com/dataflow/model/composite-transforms (in Beam
one overrides the expand method; but
https://issues.apache.org/jira/browse/BEAM-1452 ). The "tagging" that
is done is that the full name of the inner transforms are all prefixed
with the name of the outer transform.
On Wed, Apr 12, 2017 at 11:53 PM, 송원욱 <[email protected]> wrote:
> Hi,
>
> I have been using BEAM to express different workloads to run them using
> different runners. While doing this, I have encountered a question
> regarding expression of workloads with multiple iterations of operators (ML
> applications like ALS, MLR). As far as I know, it is currently not
> supported to express loops with BEAM programs, so they are simply
> 'unrolled' to be expressed as an acyclic graph. However, would it be
> possible to recognize operators that are inside loops from other operators
> that are outside of those loops? If not, would it be possible to tag them
> somehow to express them from others? Any other suggestions or plans for
> carrying out the approach are welcome.
>
> Thanks,
> Wonook
