Hello All,
I noticed that Flink/Spark/Samza runners are translating portable pipeline in
the similar manner:
> QueryablePipeline p =
> QueryablePipeline.forTransforms(
> pipeline.getRootTransformIdsList(), pipeline.getComponents());
>
> for (PipelineNode.PTransformNode transform :
> p.getTopologicallyOrderedTransforms()) {
> // Translation logic
> }
However, IIUC, this only iterates through leaf nodes of the pipeline, i.e.
composite transforms are NOT being translated at all.
Is this the expected behavior for runner to implement translation logic for
portable pipeline? If Yes, what are the suggestions if certain runners need to
translate composite transforms?
Best,
Ke
