Re: [DISCUSS] Beam pipeline logical and physical DAGs visualization.

2017-08-04 Thread Ben Chambers
I think with GraphViz you can put a label on the "subgraph" that represents each composite. This would allow each leaf transform to be just the name of that leaf within the composite, rather than full name. Is that what you were suggesting Robert? On Thu, Aug 3, 2017 at 10:06 PM Robert Bradshaw w

Re: [DISCUSS] Beam pipeline logical and physical DAGs visualization.

2017-08-03 Thread Robert Bradshaw
Nice. In terms of shared data structures, we have https://github.com/apache/beam/blob/master/sdks/common/runner-api/src/main/proto/beam_runner_api.proto . Presumably a utility that converts this to a dot file would be quite useful. It might be interesting to experiment with different ways of hand

Re: [DISCUSS] Beam pipeline logical and physical DAGs visualization.

2017-08-03 Thread Ahmet Altay
+1, this looks great and it will be very useful for users to understand their pipelines. On Thu, Aug 3, 2017 at 8:25 PM, Pei HE wrote: > Hi all, > While working on JStorm and MapReduce runners, I found that it is very > helpful to understand Beam pipelines by visualizing them. > > Logical graph:

[DISCUSS] Beam pipeline logical and physical DAGs visualization.

2017-08-03 Thread Pei HE
Hi all, While working on JStorm and MapReduce runners, I found that it is very helpful to understand Beam pipelines by visualizing them. Logical graph: https://drive.google.com/file/d/0B6iZ7iRh-LOYc0dUS0Rwb2tvWGM/view?usp=sharing Physical graph: https://drive.google.com/file/d/0B6iZ7iRh-LOYbDFWeD