+1, this looks great and it will be very useful for users to understand their pipelines.
On Thu, Aug 3, 2017 at 8:25 PM, Pei HE <[email protected]> wrote: > Hi all, > While working on JStorm and MapReduce runners, I found that it is very > helpful to understand Beam pipelines by visualizing them. > > Logical graph: > https://drive.google.com/file/d/0B6iZ7iRh-LOYc0dUS0Rwb2tvWGM/view?usp= > sharing > > Physical graph: > https://drive.google.com/file/d/0B6iZ7iRh-LOYbDFWeDlCcDhnQmc/view?usp= > sharing > > I think we can visualize Beam logical DAG in runner-core. It should also be > easy to visualize the physical DAG in each runners. (Maybe we can define > some shared data structures to make it more automatic, and even support > visualizing them in Apex/Flink/Spark/Gearpump UIs). > > I have a commit for MapReduce runner in here (<200 lines). And, this commit > generates dotfiles for logical and physical DAGs. > > https://github.com/peihe/incubator-beam/commit/ > bb3349e10c0cfacd81b610880ddfec030fedf34d > > Looking forward to ideas and feedbacks. > -- > Pei >
