I wrote a pipeline proto -> dot graph loop at least a dozen times before I finally checked in that dot "runner." The key interesting bit is figuring out at what point to render a composite as a single node vs. render its internals to be the most useful to the user. Interactivity is a good cop-out for that generically unanswerable question.
On Tue, Feb 24, 2026 at 11:02 AM Yousuf Ansari <[email protected]> wrote: > > Thank you both for the helpful feedback. > > Robert - that makes sense about avoiding premature abstraction. I’ll focus > on getting a clean portable-proto-based DOT implementation working first and > let the abstraction evolve naturally as additional renderers are introduced. > > Joey - thanks for pointing out the Python implementation. I’ll review that to > better understand how the proto traversal is structured there and see what > patterns can inform the Go SDK approach. > > Appreciate the guidance. > > Best, > Yousuf > > On Wed, 25 Feb 2026 at 00:11, Joey Tran <[email protected]> wrote: >> >> FWIW, there's already a python-implemented "dot runner" >> >> https://github.com/apache/beam/blob/eac11fed1d45bdfd50c9499b8c4a75f36dfbdfe7/sdks/python/apache_beam/runners/render.py >> >> It can generate dot graphs from a portable pipeline protos already. >> >> On Tue, Feb 24, 2026 at 1:36 PM Robert Burke <[email protected]> wrote: >>> >>> That's aligned with what I was thinking. >>> >>> I would caution against in premature-abstraction of course. Since this is >>> all iterable locally, it's easy enough to first target Dot with the initial >>> abstraction, get that working then try to get a 2nd working, allowing the >>> abstraction to evolve to support both, and not be overly coupled to either. >>> >>> Robert Burke >>> >>> On Tue, Feb 24, 2026 at 10:28 AM Yousuf Ansari <[email protected]> >>> wrote: >>>> >>>> Subject: Re: [GSoC 2026] Go SDK Project Idea – Pipeline Visualization in >>>> Prism Runner >>>> >>>> Hi Robert, >>>> >>>> Thank you for the clarification, that perspective helps a lot. >>>> >>>> I agree that the real value is introducing a clean abstraction between the >>>> portable pipeline proto and the rendering layer, rather than coupling >>>> directly to DOT. >>>> >>>> As a next step, I’m planning to explore defining an intermediate graph >>>> representation derived from the portable proto, so that DOT becomes one >>>> pluggable renderer, with room for future formats and Prism integration. >>>> >>>> Does that direction sound reasonable from a package/layout perspective >>>> within the Go SDK? >>>> >>>> Thanks again for the guidance. >>>> >>>> Best, >>>> Yousuf >>>> >>>> On Tue, 24 Feb 2026 at 23:32, Robert Burke <[email protected]> wrote: >>>>> >>>>> As a rule, I'm in favour of this project, but outside of code reviews I'm >>>>> not sure how much time I can spend on mentoring beyond that. (But I >>>>> promise I'll look at all PRs tagged with @lostluck.) >>>>> >>>>> While the proprosal around "dot" is largely just a continuation of the >>>>> pre-existing "dot" runner, the important thing is that it gives a basic >>>>> scaffolding of going from something that currently works (Go pipeline to >>>>> some other format for rendering), not the specific outcome (dot graphs). >>>>> It's however a useful starting point for later incorporating the same >>>>> code into Prism. >>>>> >>>>> The ideal case is using the dot runner as the initial proof of concept >>>>> for converting it to use the portable pipeline proto format instead of >>>>> the existing Raw Go SDK structure. Then somehow from there abstract what >>>>> it produces from dot graphs, to some other format (mermaid flow charts, >>>>> some fun HTML thing directly that can be made responsive in the UI with >>>>> live metrics and the like, etc.). >>>>> >>>>> Incorporating it into the prism runner enables Python and Java (or other >>>>> SDKs) to take advantage of it as well, which is where the value of this >>>>> proposal is. >>>>> >>>>> Robert >>>>> >>>>> On Mon, Feb 23, 2026 at 12:18 AM Yousuf Ansari >>>>> <[email protected]> wrote: >>>>>> >>>>>> Hi Beam community, >>>>>> >>>>>> My name is Yousuf. I'm a student who has been contributing to Apache >>>>>> projects for a while now - I have 4 merged PRs in Apache Superset and >>>>>> recently opened PR #37673 in Beam, which rewrites the Go SDK dot runner >>>>>> to generate DOT graphs from the portable pipeline proto. >>>>>> >>>>>> The PR notes that the new portable-proto-based approach enables future >>>>>> reuse in Prism and other portable tooling. I'd like to propose a Go SDK >>>>>> GSoC project built around that direction: >>>>>> >>>>>> Integrating the portable-proto DOT generation into the Prism runner's >>>>>> web UI, so developers can visualize their pipeline while it runs locally >>>>>> Adding per-transform metrics display on the graph nodes >>>>>> Potentially surfacing watermark progress per PCollection for streaming >>>>>> pipelines >>>>>> >>>>>> This feels like a natural 350-hour project that directly extends the >>>>>> work already started in #37673 and aligns with the Go SDK roadmap's >>>>>> focus on streaming features and portable runner improvements. >>>>>> >>>>>> I'd love to know: >>>>>> >>>>>> Is there mentor interest in a Go SDK project for GSoC 2026? >>>>>> Does this direction align with the priorities you have in mind? >>>>>> Are there related issues I should pick up to strengthen my proposal >>>>>> before the application window opens? >>>>>> >>>>>> Happy to discuss scope and adjust based on your feedback. >>>>>> >>>>>> Thanks, Yousuf
