(replying from my other account, apologies) Ultimately, they serve different purposes, but unlike a more general runner, they aren't too hard to implement. Robert's comment saying he's written it a few times is illustrative of that
The dot runner for the Go SDK is old, but illustrative. Updating it provides the implementer some familiarity with the Beam Pipeline proto structures, used throughout, but in particular the Prism runner. The Go dot runner it's largely something that can be statically real run and *doesn't* execute the pipeline. The same can be said for the python "render" runner though. But that's not likely to be hooked up for non-python use, just as Go's dot runner isn't standalone either. By contrast, Prism's already been made available to the Java, Python, and Go SDKs and can already be used as a standalone static binary to execute pipelines, and view the status of the those pipelines using a web UI. It's UI doesn't show the pipeline's structure very well though. So implementing a pluggable renderer in Go from the pipeline proto would enable a richer local developer experience, and not simply require us to use Dot, if there's something better or more appropriately dynamic that's developed in the future. Though I'm admittedly biased, IMO we should converge to upgrading prism with a better UI at least. Updating the Go SDK's dot runner is just a means to that end. Finally, the proposed implementation plan at least has a few Milestones baked in. Even if we don't get a fully dynamic rendering in the end, there are some nice artifacts along the way. Robert Burke On Tue, Feb 24, 2026, 11:59 AM Joey Tran <[email protected]> wrote: > Do we want to have multiple portable dot runners? Or would it make sense > to converge on one? > > On Tue, Feb 24, 2026 at 2:47 PM Yousuf Ansari <[email protected]> > wrote: > >> Hi Robert, >> >> That's a very helpful context, especially the point about choosing when >> to collapse composites versus expanding their internals. >> >> I agree that this is really the core usability question, and that >> interactivity is a practical way to avoid hardcoding a single expansion >> strategy. I’ll think more carefully about how the Go implementation can >> support selective expansion rather than enforcing a fixed view. >> >> Appreciate the insight. >> >> On Wed, 25 Feb 2026 at 01:02, Robert Bradshaw via dev < >> [email protected]> wrote: >> >>> I wrote a pipeline proto -> dot graph loop at least a dozen times >>> before I finally checked in that dot "runner." The key interesting bit >>> is figuring out at what point to render a composite as a single node >>> vs. render its internals to be the most useful to the user. >>> Interactivity is a good cop-out for that generically unanswerable >>> question. >>> >>> On Tue, Feb 24, 2026 at 11:02 AM Yousuf Ansari >>> <[email protected]> wrote: >>> > >>> > Thank you both for the helpful feedback. >>> > >>> > Robert - that makes sense about avoiding premature abstraction. I’ll >>> focus on getting a clean portable-proto-based DOT implementation working >>> first and let the abstraction evolve naturally as additional renderers are >>> introduced. >>> > >>> > Joey - thanks for pointing out the Python implementation. I’ll review >>> that to better understand how the proto traversal is structured there and >>> see what patterns can inform the Go SDK approach. >>> > >>> > Appreciate the guidance. >>> > >>> > Best, >>> > Yousuf >>> > >>> > On Wed, 25 Feb 2026 at 00:11, Joey Tran <[email protected]> >>> wrote: >>> >> >>> >> FWIW, there's already a python-implemented "dot runner" >>> >> >>> >> >>> https://github.com/apache/beam/blob/eac11fed1d45bdfd50c9499b8c4a75f36dfbdfe7/sdks/python/apache_beam/runners/render.py >>> >> >>> >> It can generate dot graphs from a portable pipeline protos already. >>> >> >>> >> On Tue, Feb 24, 2026 at 1:36 PM Robert Burke <[email protected]> >>> wrote: >>> >>> >>> >>> That's aligned with what I was thinking. >>> >>> >>> >>> I would caution against in premature-abstraction of course. Since >>> this is all iterable locally, it's easy enough to first target Dot with the >>> initial abstraction, get that working then try to get a 2nd working, >>> allowing the abstraction to evolve to support both, and not be overly >>> coupled to either. >>> >>> >>> >>> Robert Burke >>> >>> >>> >>> On Tue, Feb 24, 2026 at 10:28 AM Yousuf Ansari < >>> [email protected]> wrote: >>> >>>> >>> >>>> Subject: Re: [GSoC 2026] Go SDK Project Idea – Pipeline >>> Visualization in Prism Runner >>> >>>> >>> >>>> Hi Robert, >>> >>>> >>> >>>> Thank you for the clarification, that perspective helps a lot. >>> >>>> >>> >>>> I agree that the real value is introducing a clean abstraction >>> between the portable pipeline proto and the rendering layer, rather than >>> coupling directly to DOT. >>> >>>> >>> >>>> As a next step, I’m planning to explore defining an intermediate >>> graph representation derived from the portable proto, so that DOT becomes >>> one pluggable renderer, with room for future formats and Prism integration. >>> >>>> >>> >>>> Does that direction sound reasonable from a package/layout >>> perspective within the Go SDK? >>> >>>> >>> >>>> Thanks again for the guidance. >>> >>>> >>> >>>> Best, >>> >>>> Yousuf >>> >>>> >>> >>>> On Tue, 24 Feb 2026 at 23:32, Robert Burke <[email protected]> >>> wrote: >>> >>>>> >>> >>>>> As a rule, I'm in favour of this project, but outside of code >>> reviews I'm not sure how much time I can spend on mentoring beyond that. >>> (But I promise I'll look at all PRs tagged with @lostluck.) >>> >>>>> >>> >>>>> While the proprosal around "dot" is largely just a continuation of >>> the pre-existing "dot" runner, the important thing is that it gives a basic >>> scaffolding of going from something that currently works (Go pipeline to >>> some other format for rendering), not the specific outcome (dot graphs). >>> It's however a useful starting point for later incorporating the same code >>> into Prism. >>> >>>>> >>> >>>>> The ideal case is using the dot runner as the initial proof of >>> concept for converting it to use the portable pipeline proto format instead >>> of the existing Raw Go SDK structure. Then somehow from there abstract >>> what it produces from dot graphs, to some other format (mermaid flow >>> charts, some fun HTML thing directly that can be made responsive in the UI >>> with live metrics and the like, etc.). >>> >>>>> >>> >>>>> Incorporating it into the prism runner enables Python and Java (or >>> other SDKs) to take advantage of it as well, which is where the value of >>> this proposal is. >>> >>>>> >>> >>>>> Robert >>> >>>>> >>> >>>>> On Mon, Feb 23, 2026 at 12:18 AM Yousuf Ansari < >>> [email protected]> wrote: >>> >>>>>> >>> >>>>>> Hi Beam community, >>> >>>>>> >>> >>>>>> My name is Yousuf. I'm a student who has been contributing to >>> Apache projects for a while now - I have 4 merged PRs in Apache Superset >>> and recently opened PR #37673 in Beam, which rewrites the Go SDK dot runner >>> to generate DOT graphs from the portable pipeline proto. >>> >>>>>> >>> >>>>>> The PR notes that the new portable-proto-based approach enables >>> future reuse in Prism and other portable tooling. I'd like to propose a Go >>> SDK GSoC project built around that direction: >>> >>>>>> >>> >>>>>> Integrating the portable-proto DOT generation into the Prism >>> runner's web UI, so developers can visualize their pipeline while it runs >>> locally >>> >>>>>> Adding per-transform metrics display on the graph nodes >>> >>>>>> Potentially surfacing watermark progress per PCollection for >>> streaming pipelines >>> >>>>>> >>> >>>>>> This feels like a natural 350-hour project that directly extends >>> the work already started in #37673 and aligns with the Go SDK roadmap's >>> focus on streaming features and portable runner improvements. >>> >>>>>> >>> >>>>>> I'd love to know: >>> >>>>>> >>> >>>>>> Is there mentor interest in a Go SDK project for GSoC 2026? >>> >>>>>> Does this direction align with the priorities you have in mind? >>> >>>>>> Are there related issues I should pick up to strengthen my >>> proposal before the application window opens? >>> >>>>>> >>> >>>>>> Happy to discuss scope and adjust based on your feedback. >>> >>>>>> >>> >>>>>> Thanks, Yousuf >>> >>
