Hi Robert, That's a very helpful context, especially the point about choosing when to collapse composites versus expanding their internals.
I agree that this is really the core usability question, and that interactivity is a practical way to avoid hardcoding a single expansion strategy. I’ll think more carefully about how the Go implementation can support selective expansion rather than enforcing a fixed view. Appreciate the insight. On Wed, 25 Feb 2026 at 01:02, Robert Bradshaw via dev <[email protected]> wrote: > I wrote a pipeline proto -> dot graph loop at least a dozen times > before I finally checked in that dot "runner." The key interesting bit > is figuring out at what point to render a composite as a single node > vs. render its internals to be the most useful to the user. > Interactivity is a good cop-out for that generically unanswerable > question. > > On Tue, Feb 24, 2026 at 11:02 AM Yousuf Ansari > <[email protected]> wrote: > > > > Thank you both for the helpful feedback. > > > > Robert - that makes sense about avoiding premature abstraction. I’ll > focus on getting a clean portable-proto-based DOT implementation working > first and let the abstraction evolve naturally as additional renderers are > introduced. > > > > Joey - thanks for pointing out the Python implementation. I’ll review > that to better understand how the proto traversal is structured there and > see what patterns can inform the Go SDK approach. > > > > Appreciate the guidance. > > > > Best, > > Yousuf > > > > On Wed, 25 Feb 2026 at 00:11, Joey Tran <[email protected]> > wrote: > >> > >> FWIW, there's already a python-implemented "dot runner" > >> > >> > https://github.com/apache/beam/blob/eac11fed1d45bdfd50c9499b8c4a75f36dfbdfe7/sdks/python/apache_beam/runners/render.py > >> > >> It can generate dot graphs from a portable pipeline protos already. > >> > >> On Tue, Feb 24, 2026 at 1:36 PM Robert Burke <[email protected]> > wrote: > >>> > >>> That's aligned with what I was thinking. > >>> > >>> I would caution against in premature-abstraction of course. Since this > is all iterable locally, it's easy enough to first target Dot with the > initial abstraction, get that working then try to get a 2nd working, > allowing the abstraction to evolve to support both, and not be overly > coupled to either. > >>> > >>> Robert Burke > >>> > >>> On Tue, Feb 24, 2026 at 10:28 AM Yousuf Ansari < > [email protected]> wrote: > >>>> > >>>> Subject: Re: [GSoC 2026] Go SDK Project Idea – Pipeline Visualization > in Prism Runner > >>>> > >>>> Hi Robert, > >>>> > >>>> Thank you for the clarification, that perspective helps a lot. > >>>> > >>>> I agree that the real value is introducing a clean abstraction > between the portable pipeline proto and the rendering layer, rather than > coupling directly to DOT. > >>>> > >>>> As a next step, I’m planning to explore defining an intermediate > graph representation derived from the portable proto, so that DOT becomes > one pluggable renderer, with room for future formats and Prism integration. > >>>> > >>>> Does that direction sound reasonable from a package/layout > perspective within the Go SDK? > >>>> > >>>> Thanks again for the guidance. > >>>> > >>>> Best, > >>>> Yousuf > >>>> > >>>> On Tue, 24 Feb 2026 at 23:32, Robert Burke <[email protected]> > wrote: > >>>>> > >>>>> As a rule, I'm in favour of this project, but outside of code > reviews I'm not sure how much time I can spend on mentoring beyond that. > (But I promise I'll look at all PRs tagged with @lostluck.) > >>>>> > >>>>> While the proprosal around "dot" is largely just a continuation of > the pre-existing "dot" runner, the important thing is that it gives a basic > scaffolding of going from something that currently works (Go pipeline to > some other format for rendering), not the specific outcome (dot graphs). > It's however a useful starting point for later incorporating the same code > into Prism. > >>>>> > >>>>> The ideal case is using the dot runner as the initial proof of > concept for converting it to use the portable pipeline proto format instead > of the existing Raw Go SDK structure. Then somehow from there abstract > what it produces from dot graphs, to some other format (mermaid flow > charts, some fun HTML thing directly that can be made responsive in the UI > with live metrics and the like, etc.). > >>>>> > >>>>> Incorporating it into the prism runner enables Python and Java (or > other SDKs) to take advantage of it as well, which is where the value of > this proposal is. > >>>>> > >>>>> Robert > >>>>> > >>>>> On Mon, Feb 23, 2026 at 12:18 AM Yousuf Ansari < > [email protected]> wrote: > >>>>>> > >>>>>> Hi Beam community, > >>>>>> > >>>>>> My name is Yousuf. I'm a student who has been contributing to > Apache projects for a while now - I have 4 merged PRs in Apache Superset > and recently opened PR #37673 in Beam, which rewrites the Go SDK dot runner > to generate DOT graphs from the portable pipeline proto. > >>>>>> > >>>>>> The PR notes that the new portable-proto-based approach enables > future reuse in Prism and other portable tooling. I'd like to propose a Go > SDK GSoC project built around that direction: > >>>>>> > >>>>>> Integrating the portable-proto DOT generation into the Prism > runner's web UI, so developers can visualize their pipeline while it runs > locally > >>>>>> Adding per-transform metrics display on the graph nodes > >>>>>> Potentially surfacing watermark progress per PCollection for > streaming pipelines > >>>>>> > >>>>>> This feels like a natural 350-hour project that directly extends > the work already started in #37673 and aligns with the Go SDK roadmap's > focus on streaming features and portable runner improvements. > >>>>>> > >>>>>> I'd love to know: > >>>>>> > >>>>>> Is there mentor interest in a Go SDK project for GSoC 2026? > >>>>>> Does this direction align with the priorities you have in mind? > >>>>>> Are there related issues I should pick up to strengthen my proposal > before the application window opens? > >>>>>> > >>>>>> Happy to discuss scope and adjust based on your feedback. > >>>>>> > >>>>>> Thanks, Yousuf >
