Thanks for starting this work Pablo. Based on an offline discussion, the performance advantages of end-to-end Rust pipelines that use widely used connectors such as BQ and Kafka seem impressive compared to Java/Python SDKs. I think there are two ways we can make this work available to users to significantly boost the performance of their Beam pipelines.
(1) Author the pipeline using Rust SDK. This will potentially give the most performance gain but customers will need to switch the primary language they use to author the pipeline. (2) Use the multi-language pipelines framework [1] to replace Java/Python connectors (or any other transforms that could benefit from a performance boost) underneath without changing the primary SDK. This will allow users to continue using their preferred language to author the pipeline and any custom DoFns but Beam will replace key connectors underneath. We might want to introduce an option for customers to express willingness for this optimization to occur ( for example, an option *rust_optimized *in the Managed I/O API [2]). We will also need to develop and release SDK harness containers that execute Rust. I'm supportive of pursuing both avenues :) Thanks, Cham [1] https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines [2] https://beam.apache.org/documentation/io/managed-io/ On Wed, Oct 14, 2026 at 7:36 AM Kenneth Knowles <[email protected]> wrote: > I am really excited about this Rust SDK. We've had people working it here > and there for years and it finally looks like it is going to be ready. I'm > very impressed with the level of functionality it is at already! > > Kenn > > On Wed, Oct 14, 2026 at 3:43 AM pablo rodriguez defino < > [email protected]> wrote: > >> If the community decides to move forward with the proposal and the SDK >> reaches a stable status, then eventually, yes. >> >> On Tue, Oct 13, 2026 at 10:58 PM Ganesh Sivakumar <[email protected]> >> wrote: >> >>> Hi Pablo, >>> >>> Thanks for sharing this proposal, it's satisfying to see Beam pipelines >>> written in Rust. Any plans for publishing Beam Rust SDK crates to >>> crates.io? and as I am also working on a Rust Runner, I'd feel >>> comfortable contributing to SDK as well. >>> >>> Thanks, >>> Ganesh >>> >>> >>> >>> From: pablo rodriguez defino <[email protected]> >>> To: <[email protected]> >>> Date: Tue, 13 Oct 2026 07:56:13 +0530 >>> Subject: Fwd: [Request for Feedback] Apache Beam Rust SDK >>> >>> Hi all, >>> >>> I've been working on an Apache Beam SDK in Rust, building on the ideas >>> from the 2023 hackathon and design notes [1, 2], and I wanted to share the >>> design doc and current progress on the implementation with the list: >>> >>> - Design doc >>> >>> <https://docs.google.com/document/d/1M4ZsHNobSt526__rtcwp7K0ULvL9sAKnKBt39CcA7Vw/view> >>> . >>> - Code (branch `feat/rust-sdk` >>> >>> <https://github.com/prodriguezdefino/beam-pabs/tree/feat/rust-sdk/sdks/rust> >>> ). >>> >>> The doc goes through the main design choices (targeting portability with >>> Prism and Dataflow Runner v2, static typing and coders, the `DoFn` trait, >>> and how the worker locates user code) and what is currently implemented and >>> minimally validated on the branch. >>> >>> First of all, thanks for your attention =) I'd love to hear if there is >>> interest in the community in having this contributed to Beam, and get >>> general thoughts on the approach. Section 6 of the doc also lists a few >>> open topics, if anyone wants to chime in: >>> >>> - worker code registration, >>> - `DoFn` ergonomics, >>> - whether it should live in the main repo or a separate one, >>> - and a proposed stacked-PR plan for review. >>> >>> Any comments on the doc or replies on this thread are very welcome! >>> >>> Thanks, >>> Pablo >>> >>> [1] https://lists.apache.org/thread/xg9xq0btp8k1wh2v1gpqyfhwpsyxq4ds >>> [2] https://lists.apache.org/thread/gwnn0l9qq3x8svjbwmsr8cykq8600vhz >>> >>> >>> >>>
