Hi Everyone, My name is Ganesh. I've been contributing to the Apache Beam Java SDK, and over the last few months I've been spending most of my time building a new Apache Beam portable runner in Rust. It has now reached a stage of what I'd consider a "minimum viable runner", and I'm very excited to share it with the community.
I previously sent an email about the project, but unfortunately it ended up in Gmail's spam folder for many people and didn't reach most of the mailing list. So I decided to write a fresh email. So, Why new runner?. Beam is an amazing programing model but highly underrated one and doesn't get the marketing it deserves. Current runners are good at execution. But I felt there was a need for different kind of runner, one that natively speaks Apache Beam portability without a translation layer to different engine's execution engine's abstractions, one that's built in a safe, high-performance systems language like Rust, one that runs on modern disaggregated database architecture and that's easy and resource efficient to operate. The project is called FlareDB[1]. I wrote a design doc[2] explaining its architecture and design. I'd like to kindly request feedback/questions or any comments on FlareDB and I really hope it evolves as a Beam community project. P.S: after my previous mail I shipped a CLI for FlareDB to easily start and manage local instances. Please check out readme for usage guide. [1] FlareDB Repo: https://github.com/flare-db/flare-db [2] Design Doc: https://docs.google.com/document/d/17R2iFtHRFkAAwDYNGRddSH7B3U9tpbPGoKKbLSw76yU/edit?usp=sharing Thanks, Ganesh.
