Welcome! This is very cool to hear about. A major caveat about https://beam.apache.org/contribute/runner-guide/ is that it was written when Beam's portability framework was more of a sketch. The conceptual descriptions are mostly fine, but the pointers to Java helper code will lead you to build a "legacy" runner when it is better to build a portable runner from the start*.
We now have four portable runners in various levels of completeness: Spark, Flink, Samza, and Dataflow. I have added some relevant people to the CC for emphasis. You might also join https://the-asf.slack.com/#beam-portability though I prefer the dev list since it gives visibility to a much greater portion of the community. Kenn *volunteers welcome to update the guide to emphasize portability first *From: *Pulasthi Supun Wickramasinghe <[email protected]> *Date: *Mon, May 13, 2019 at 11:03 AM *To: * <[email protected]> Hi All, > > I am Pulasthi a Ph.D. student at Indiana University. We are planning to > develop a beam runner for our project Twister2 [1] [2]. Twister2 is a big > data framework which supports both batch and stream processing. If you > are interested you can find more information on [2] or read some of our > publications [3] > > I wanted to share our intent and get some guidance from the beam developer > community before starting on the project. I was planning on going through > the code for Apache Spark and Apache Flink runners to get a better > understanding of what I need to do. It would be great if I can get any > pointers on how I should approach this project. I am currently reading > through the runner-guide > <https://beam.apache.org/contribute/runner-guide/>. > > Finally, I assume that I need to create a JIRA issue to track the progress > of this project, right?. I can create the issue but from what I read from > the contribute section I would need some permission to assign it to my > self, I hope someone would be able to help me with that. Looking forward to > working with the Beam community. > > [1] https://github.com/DSC-SPIDAL/twister2 > [2] https://twister2.gitbook.io/twister2/ > [3] https://twister2.gitbook.io/twister2/publications > > Best Regards, > Pulasthi > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 >
