Thanks for the excellent article! It's great to see what the experience is like from an outside perspective, and it's comforting that it mirrors my own. It means I'm not missing much.
It's been on my to-do list to make the Go SDK direct runner more robust, so transitioning to other runners wouldn't be such a burden. I'd love for it to have better error messages, and be more useful for testing. Daniel Oliveira recently updated the Universal Runner guide to include how to run Go SDK jobs against it. It should also catch the same issues, and provides a free way to check that pipelines are correct. It has the same single machine limitation though. https://cwiki.apache.org/confluence/display/BEAM/Usage+Guide User Defined Coders and Pointer Elements (and their semantics) is something I've been thinking about as well and will be working on within the next month. JSON is ok for debugging but less so for performance at scale. Let me know if you have any opinions on that! I intend to post to the list about it this week. As for reshuffle, other IOs, and scalability, as mentioned in the roadmap ( https://beam.apache.org/roadmap/go-sdk/) we're mostly blocked on SplittableDoFn support. With it, we wouldn't need to reshuffle, and would gain more natural scaling of IOs. Once the Go SDK havs these it will br well on it's way to not being experimental. :) Finally, I'm obligated to mention that while the SDK works on Dataflow it's not yet officially supported by the service. Cheers Robert B On Mon, Dec 17, 2018, 8:07 AM Kenneth Knowles <[email protected]> wrote: > Nice! > > It reads really well. For the benefit of this list, would you be willing > to summarize the rough edges (and maybe the "couple of other things" you > had to implement) in a few bullet points? and/or file Jira issues if they > are clear feature requests or bugs. > > Kenn > > On Mon, Dec 17, 2018 at 10:40 AM Andrew Brampton <[email protected]> > wrote: > >> Hey all, >> >> I've recently been playing with the Go Beam SDK running on Dataflow. I >> wrote up a tutorial for today's Go Advent blog. >> >> Feel free to check it out: >> https://blog.gopheracademy.com/advent-2018/apache-beam/ >> >> Feedback is welcomed. I know the Go SDK is experimental, but I hit a few >> rough edges. I also had to implement my own csvio, reshuffle, and a couple >> of other things. I will be sharing my feedback on using go and dataflow >> shortly. >> >> Thanks >> Andrew >> >
