Max is right, it isn't generally possible to upgrade Beam version for scio. Scio doesn't use much of Beam internals, but Beam API isn't guaranteed to be binary compatible in a minor release. There is a small chance that it might work.
I would recommend using the latest version 0.7.0-beta3 built for Beam 2.9.0. On Wed, Jan 9, 2019 at 4:49 PM Maximilian Michels <[email protected]> wrote: > @Gleb Thanks for pointing out some of the features unique to Scio! Have > yet to > try it out, but I see many Beam folks liking it a lot. > > @Jan I think the Beam version is hardcoded in Scio because Scio is > essentially a > layer on top of Beam and some of its internals. While the public API does > not > change much, some of the internals do and thus you can't freely choose the > Beam > version. Perhaps Gleb can give more insight? > > I think it would be fine to add your project to the page. We don't have > strict > requirements for the level of maturity of the project. Feel free to open a > PR. > > Let's have a chat at Fosdem! :) > > Cheers, > Max > > On 07.01.19 16:43, [email protected] wrote: > > Hi all, > > > > I think I'm going to stick with scala and scio :-). > > > > I'm curious though: why is there a hard coupling between scio and beam > versions? > > I was hoping to use latest scio 0.7.0-beta2 with beam 2.9.0 but that > appears to > > get blocked, which was unexpected to me. > > > > Regarding the suggestion to add my project to this page: I'm flattered, > but it's > > all still very early and prototype like... > > > > Btw, Max, I was wondering where I heard your name before. Apparently > it's > > because I was planning to go to your fosdem talk. > > So if you or anyone else want to have a chat, I should probably be there > on > > sunday :-). > > > > Kind regards, > > Jan > > > > > > On Mon, 7 Jan 2019 at 17:13, Gleb Kanterov <[email protected] > > <mailto:[email protected]>> wrote: > > > > Agree with Max that scio is lagging behind. However, it also has > features > > that significantly reduce boilerplate, and even improve performance. > For > > instance, the latest version (0.7.0) automatically derives binary > coders for > > case classes using macro at compile-time, that is a way better > > than Java/Kryo serialization. > > > > Jan, if you find that you are missing features, or have general > feedback, > > you are always welcome to create issues or pull requests in > spotify/scio > > <https://github.com/spotify/scio> repository :). > > > > Gleb > > > > On Mon, Jan 7, 2019 at 4:57 PM Maximilian Michels <[email protected] > > <mailto:[email protected]>> wrote: > > > > Interesting project, Jan! I think we could add your project to > this page: > > https://beam.apache.org/community/integrations/ > > > > The benefit of using the Java DSL would be to be able to > directly track > > Beam. > > The Scio Scala DSL usually lags a bit behind. But since you > probably don't > > require the latest features and Scala is more enjoyable for you, > I think > > your > > current design choice is sensible. > > > > Best, > > Max > > > > On 04.01.19 16:22, Kenneth Knowles wrote: > > > Scio is a Scala wrapper on top of Beam's Java SDK. So it still > > benefits from the > > > maturity of Beam Java in terms of performance and > reliability. Using > > Scala will > > > definitely be less verbose than Java. You can use Scio or the > Java > > SDK directly > > > with Scala's support for calling Java libraries. > > > > > > Kenn > > > > > > On Sun, Dec 30, 2018 at 12:38 PM <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > > > Hi Chak, > > > > > > I'm not sure if it's the correct decision... > > > To be completely honest, the first iterations (which I > haven't > > made public > > > so far) were actually in java. > > > However I find java to be a bit verbose for my taste. > > > The past 5 years I've worked in OCaml, and despite the > lacking > > > tooling/ecosystem I really liked the language. It's really > > expressive. > > > > > > Scala has a similar feel as OCaml, which is why I want to > pick it > > up, and > > > thus why I experimented with it on this project. > > > > > > But if most people who would want to contribute would > only do so > > in a java > > > codebase, then I don't mind continuing this in java. > > > > > > Kind regards, > > > Jan > > > > > > > > > On Sun, 30 Dec 2018 at 21:14, Chak-Pong Chung > > <[email protected] <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > > > Hi Jan, > > > > > > This is quite interesting. As far as I know, Beam and > Flink > > have more > > > mature and stable API in Java. What is the motivation > here to use > > > scala/scio in your project? > > > > > > Kind regards, > > > Chak > > > > > > On Sun, Dec 30, 2018 at 1:02 PM <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > > > Hi all, > > > > > > I figured out how to build deterministic > transaction > > processing on > > > top of apache beam/flink. > > > > > > https://domsj.info/2018/12/30/introducing-streamy-db.html > > > https://github.com/domsj/streamy-db > > > > > > I can use some help, please join me! > > > > > > Kind regards, > > > Jan > > > > > > > > > > > -- > > Cheers, > > Gleb > > > -- Cheers, Gleb
