@Gleb Thanks for pointing out some of the features unique to Scio! Have yet to
try it out, but I see many Beam folks liking it a lot.
@Jan I think the Beam version is hardcoded in Scio because Scio is essentially a
layer on top of Beam and some of its internals. While the public API does not
change much, some of the internals do and thus you can't freely choose the Beam
version. Perhaps Gleb can give more insight?
I think it would be fine to add your project to the page. We don't have strict
requirements for the level of maturity of the project. Feel free to open a PR.
Let's have a chat at Fosdem! :)
Cheers,
Max
On 07.01.19 16:43, [email protected] wrote:
Hi all,
I think I'm going to stick with scala and scio :-).
I'm curious though: why is there a hard coupling between scio and beam versions?
I was hoping to use latest scio 0.7.0-beta2 with beam 2.9.0 but that appears to
get blocked, which was unexpected to me.
Regarding the suggestion to add my project to this page: I'm flattered, but it's
all still very early and prototype like...
Btw, Max, I was wondering where I heard your name before. Apparently it's
because I was planning to go to your fosdem talk.
So if you or anyone else want to have a chat, I should probably be there on
sunday :-).
Kind regards,
Jan
On Mon, 7 Jan 2019 at 17:13, Gleb Kanterov <[email protected]
<mailto:[email protected]>> wrote:
Agree with Max that scio is lagging behind. However, it also has features
that significantly reduce boilerplate, and even improve performance. For
instance, the latest version (0.7.0) automatically derives binary coders for
case classes using macro at compile-time, that is a way better
than Java/Kryo serialization.
Jan, if you find that you are missing features, or have general feedback,
you are always welcome to create issues or pull requests in spotify/scio
<https://github.com/spotify/scio> repository :).
Gleb
On Mon, Jan 7, 2019 at 4:57 PM Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Interesting project, Jan! I think we could add your project to this
page:
https://beam.apache.org/community/integrations/
The benefit of using the Java DSL would be to be able to directly track
Beam.
The Scio Scala DSL usually lags a bit behind. But since you probably
don't
require the latest features and Scala is more enjoyable for you, I think
your
current design choice is sensible.
Best,
Max
On 04.01.19 16:22, Kenneth Knowles wrote:
> Scio is a Scala wrapper on top of Beam's Java SDK. So it still
benefits from the
> maturity of Beam Java in terms of performance and reliability. Using
Scala will
> definitely be less verbose than Java. You can use Scio or the Java
SDK directly
> with Scala's support for calling Java libraries.
>
> Kenn
>
> On Sun, Dec 30, 2018 at 12:38 PM <[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> Hi Chak,
>
> I'm not sure if it's the correct decision...
> To be completely honest, the first iterations (which I haven't
made public
> so far) were actually in java.
> However I find java to be a bit verbose for my taste.
> The past 5 years I've worked in OCaml, and despite the lacking
> tooling/ecosystem I really liked the language. It's really
expressive.
>
> Scala has a similar feel as OCaml, which is why I want to pick it
up, and
> thus why I experimented with it on this project.
>
> But if most people who would want to contribute would only do so
in a java
> codebase, then I don't mind continuing this in java.
>
> Kind regards,
> Jan
>
>
> On Sun, 30 Dec 2018 at 21:14, Chak-Pong Chung
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> Hi Jan,
>
> This is quite interesting. As far as I know, Beam and Flink
have more
> mature and stable API in Java. What is the motivation here
to use
> scala/scio in your project?
>
> Kind regards,
> Chak
>
> On Sun, Dec 30, 2018 at 1:02 PM <[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>
wrote:
>
> Hi all,
>
> I figured out how to build deterministic transaction
processing on
> top of apache beam/flink.
>
> https://domsj.info/2018/12/30/introducing-streamy-db.html
> https://github.com/domsj/streamy-db
>
> I can use some help, please join me!
>
> Kind regards,
> Jan
>
--
Cheers,
Gleb
