[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420307#comment-16420307 ] Jozef Vilcek commented on BEAM-302: --- Hello, I do not want to bug you, but is there an update on this JIRA? I get that we are all busy, therefore not looking for an exact ETA. I am looking for an information how likely it is that Beam will get a Scala friendly API in near future. [~sinisa_lyh], is integrating a Scio into Beam still the plan? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356926#comment-16356926 ] Jean-Baptiste Onofré commented on BEAM-302: --- +1 ! [~sinisa_lyh] any chance we can discuss about this a new time, in regards of latest change in Beam ? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356923#comment-16356923 ] Jozef Vilcek commented on BEAM-302: --- BEAM-1920 is resolved. Looking forward to see next steps of integration Scio with Beam > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996851#comment-15996851 ] Chaoran Yu commented on BEAM-302: - Thanks for the update. Beam is updating its Spark runner to 2.1 built with Scala 2.11 as well. https://issues.apache.org/jira/browse/BEAM-1920 I'll test scio integration with Spark runner once that jira issue is done. > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996831#comment-15996831 ] Neville Li commented on BEAM-302: - Yes that ecosystem has too many build params, scala version, spark version, hadoop version, etc. 2.10 is outdated, quite different from 2.11/2.12 and hard to maintain. That's why we stopped supporting it. 2.12 support should be available soon once some compiler lambda serialization issues are addressed. > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995814#comment-15995814 ] Chaoran Yu commented on BEAM-302: - Thanks Neville for the information! I tried Spark 1.6.3 but it had Scala incompatibilities with Scio. The version of Spark 1.6.3 included in Beam Spark runner is compiled using Scala 2.10, but Scio is compiled using Scala 2.11. I had to change a few other dependencies to 2.10-compiled version such as https://mvnrepository.com/artifact/me.lyh/protobuf-generic_2.10, but still got errors. My team and myself will be thinking about contributing to get Spark runner to fully work with scio when we can spend more time on the project. > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995075#comment-15995075 ] Neville Li commented on BEAM-302: - Looks like Spark runner still depends on 1.6.3. Can you give Spark 1.6 a shot instead? https://mvnrepository.com/artifact/org.apache.beam/beam-runners-spark/0.6.0 We'd love to support all runners but we use Dataflow runner only and vanilla Spark. Contributions will be awesome and are definitely welcome. Feel free to submit issues or PRs on our GH repo. There's also a GItter room and a Google group for discussions. https://github.com/spotify/scio > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994104#comment-15994104 ] Chaoran Yu commented on BEAM-302: - After including SparkRunner dependency, I got the following exception "java.lang.NoClassDefFoundError: org/apache/spark/streaming/api/java/JavaStreamingContextFactory", which is a class that has ceased to exist for Spark 2.0+. I'm sure by digging deeper, the word count example can be run with SparkRunner. But my idea is that currently it's still not as simple as it should be. Spark runner, or basically most other runners supported by Beam, should receive the same support from scio as is the case for GCD runner. Is this an aspect where Spotify would like to see contributions from the community? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994005#comment-15994005 ] Neville Li commented on BEAM-302: - You need the spark runner dependency which is not included by default. > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993999#comment-15993999 ] Chaoran Yu commented on BEAM-302: - [~neville] I tried the latest SNAPSHOT version of scio but it's still not working with Spark runner out of the box. For example, running sbt "scio-examples/run-main com.spotify.scio.examples.WordCount --runner=SparkRunner --input=README.md --output=wc" gave me the following exception: java.lang.IllegalArgumentException: Unknown 'runner' specified 'SparkRunner', supported pipeline runners [DataflowRunner, DirectRunner] Caused by: java.lang.ClassNotFoundException: SparkRunner The same thing happened in the scio repl. Looking at the code, more things need to be done to integrate with Spark/Flink etc. > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953584#comment-15953584 ] Neville Li commented on BEAM-302: - We prefer to keep it separate for now mainly for logistics reasons: - we use SBT with lots of custom logic - we release very often, once every 1-2 weeks - we monkey patch Beam bugs, test in our production jobs, before upstream Beam release - we use a lightweight collaboration model, mainly just Github issues & PRs - there're only 3 Scio developers at Spotify supporting 150+ internal users and many external ones, all running on Dataflow However I also want to point out that nothing should stop those interested from trying it out or contributing - we decoupled Dataflow runner as much as possible - Scio should run on other runners without modification, just a matter of changing dependencies and arguments - there're still parts coupled with GCP and Dataflow runner but hopefully we can gradually decouple them as the file system and other related API improves - it'd be great to see bug reports and PRs from the community > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934149#comment-15934149 ] Jean-Baptiste Onofré commented on BEAM-302: --- [~neville] updated the branch to Beam 0.6.0, so, I think we can discuss about a merge in Apache codebase after a little cleanup. Thought ? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933886#comment-15933886 ] Nicholaus E Halecky commented on BEAM-302: -- Hi all! Wonderful to see the progress made here so far, and was interested to know the status of this effort? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836555#comment-15836555 ] Amit Sela commented on BEAM-302: Oh, got it, thanks! > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836553#comment-15836553 ] Neville Li commented on BEAM-302: - WIP branch here using 0.4.0 https://github.com/spotify/scio/tree/apache-beam Ticket https://github.com/spotify/scio/issues/279 > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836551#comment-15836551 ] Amit Sela commented on BEAM-302: You mean 0.5.0 ? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836549#comment-15836549 ] Jean-Baptiste Onofré commented on BEAM-302: --- I updated to 0.4.0 release and I will deal with Neville for the merge. > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836540#comment-15836540 ] Amit Sela commented on BEAM-302: [~davor] where are we with Scio integration ? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836535#comment-15836535 ] Amit Sela commented on BEAM-302: Scio currently supports the Dataflow SDK (sort of Beam predecessor), and once it will support Beam it could interact with any runner supporting the Java SDK since Scio is a Scala DSL running on top of the Java SDK. > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836490#comment-15836490 ] Chaoran Yu commented on BEAM-302: - Looks like scio currently only supports Google Cloud Dataflow as the underlying runner. Now that the project is donated to Beam, are there any plans to support Spark, Flink and other runners? > Add Scio Scala DSL to Beam > -- > > Key: BEAM-302 > URL: https://issues.apache.org/jira/browse/BEAM-302 > Project: Beam > Issue Type: Wish > Components: sdk-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Neville Li > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
