Re: Scio Beam Scala API

Hi guys, I will share the branch with the 0.4.0-incubating-SNAPSHOT update and tested with Spark runner. Regards JB On 12/11/2016 09:13 PM, Neville Li wrote: Aviem, You're right we have a scio/apache-beam branch that works with Beam

Re: New DoFn and WindowedValue/WinowingInternals

Hi Amit, Yes, this is correct. Part of the motivation for this is that DoFn API is user-facing, and the compressed representation of windowed elements (e.g. access to all windows of an element), as well as the ability to emit directly into a specified window, is an implementation detail of the

Re: Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Spark #385

Filed https://issues.apache.org/jira/browse/BEAM-1130 On Sun, Dec 11, 2016 at 4:52 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See RunnableOnService_Spark/changes> > > -- --- Jason Kuster Apache Beam (Incubating) /

Re: Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2066

Filed https://issues.apache.org/jira/browse/BEAM-1130 for breaks here and in spark RoS tests On Sun, Dec 11, 2016 at 5:00 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See MavenInstall/changes> > > -- --- Jason

Re: Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1787

> > Is there any way to retry staging if it fails? >> > I believe the code already does this, but Pei would know for sure.

Re: Scio Beam Scala API

Aviem, You're right we have a scio/apache-beam branch that works with Beam 0.2.0-incubating and are working on keeping it up with the latest releases. A few ways you can contribute: - It runs with the Dataflow runner but hasn't been tested with

Re: New DoFn and WindowedValue/WinowingInternals

You've got it right. My recommendations is to just directly implement it for the Spark runner. It will often actually clean things up a bit. Here's the analogous change for the Flink runner: https://github.com/apache/incubator-beam/pull/1435/files. With GABW, I tried going through the process of

Re: New DoFn and WindowedValue/WinowingInternals

So basically using new DoFn with *SplittableParDo*, *Window.Bound*, and *GroupAlsoByWindow* requires a custom implementation by per runner as they are not handled by DoFn anymore, right ? On Sun, Dec 11, 2016 at 3:42 PM Eugene Kirpichov wrote: > Hi Amit, I'll

Re: Scio Beam Scala API

Hi I'm working on a feature branch with Neville and his guys. I already updated to last changes. I would like to propose a feature branch later this week. Regards JB⁣​ On Dec 11, 2016, 16:39, at 16:39, Aviem Zur wrote: >Hi, > >I've heard there has been work towards porting

Scio Beam Scala API

Hi, I've heard there has been work towards porting Scio Dataflow Scala API to Beam. I was wondering at what stage this is in, where is this happening (Saw no branch in BEAM repository, and one in Scio repository that is dependent on beam 0.2.0-INCUBATING) and if there is a way to contribute?

Re: New DoFn and WindowedValue/WinowingInternals

Hi Amit, I'll comment in more detail later, but meanwhile please take a look at https://github.com/apache/incubator-beam/pull/1565 There is a small amount of relevant changes to spark runner. Take a look at implementation of SplittableParDo (already committed) in particular ProcessFn and it's