Jenkins build is back to normal : beam_SeedJob_Standalone #283

See

Re: Callbacks/other functions run after a PDone/output transform

I'm a bit confused by all of these suggestions: they sound plausible at a high level, but I'm having a hard time making any one of them concrete. So suppose we want to create a transform Wait.on(PCollection signal): PCollection -> PCollection. a.apply(Wait.on(sig)) returns a PCollection that is

Re: [INFO] Spark runner updated to Spark 2.2.1

Sweet! On Mon, Dec 18, 2017 at 9:20 AM, Reuven Lax wrote: > Very nice! > > On Mon, Dec 18, 2017 at 2:50 AM, Jean-Baptiste Onofré > wrote: > >> Hi all, >> >> We are pleased to announce that Spark 2.x support in Spark runner has >> been merged this morning.

Re: [INFO] Spark runner updated to Spark 2.2.1

Very nice! On Mon, Dec 18, 2017 at 2:50 AM, Jean-Baptiste Onofré wrote: > Hi all, > > We are pleased to announce that Spark 2.x support in Spark runner has been > merged this morning. It supports Spark 2.2.1. > > In the same PR, we did update to Scala 2.11, including Flink

Re: [INFO] Spark runner updated to Spark 2.2.1

Congrats, was waited for a long time! Very impatient to see the announcement of the 2.3! Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github |

Re: [INFO] Spark runner updated to Spark 2.2.1

By the way, Flink has been updated to Flink 1.4.0 as well (as the artifacts already used Scala 2.11). Regards JB On 12/18/2017 11:50 AM, Jean-Baptiste Onofré wrote: Hi all, We are pleased to announce that Spark 2.x support in Spark runner has been merged this morning. It supports Spark

Re: Euphoria Java 8 DSL - proposal

Thanks Jan, It makes sense. Let me take a look on the code to understand the "interaction". Regards JB On 12/18/2017 04:26 PM, Jan Lukavský wrote: Hi JB, basically you are not wrong. The project started about three or four years ago with a goal to unify batch and streaming processing into

Re: Euphoria Java 8 DSL - proposal

Hi JB, basically you are not wrong. The project started about three or four years ago with a goal to unify batch and streaming processing into single portable, executor independent API. Because of that, it is currently "close" to Beam in this sense. But we don't see much added value keeping

Looking for sb to do review of kubernetes scripts with HDFS datastore

Hi all, I recently submitted PR https://github.com/apache/beam/pull/4261 that allows to set up small Kubernetes hdfs cluster and run filebased io tests on it using Direct and Dataflow runner. This is basically enabler for doing performance testing on hdfs. Is there anyone who can do a review of

Re: Euphoria Java 8 DSL - proposal

Hi Jan, Thanks for your answers. However, they confused me ;) Regarding what you replied, Euphoria seems like a programming model/SDK "close" to Beam more than a DSL on top of an existing Beam SDK. Am I wrong ? Regards JB On 12/18/2017 03:44 PM, Jan Lukavský wrote: Hi Ismael, basically

Re: Euphoria Java 8 DSL - proposal

Hi Ismael, basically we adopted the Beam's design regarding partitioning (https://github.com/seznam/euphoria/issues/160) and implemented the sorting manually (https://github.com/seznam/euphoria/issues/158). I'm not aware of the time model differences (Euphoria supports ingestion and event

Re: Euphoria Java 8 DSL - proposal

Hi, It is great to see that you guys have achieved a maturity point to propose this. Congratulations for your work and the idea to contribute it into Beam. I remember from a previous discussion with Jan about the model mismatch between Euphoria and Beam, because of some design decisions of both

Re: [INFO] Spark runner updated to Spark 2.2.1

Nice! On Mon, Dec 18, 2017 at 12:51 PM Jean-Baptiste Onofré wrote: > Hi all, > > We are pleased to announce that Spark 2.x support in Spark runner has been > merged this morning. It supports Spark 2.2.1. > > In the same PR, we did update to Scala 2.11, including Flink

[INFO] Spark runner updated to Spark 2.2.1

Hi all, We are pleased to announce that Spark 2.x support in Spark runner has been merged this morning. It supports Spark 2.2.1. In the same PR, we did update to Scala 2.11, including Flink artifacts update to 2.11 (it means it's already ready to upgrade to Flink 1.4 !). It also means, as

Re: Euphoria Java 8 DSL - proposal

Depending of the donation, you would need ICLA for each contributor, and CCLA in addition of SGA. We can sync with Davor and I for the legal stuff. However, I would wait a little bit just to have feedback from the whole team and start a formal vote. I would be happy to start the formal vote.

Re: Euphoria Java 8 DSL - proposal

Hello, Thanks for the awesome feedback! Romain: We already use Java Stream API in all operators where it makes sense (eg.: ReduceByKey). Still not sure if it was a good choice, but i can be easily converted to iterator anyway. Side outputs support is coming soon, we already made an initial