[INFO] Spark runner updated to Spark 2.2.1

Hi all, We are pleased to announce that Spark 2.x support in Spark runner has been merged this morning. It supports Spark 2.2.1. In the same PR, we did update to Scala 2.11, including Flink artifacts update to 2.11 (it means it's already ready to upgrade to Flink 1.4 !). It also means, as

Re: [INFO] Spark runner updated to Spark 2.2.1

Congrats, was waited for a long time! Very impatient to see the announcement of the 2.3! Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github |

Re: [INFO] Spark runner updated to Spark 2.2.1

By the way, Flink has been updated to Flink 1.4.0 as well (as the artifacts already used Scala 2.11). Regards JB On 12/18/2017 11:50 AM, Jean-Baptiste Onofré wrote: Hi all, We are pleased to announce that Spark 2.x support in Spark runner has been merged this morning. It supports Spark

Re: [INFO] Spark runner updated to Spark 2.2.1

Nice! On Mon, Dec 18, 2017 at 12:51 PM Jean-Baptiste Onofré wrote: > Hi all, > > We are pleased to announce that Spark 2.x support in Spark runner has been > merged this morning. It supports Spark 2.2.1. > > In the same PR, we did update to Scala 2.11, including Flink

Re: Callbacks/other functions run after a PDone/output transform

I'm a bit confused by all of these suggestions: they sound plausible at a high level, but I'm having a hard time making any one of them concrete. So suppose we want to create a transform Wait.on(PCollection signal): PCollection -> PCollection. a.apply(Wait.on(sig)) returns a PCollection that is

Re: Usecase scenario: Job definition from low frequently changing storage

1) Based upon your question, it seems like you want to have one large job that then spins off really small jobs. Any reason you can't just do it all within one pipeline? 2. Am I capable of defining flexible window sizes per "device-task"? Yes, you'll want a custom WindowFn which when going