[PROPOSAL] Having 2 Spark runners to support Spark 1 users while advancing towards better streaming implementation with Spark 2

2016-08-03 Thread Amit Sela
After discussions with JB, and understanding that a lot of companies running Spark will probably run 1.6.x for a while, we thought it would be a good idea to have (some) support for both branches. The SparkRunnerV1 will mostly support Batch, but could also support “KeyedState” workflows and Sessio

Podling Report Reminder - August 2016

2016-08-03 Thread johndament
Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 17 August 2016, 10:30 am PDT. The report for your podling will form

Re: [REFLECT] Beam’s Half Birthday!

2016-08-03 Thread Frances Perry
Thanks for the suggestions, folks! Incorporated them and posted the blog: http://beam.incubator.apache.org/blog/2016/08/03/six-months.html On Tue, Aug 2, 2016 at 12:42 AM, Ismaël Mejía wrote: > Hello, > > Nice reminder of the work done, I feel quite proud of what this community > has > accomplis

Re: Utility for Serializing PipelineOptions

2016-08-03 Thread Lukasz Cwik
The core Java SDK already contains the code to serialize/deserialize PipelineOptions using Jackson's ObjectMapper since PipelineOptions has Jackson databind annotations @JsonSerialize and @JsonDeserialize. Is there something more specific that you were looking for? On Wed, Aug 3, 2016 at 12:54 PM

Utility for Serializing PipelineOptions

2016-08-03 Thread P. Taylor Goetz
Looking at the runner implementations for Spark, Flink, and Gearpump, they all have the same requirement of being able to serialize/deserialize PipelineOptions, and they each seem to use slightly different approaches. In working on a Beam runner for Storm I have the same requirement, and imagine