Re: Proposal: Dynamic PIpelineOptions

2016-08-04 Thread Frances Perry
+Amit, Aljoscha, Manu Any comments from folks on the Flink, Spark, or Gearpump runners? On Tue, Aug 2, 2016 at 11:10 AM, Robert Bradshaw < rober...@google.com.invalid> wrote: > Being able to "late-bind" parameters like input paths to a > pre-constructed program would be a very useful feature,

Re: [PROPOSAL] Having 2 Spark runners to support Spark 1 users while advancing towards better streaming implementation with Spark 2

2016-08-04 Thread Kenneth Knowles
+1 I definitely think it is important to support spark 1 and 2 simultaneously, and I agree that side-by-side seems the best way to do it. I'll refrain from commenting on the specific technical aspects of the two runners and focus just on the split: I am also curious about the answer to Dan's

[PROPOSAL] Splittable DoFn - Replacing the Source API with non-monolithic element processing in DoFn

2016-08-04 Thread Eugene Kirpichov
Hello Beam community, We (myself, Daniel Mills and Robert Bradshaw) would like to propose "Splittable DoFn" - a major generalization of DoFn, which allows processing of a single element to be non-monolithic, i.e. checkpointable and parallelizable, as well as doing an unbounded amount of work per