+Amit, Aljoscha, Manu Any comments from folks on the Flink, Spark, or Gearpump runners?
On Tue, Aug 2, 2016 at 11:10 AM, Robert Bradshaw < [email protected]> wrote: > Being able to "late-bind" parameters like input paths to a > pre-constructed program would be a very useful feature, and I think is > worth adding to Beam. > > Of the four API proposals, I have a strong preference for (4). > Further, it seems that these need not be bound to the PipelineOptions > object itself (i.e. a named RuntimeValueSupplier could be constructed > off of a pipeline object), which the Python API makes less heavy use > of (encouraging the user to use familiar, standard libraries for > argument parsing), though of course such integration is useful to > provide for convenience. > > - Robert > > On Fri, Jul 29, 2016 at 12:14 PM, Sam McVeety <[email protected]> > wrote: > > During the graph construction phase, the given SDK generates an initial > > execution graph for the program. At execution time, this graph is > > executed, either locally or by a service. Currently, Beam only supports > > parameterization at graph construction time. Both Flink and Spark supply > > functionality that allows a pre-compiled job to be run without SDK > > interaction with updated runtime parameters. > > > > In its current incarnation, Dataflow can read values of PipelineOptions > at > > job submission time, but this requires the presence of an SDK to properly > > encode these values into the job. We would like to build a common layer > > into the Beam model so that these dynamic options can be properly > provided > > to jobs. > > > > Please see > > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_I > K1r1YAJ90JG5Fz0_28o/edit > > for the high-level model, and > > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMM > kOSgGi8ZUH-MOnFatZ8/edit > > for > > the specific API proposal. > > > > Cheers, > > Sam >
