+1 It sounds really nice, (4) is by far the most consistent with the current Options implementation. One extra thing, maybe it is a good idea to sketch a 'real' use case to make the concepts/need more evident.
Ismaël On Tue, Aug 9, 2016 at 8:49 PM, Sam McVeety <[email protected]> wrote: > Thanks, Amit and JB. Amit, to your question: the intention with > availability to PTransforms is provide the ValueProvider abstraction (which > may be implemented on top of PipelineOptions) so that they do not take a > dependency on PipelineOptions. > > Cheers, > Sam > > On Mon, Aug 8, 2016 at 12:26 AM, Jean-Baptiste Onofré <[email protected]> > wrote: > > > +1 > > > > Thanks Sam, it sounds interesting. > > > > Regards > > JB > > > > > > On 07/29/2016 09:14 PM, Sam McVeety wrote: > > > >> During the graph construction phase, the given SDK generates an initial > >> execution graph for the program. At execution time, this graph is > >> executed, either locally or by a service. Currently, Beam only supports > >> parameterization at graph construction time. Both Flink and Spark > supply > >> functionality that allows a pre-compiled job to be run without SDK > >> interaction with updated runtime parameters. > >> > >> In its current incarnation, Dataflow can read values of PipelineOptions > at > >> job submission time, but this requires the presence of an SDK to > properly > >> encode these values into the job. We would like to build a common layer > >> into the Beam model so that these dynamic options can be properly > provided > >> to jobs. > >> > >> Please see > >> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_I > >> K1r1YAJ90JG5Fz0_28o/edit > >> for the high-level model, and > >> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMM > >> kOSgGi8ZUH-MOnFatZ8/edit > >> for > >> the specific API proposal. > >> > >> Cheers, > >> Sam > >> > >> > > -- > > Jean-Baptiste Onofré > > [email protected] > > http://blog.nanthrax.net > > Talend - http://www.talend.com > > >
