+1 Thanks Sam, it sounds interesting.
Regards JB On 07/29/2016 09:14 PM, Sam McVeety wrote:
During the graph construction phase, the given SDK generates an initial execution graph for the program. At execution time, this graph is executed, either locally or by a service. Currently, Beam only supports parameterization at graph construction time. Both Flink and Spark supply functionality that allows a pre-compiled job to be run without SDK interaction with updated runtime parameters. In its current incarnation, Dataflow can read values of PipelineOptions at job submission time, but this requires the presence of an SDK to properly encode these values into the job. We would like to build a common layer into the Beam model so that these dynamic options can be properly provided to jobs. Please see https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit for the high-level model, and https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit for the specific API proposal. Cheers, Sam
-- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
