The one in Dataflow 1.x was one system property that contained all the JSON so it wasn't exactly what you were looking for.
On Tue, Feb 13, 2018 at 9:51 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > I like your proposal Kenneth. Perfectly fits my use case and deployment > one as well - when ops configure the env without modifying the code. > > How do we move forward on that? Should I send a PR or do you want to > import what was in dataflow? > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > 2018-02-13 18:49 GMT+01:00 Lukasz Cwik <lc...@google.com>: > >> PipelineOptionsFactory.fromSystemProperties did exist in Dataflow 1.x >> and was dropped for the reason that Ken mentioned. >> >> On Tue, Feb 13, 2018 at 9:46 AM, Kenneth Knowles <k...@google.com> wrote: >> >>> Pipeline options are not global - they are a property of a single job. >>> The TestPipeline reads them from a very particular system property because >>> it is a special testing rule. >>> >>> If you want a generic way to build pipeline options from a set of system >>> properties, it should be from an explicit prefix, not global defaults. So >>> if a user wants to put a bunch of options into properties, they choose a >>> namespace like "my.random.namespace" and everything under it can be >>> interpreted as a pipelineoption: >>> >>> my.random.namespace.SomeOption=bizzle >>> my.random.namespace.SomeOtherOption=whatever >>> >>> And you could do PipelineOptionsFactory.fromSys >>> temProperties("my.random.namespace") >>> >>> We should use the full name of the option not split it with dots - dots >>> represent hierarchy not words separation. >>> >>> interface FooOptions extends PipelineOptions { >>> getSomeOption() >>> getSomeOtherOption() >>> } >>> >>> I think I can be +0.5 on this. We might also reserve a namespace like >>> "beam.TestPipeline.options" and use it for specification of test config. >>> Could be easier than embedding JSON in a property, in some cases. Easier to >>> override little pieces for sure. >>> >>> Kenn >>> >>> On Tue, Feb 13, 2018 at 9:29 AM, Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>>> makes sense, do we want beam.foo.bar -> --foo-bar conversion too? >>>> >>>> >>>> Romain Manni-Bucau >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> <http://rmannibucau.wordpress.com> | Github >>>> <https://github.com/rmannibucau> | LinkedIn >>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>> >>>> 2018-02-13 18:19 GMT+01:00 Eugene Kirpichov <kirpic...@google.com>: >>>> >>>>> Neutral about this one: haven't seen a case where this was needed, but >>>>> don't see anything wrong with it either. One thing I'd recommend if you go >>>>> through with it, extract from system properties under "beam." rather than >>>>> all of them, to avoid clashes. >>>>> >>>>> On Tue, Feb 13, 2018, 7:53 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>>>> wrote: >>>>> >>>>>> Hi Romain, >>>>>> >>>>>> it sounds interesting to me, and doesn't break anything, so +1 from >>>>>> my side. >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 02/13/2018 03:42 PM, Romain Manni-Bucau wrote: >>>>>> > Hi guys, >>>>>> > >>>>>> > there are hacks in beam testing code to read the args from a system >>>>>> property but >>>>>> > I wonder if we shouldnt add a PipelineOptionsFactory.fromSys >>>>>> temProperties(). >>>>>> > >>>>>> > It would iterate over the system properties and take all --xxx=foo >>>>>> as potential >>>>>> > argument it tries to bind. >>>>>> > >>>>>> > Rational behind that is to enable users to wrap the pipeline API >>>>>> but still >>>>>> > expose the pipeline options to end users for advanced cases. >>>>>> > >>>>>> > Any discussion on this kind of usages already? What do you think of >>>>>> this proposal? >>>>>> > >>>>>> > Side note: we can think about a fromEnv() too. >>>>>> > >>>>>> > Romain Manni-Bucau >>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog >>>>>> > <http://rmannibucau.wordpress.com> | Github < >>>>>> https://github.com/rmannibucau> | >>>>>> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book >>>>>> > <https://www.packtpub.com/application-development/java-ee-8- >>>>>> high-performance> >>>>>> >>>>>> -- >>>>>> Jean-Baptiste Onofré >>>>>> jbono...@apache.org >>>>>> http://blog.nanthrax.net >>>>>> Talend - http://www.talend.com >>>>>> >>>>> >>>> >>> >> >
