this is part of the PR without any transformation - as requested. I'm happy to do a follow up PR to do the same transformations than in deltaspike for instance: https://github.com/apache/deltaspike/blob/master/deltaspike/core/impl/src/main/java/org/apache/deltaspike/core/impl/config/EnvironmentPropertyConfigSource.java#L55 or a more fancy - but real world - one like key -> key.substring(prefix.length()).toUpperCase(ROOT).replace('.', '_')
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> 2018-02-14 15:42 GMT+01:00 Ismaël Mejía <ieme...@gmail.com>: > +1 to Romain idea with Kenn's approach, we should probably reserve the > 'beam.' namespace too because we are already using it for system > properties in the spark runner, following along the convention of > 'spark.*' in Apache Spark. > > Any ideas on how to handle this for the env vraiables case ? maybe this? > export my.random.namespace="SomeOption=bizzle SomeOtherOption=whatever" > > > On Wed, Feb 14, 2018 at 9:35 AM, Romain Manni-Bucau > <rmannibu...@gmail.com> wrote: > > FYI created https://github.com/apache/beam/pull/4683 about it > > > > > > Romain Manni-Bucau > > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > > > 2018-02-13 19:21 GMT+01:00 Romain Manni-Bucau <rmannibu...@gmail.com>: > >> > >> Oki, will try a PR after the classloader one then. Thanks a lot. > >> > >> Le 13 févr. 2018 19:14, "Lukasz Cwik" <lc...@google.com> a écrit : > >>> > >>> The one in Dataflow 1.x was one system property that contained all the > >>> JSON so it wasn't exactly what you were looking for. > >>> > >>> On Tue, Feb 13, 2018 at 9:51 AM, Romain Manni-Bucau > >>> <rmannibu...@gmail.com> wrote: > >>>> > >>>> I like your proposal Kenneth. Perfectly fits my use case and > deployment > >>>> one as well - when ops configure the env without modifying the code. > >>>> > >>>> How do we move forward on that? Should I send a PR or do you want to > >>>> import what was in dataflow? > >>>> > >>>> > >>>> Romain Manni-Bucau > >>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > >>>> > >>>> 2018-02-13 18:49 GMT+01:00 Lukasz Cwik <lc...@google.com>: > >>>>> > >>>>> PipelineOptionsFactory.fromSystemProperties did exist in Dataflow > 1.x > >>>>> and was dropped for the reason that Ken mentioned. > >>>>> > >>>>> On Tue, Feb 13, 2018 at 9:46 AM, Kenneth Knowles <k...@google.com> > >>>>> wrote: > >>>>>> > >>>>>> Pipeline options are not global - they are a property of a single > job. > >>>>>> The TestPipeline reads them from a very particular system property > because > >>>>>> it is a special testing rule. > >>>>>> > >>>>>> If you want a generic way to build pipeline options from a set of > >>>>>> system properties, it should be from an explicit prefix, not global > >>>>>> defaults. So if a user wants to put a bunch of options into > properties, they > >>>>>> choose a namespace like "my.random.namespace" and everything under > it can be > >>>>>> interpreted as a pipelineoption: > >>>>>> > >>>>>> my.random.namespace.SomeOption=bizzle > >>>>>> my.random.namespace.SomeOtherOption=whatever > >>>>>> > >>>>>> And you could do > >>>>>> PipelineOptionsFactory.fromSystemProperties("my.random.namespace") > >>>>>> > >>>>>> We should use the full name of the option not split it with dots - > >>>>>> dots represent hierarchy not words separation. > >>>>>> > >>>>>> interface FooOptions extends PipelineOptions { > >>>>>> getSomeOption() > >>>>>> getSomeOtherOption() > >>>>>> } > >>>>>> > >>>>>> I think I can be +0.5 on this. We might also reserve a namespace > like > >>>>>> "beam.TestPipeline.options" and use it for specification of test > config. > >>>>>> Could be easier than embedding JSON in a property, in some cases. > Easier to > >>>>>> override little pieces for sure. > >>>>>> > >>>>>> Kenn > >>>>>> > >>>>>> On Tue, Feb 13, 2018 at 9:29 AM, Romain Manni-Bucau > >>>>>> <rmannibu...@gmail.com> wrote: > >>>>>>> > >>>>>>> makes sense, do we want beam.foo.bar -> --foo-bar conversion too? > >>>>>>> > >>>>>>> > >>>>>>> Romain Manni-Bucau > >>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > >>>>>>> > >>>>>>> 2018-02-13 18:19 GMT+01:00 Eugene Kirpichov <kirpic...@google.com > >: > >>>>>>>> > >>>>>>>> Neutral about this one: haven't seen a case where this was needed, > >>>>>>>> but don't see anything wrong with it either. One thing I'd > recommend if you > >>>>>>>> go through with it, extract from system properties under "beam." > rather than > >>>>>>>> all of them, to avoid clashes. > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, Feb 13, 2018, 7:53 AM Jean-Baptiste Onofré < > j...@nanthrax.net> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Hi Romain, > >>>>>>>>> > >>>>>>>>> it sounds interesting to me, and doesn't break anything, so +1 > from > >>>>>>>>> my side. > >>>>>>>>> > >>>>>>>>> Regards > >>>>>>>>> JB > >>>>>>>>> > >>>>>>>>> On 02/13/2018 03:42 PM, Romain Manni-Bucau wrote: > >>>>>>>>> > Hi guys, > >>>>>>>>> > > >>>>>>>>> > there are hacks in beam testing code to read the args from a > >>>>>>>>> > system property but > >>>>>>>>> > I wonder if we shouldnt add a > >>>>>>>>> > PipelineOptionsFactory.fromSystemProperties(). > >>>>>>>>> > > >>>>>>>>> > It would iterate over the system properties and take all > >>>>>>>>> > --xxx=foo as potential > >>>>>>>>> > argument it tries to bind. > >>>>>>>>> > > >>>>>>>>> > Rational behind that is to enable users to wrap the pipeline > API > >>>>>>>>> > but still > >>>>>>>>> > expose the pipeline options to end users for advanced cases. > >>>>>>>>> > > >>>>>>>>> > Any discussion on this kind of usages already? What do you > think > >>>>>>>>> > of this proposal? > >>>>>>>>> > > >>>>>>>>> > Side note: we can think about a fromEnv() too. > >>>>>>>>> > > >>>>>>>>> > Romain Manni-Bucau > >>>>>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog > >>>>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog > >>>>>>>>> > <http://rmannibucau.wordpress.com> | Github > >>>>>>>>> > <https://github.com/rmannibucau> | > >>>>>>>>> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book > >>>>>>>>> > > >>>>>>>>> > <https://www.packtpub.com/application-development/java- > ee-8-high-performance> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Jean-Baptiste Onofré > >>>>>>>>> jbono...@apache.org > >>>>>>>>> http://blog.nanthrax.net > >>>>>>>>> Talend - http://www.talend.com > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > > >
