I like your proposal Kenneth. Perfectly fits my use case and deployment one
as well - when ops configure the env without modifying the code.

How do we move forward on that? Should I send a PR or do you want to import
what was in dataflow?


Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>

2018-02-13 18:49 GMT+01:00 Lukasz Cwik <lc...@google.com>:

> PipelineOptionsFactory.fromSystemProperties did exist in Dataflow 1.x and
> was dropped for the reason that Ken mentioned.
>
> On Tue, Feb 13, 2018 at 9:46 AM, Kenneth Knowles <k...@google.com> wrote:
>
>> Pipeline options are not global - they are a property of a single job.
>> The TestPipeline reads them from a very particular system property because
>> it is a special testing rule.
>>
>> If you want a generic way to build pipeline options from a set of system
>> properties, it should be from an explicit prefix, not global defaults. So
>> if a user wants to put a bunch of options into properties, they choose a
>> namespace like "my.random.namespace" and everything under it can be
>> interpreted as a pipelineoption:
>>
>> my.random.namespace.SomeOption=bizzle
>> my.random.namespace.SomeOtherOption=whatever
>>
>> And you could do PipelineOptionsFactory.fromSystemProperties("my.random.
>> namespace")
>>
>> We should use the full name of the option not split it with dots - dots
>> represent hierarchy not words separation.
>>
>> interface FooOptions extends PipelineOptions {
>>   getSomeOption()
>>   getSomeOtherOption()
>> }
>>
>> I think I can be +0.5 on this. We might also reserve a namespace like
>> "beam.TestPipeline.options" and use it for specification of test config.
>> Could be easier than embedding JSON in a property, in some cases. Easier to
>> override little pieces for sure.
>>
>> Kenn
>>
>> On Tue, Feb 13, 2018 at 9:29 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> makes sense, do we want beam.foo.bar -> --foo-bar conversion too?
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>> 2018-02-13 18:19 GMT+01:00 Eugene Kirpichov <kirpic...@google.com>:
>>>
>>>> Neutral about this one: haven't seen a case where this was needed, but
>>>> don't see anything wrong with it either. One thing I'd recommend if you go
>>>> through with it, extract from system properties under "beam." rather than
>>>> all of them, to avoid clashes.
>>>>
>>>> On Tue, Feb 13, 2018, 7:53 AM Jean-Baptiste Onofré <j...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> Hi Romain,
>>>>>
>>>>> it sounds interesting to me, and doesn't break anything, so +1 from my
>>>>> side.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 02/13/2018 03:42 PM, Romain Manni-Bucau wrote:
>>>>> > Hi guys,
>>>>> >
>>>>> > there are hacks in beam testing code to read the args from a system
>>>>> property but
>>>>> > I wonder if we shouldnt add a PipelineOptionsFactory.fromSys
>>>>> temProperties().
>>>>> >
>>>>> > It would iterate over the system properties and take all --xxx=foo
>>>>> as potential
>>>>> > argument it tries to bind.
>>>>> >
>>>>> > Rational behind that is to enable users to wrap the pipeline API but
>>>>> still
>>>>> > expose the pipeline options to end users for advanced cases.
>>>>> >
>>>>> > Any discussion on this kind of usages already? What do you think of
>>>>> this proposal?
>>>>> >
>>>>> > Side note: we can think about a fromEnv() too.
>>>>> >
>>>>> > Romain Manni-Bucau
>>>>> > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> > <http://rmannibucau.wordpress.com> | Github <
>>>>> https://github.com/rmannibucau> |
>>>>> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> > <https://www.packtpub.com/application-development/java-ee-8-
>>>>> high-performance>
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbono...@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>>>
>>>>
>>>
>>
>

Reply via email to