Re: PipelineOptions fromSystemProps?

will create it now, thanks Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book

Re: PipelineOptions fromSystemProps?

Good idea to split the discussion. We can complete PipelineOptions.fromSystemProperties immediately for Java and separate the cross-language "multi-env var representation" for pipeline options. Probably Python & Go fans have muted this thread, after all. On Fri, Feb 16, 2018 at 8:07 AM, Romain

Re: PipelineOptions fromSystemProps?

Oh, so the point was than env would be under the portability umbrella versus system properties are not? Kind of makes sense phrased this way for me. Do we want another thread for that? Romain Manni-Bucau @rmannibucau | Blog

Re: PipelineOptions fromSystemProps?

> On Thu, Feb 15, 2018, 11:47 PM Romain Manni-Bucau > wrote: > >> >> 2018-02-15 20:00 GMT+01:00 Kenneth Knowles : >> >>> On Thu, Feb 15, 2018 at 12:03 AM, Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: 2. default properties = env + system

Re: PipelineOptions fromSystemProps?

I don't have a strong feeling about null here (though we're Java 8 now - can't we start using Optional instead?). However it's always easier to add things and harder to remove things. If it's not really needed now, id vote to leave it out of this PR and add later if necessary. On Thu, Feb 15,

Re: PipelineOptions fromSystemProps?

2018-02-15 20:00 GMT+01:00 Kenneth Knowles : > On Thu, Feb 15, 2018 at 12:03 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> wrote: >> >> 2. default properties = env + system properties: this is what all config >> libs do (spring config, tamaya, deltaspike, microprofile-config,

Re: PipelineOptions fromSystemProps?

On Thu, Feb 15, 2018 at 12:03 AM, Romain Manni-Bucau wrote: > > 2. default properties = env + system properties: this is what all config > libs do (spring config, tamaya, deltaspike, microprofile-config, ...) so it > is very comfortable and sane for end users. It enables

Re: PipelineOptions fromSystemProps?

this is part of the PR without any transformation - as requested. I'm happy to do a follow up PR to do the same transformations than in deltaspike for instance:

Re: PipelineOptions fromSystemProps?

+1 to Romain idea with Kenn's approach, we should probably reserve the 'beam.' namespace too because we are already using it for system properties in the spark runner, following along the convention of 'spark.*' in Apache Spark. Any ideas on how to handle this for the env vraiables case ? maybe

Re: PipelineOptions fromSystemProps?

FYI created https://github.com/apache/beam/pull/4683 about it Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: PipelineOptions fromSystemProps?

Oki, will try a PR after the classloader one then. Thanks a lot. Le 13 févr. 2018 19:14, "Lukasz Cwik" a écrit : > The one in Dataflow 1.x was one system property that contained all the > JSON so it wasn't exactly what you were looking for. > > On Tue, Feb 13, 2018 at 9:51 AM,

Re: PipelineOptions fromSystemProps?

The one in Dataflow 1.x was one system property that contained all the JSON so it wasn't exactly what you were looking for. On Tue, Feb 13, 2018 at 9:51 AM, Romain Manni-Bucau wrote: > I like your proposal Kenneth. Perfectly fits my use case and deployment > one as well -

Re: PipelineOptions fromSystemProps?

PipelineOptionsFactory.fromSystemProperties did exist in Dataflow 1.x and was dropped for the reason that Ken mentioned. On Tue, Feb 13, 2018 at 9:46 AM, Kenneth Knowles wrote: > Pipeline options are not global - they are a property of a single job. The > TestPipeline reads

Re: PipelineOptions fromSystemProps?

Pipeline options are not global - they are a property of a single job. The TestPipeline reads them from a very particular system property because it is a special testing rule. If you want a generic way to build pipeline options from a set of system properties, it should be from an explicit

Re: PipelineOptions fromSystemProps?

makes sense, do we want beam.foo.bar -> --foo-bar conversion too? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: PipelineOptions fromSystemProps?

Neutral about this one: haven't seen a case where this was needed, but don't see anything wrong with it either. One thing I'd recommend if you go through with it, extract from system properties under "beam." rather than all of them, to avoid clashes. On Tue, Feb 13, 2018, 7:53 AM Jean-Baptiste

Re: PipelineOptions fromSystemProps?

Hi Romain, it sounds interesting to me, and doesn't break anything, so +1 from my side. Regards JB On 02/13/2018 03:42 PM, Romain Manni-Bucau wrote: > Hi guys, > > there are hacks in beam testing code to read the args from a system property > but > I wonder if we shouldnt add a

PipelineOptions fromSystemProps?

Hi guys, there are hacks in beam testing code to read the args from a system property but I wonder if we shouldnt add a PipelineOptionsFactory.fromSystemProperties(). It would iterate over the system properties and take all --xxx=foo as potential argument it tries to bind. Rational behind that