Re: Better naming for runner specific options

+1 On 22.05.19 04:28, Reza Rokni wrote: Hi, Coming back to this, is the general consensus that this can be addressed via https://issues.apache.org/jira/browse/BEAM-6531 in Beam 3.0? Cheers Reza On Tue, 7 May 2019 at 23:15, Valentyn Tymofieiev > wrote: I thi

Re: Better naming for runner specific options

Hi, Coming back to this, is the general consensus that this can be addressed via https://issues.apache.org/jira/browse/BEAM-6531 in Beam 3.0? Cheers Reza On Tue, 7 May 2019 at 23:15, Valentyn Tymofieiev wrote: > I think using RunnerOptions was an idea at some point, but in Python, we > ended u

Re: Better naming for runner specific options

I think using RunnerOptions was an idea at some point, but in Python, we ended up parsing options from the runner api without populating RunnerOptions, and RunnerOptions was eventually removed [1]. If we decide to rename options, a path forward may be to have runners recognize both old and new na

Re: Better naming for runner specific options

On Mon, May 6, 2019 at 3:01 PM Ahmet Altay wrote: > There is RunnerOptions already. Its options are populated by querying the > job service. Any portable runner is able to provide a list of options that > is runner specific through that mechanism. > > *From: *Reza Rokni > *Date: *Mon, May 6, 201

Re: Better naming for runner specific options

So the options here would be moved to runner options? https://beam.apache.org/releases/pydoc/2.12.0/_modules/apache_beam/options/pipeline_options.html#WorkerOptions In Java they are in DataflowPipelineWorkerPoolOptions and of course we have FlinkPipelineOptions etc... *From: *Chamikara Jayalath

Re: Better naming for runner specific options

There is RunnerOptions already. Its options are populated by querying the job service. Any portable runner is able to provide a list of options that is runner specific through that mechanism. *From: *Reza Rokni *Date: *Mon, May 6, 2019 at 2:57 PM *To: * So the options here would be moved to run

Re: Better naming for runner specific options

On Mon, May 6, 2019 at 2:13 PM Lukasz Cwik wrote: > There were also discussions[1] in the past about scoping PipelineOptions > to specific PTransforms. Would scoping PipelineOptions to PTransforms make > this a more general solution? > > 1: > https://lists.apache.org/thread.html/05f849d39788cb0af

Re: Better naming for runner specific options

There were also discussions[1] in the past about scoping PipelineOptions to specific PTransforms. Would scoping PipelineOptions to PTransforms make this a more general solution? 1: https://lists.apache.org/thread.html/05f849d39788cb0af840cb9e86ca631586783947eb4e5a1774b647d1@%3Cdev.beam.apache.org%

Re: Better naming for runner specific options

Having namespaces for option makes sense. I think, along with a help command to print all the options given the runner name will be useful. As for the scope of name spacing, I think that assigning a logical name space gives more flexibility around how and where we declare options. It also make futu

Re: Better naming for runner specific options

Good points. As already mentioned there is no namespacing between the different pipeline option classes. In particular, there is no separate namespace for system and user options which is most concerning. I'm in favor of an optional namespace using the class name of the defining pipeline optio

Re: Better naming for runner specific options

Great point Lukasz, worker machine could be relevant to multiple runners. Perhaps for parameters that could have multiple runner relevance, the doc could be rephrased to reflect its potential multiple uses. For example change the help information to start with a generic reference " worker type on

Re: Better naming for runner specific options

Even though they are in classes named for specific runners, they are not namespaced. All PipelineOptions exist in a global namespace so they need to be careful to be very precise. It is a good point that even though they may be multiple uses for "machine type" they are probably not going to both h

Re: Better naming for runner specific options

Also, we do have runner specific options classes where truly runner specific options can go. https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java https://github.com/apache/beam/blob/master/

Re: Better naming for runner specific options

I agree, that is a good point. *From: *Lukasz Cwik *Date: *Fri, May 3, 2019 at 9:37 AM *To: *dev The concept of a machine type isn't necessarily limited to Dataflow. If it > made sense for a runner, they could use AWS/Azure machine types as well. > > On Fri, May 3, 2019 at 9:32 AM Ahmet Altay w

Re: Better naming for runner specific options

The concept of a machine type isn't necessarily limited to Dataflow. If it made sense for a runner, they could use AWS/Azure machine types as well. On Fri, May 3, 2019 at 9:32 AM Ahmet Altay wrote: > This idea was discussed in a PR a few months ago, and JIRA was filed as a > follow up [1]. IMO,

Re: Better naming for runner specific options

This idea was discussed in a PR a few months ago, and JIRA was filed as a follow up [1]. IMO, it makes sense to use a namespace prefix. The primary issue here is that, such a change will very likely be a backward incompatible change and would be hard to do before the next major version. [1] https:

Better naming for runner specific options

Hi, Was reading this SO question: https://stackoverflow.com/questions/53833171/googlecloudoptions-doesnt-have-all-options-that-pipeline-options-has And noticed that in https://beam.apache.org/releases/pydoc/2.12.0/_modules/apache_beam/options/pipeline_options.html#WorkerOptions The option is c