[
https://issues.apache.org/jira/browse/BEAM-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191064#comment-17191064
]
Suzen Fylke edited comment on BEAM-9093 at 9/5/20, 3:21 PM:
------------------------------------------------------------
I'm a bit confused about how this is supposed to work with the DataflowRunner,
since {{no_use_public_ips}} isn't listed in the docs:
[https://cloud.google.com/dataflow/docs/guides/specifying-exec-params#setting-other-cloud-dataflow-pipeline-options]
I'm using the DataflowRunner with TFX ([https://github.com/tensorflow/tfx]) and
have to pass the pipeline options as a list of args. Currently, directly
passing a PipelineOptions object is not supported, and I've filed a ticket for
that in the TFX repo ([https://github.com/tensorflow/tfx/issues/2332]).
What's the recommended way to convert \{'use_public_ips': False} to a flag?
* "--use_public_ips False" causes
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args:
['False']"
* "--use_public_ips=False" causes "argparse.ArgumentError: argument
--use_public_ips: ignored explicit argument 'False'"
* Setting \{'no_use_public_ips': True} in my PipelineOptions causes
"WARNING:apache_beam.options.pipeline_options:Discarding invalid overrides:
\{'no_use_public_ips': True}"
What I'm doing right now is a little hacky:
pipeline_options = PipelineOptions(**\{'use_public_ips': False, etc.})
all_options = pipeline_options.get_all_options(drop_default=True)
dataflow_args = []
all_options.pop('use_public_ips')
dataflow_args.append('–no_use_public_ips')
## iterate over all_options and append f'–\{key}=\{value}' to dataflow_args
was (Author: codesue):
I'm a bit confused about how this is supposed to work with the DataflowRunner,
since {{no_use_public_ips}} isn't listed in the docs:
[https://cloud.google.com/dataflow/docs/guides/specifying-exec-params#setting-other-cloud-dataflow-pipeline-options]
I'm using the DataflowRunner with TFX ([https://github.com/tensorflow/tfx]) and
have to pass the pipeline options as a list of args. Currently, directly
passing a PipelineOptions object is not supported, and I've filed a ticket for
that in the TFX repo ([https://github.com/tensorflow/tfx/issues/2332]).
What's the recommended way to convert \{'use_public_ips': False} to a flag?
* "--use_public_ips False" causes
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args:
['False']"
* "--use_public_ips=False" causes "argparse.ArgumentError: argument
--use_public_ips: ignored explicit argument 'False'"
* Setting \{'no_use_public_ips': True} in my PipelineOptions causes
"WARNING:apache_beam.options.pipeline_options:Discarding invalid overrides:
\{'no_use_public_ips': True}"
What I'm doing right now is a little hacky:
pipeline_options = PipelineOptions(**\{'use_public_ips': False, etc.})
all_options = pipeline_options.get_all_options(drop_default=True)
dataflow_args = []
all_options.pop('use_public_ips')
dataflow_args.append('--no_use_public_ips')
> Pipeline options which with different underlying store variable does not get
> over written
> -----------------------------------------------------------------------------------------
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Ankur Goenka
> Priority: P3
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
