[
https://issues.apache.org/jira/browse/BEAM-6942?focusedWorklogId=226429&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226429
]
ASF GitHub Bot logged work on BEAM-6942:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Apr/19 00:17
Start Date: 12/Apr/19 00:17
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8225: [BEAM-6942]
Make modifications to pipeline options to be visible to all views.
URL: https://github.com/apache/beam/pull/8225#discussion_r274723816
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -246,7 +276,40 @@ def display_data(self):
return self.get_all_options(True)
def view_as(self, cls):
+ """Returns a view of current object as provided PipelineOption subclass.
+
+ Example Usage::
+
+ options = PipelineOptions(['--runner', 'Direct', '--streaming'])
+ standard_options = options.view_as(StandardOptions)
+ if standard_options.streaming:
+ # ... start a streaming job ...
+
+ Note that options objects may have multiple views, and modifications
+ of values in any view-object will apply to current object and other
+ view-objects.
+
+ Args:
+ cls: PipelineOptions class or any of its subclasses.
+
+ Returns:
+ An instance of cls that is intitialized using options contained in
current
+ object.
+
+ """
view = cls(self._flags)
+ for option_name in view._visible_option_list():
+ # Initialize values of keys defined by a cls.
+ #
+ # Note that we do initialization only once per key to make sure that
+ # values in _all_options dict are not-recreated with each new view.
+ # This is important to make sure that values of multi-options keys are
+ # backed by the same list across multiple views, and that any overrides
of
+ # pipeline options already stored in _all_options are preserved.
+ if option_name not in self._all_options:
+ self._all_options[option_name] = getattr(view._visible_options,
+ option_name)
+ # Note that views will still store _all_options of the source object.
Review comment:
No, what is exposed by a view will be defined by `getattr` which calls
`_visible_option_list`, which returns `_visible_options`, which are the options
recognized by the parser which are populated during `view = cls(self._flags)`.
The view may still store default values defined by previous views, but should
not expose them. It would be prudent to add a test for this, looks like there
isn't one already.
I think there will be undefined behavior if multiple options define the same
flag with different default values, and the flag is not set on the command
line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 226429)
Time Spent: 8h 10m (was: 8h)
> Pipeline options to experiment propagation is not working in Dataflow runner.
> -----------------------------------------------------------------------------
>
> Key: BEAM-6942
> URL: https://issues.apache.org/jira/browse/BEAM-6942
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Relevant code:
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L356-L388]
> 3 experiments/options are affected. We need to fix it in 2.12.0
> cc: [~altay] [~apilloud]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
