[
https://issues.apache.org/jira/browse/BEAM-13024?focusedWorklogId=720454&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-720454
]
ASF GitHub Bot logged work on BEAM-13024:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Feb/22 22:14
Start Date: 03/Feb/22 22:14
Worklog Time Spent: 10m
Work Description: damccorm commented on a change in pull request #16719:
URL: https://github.com/apache/beam/pull/16719#discussion_r799012784
##########
File path: sdks/go/pkg/beam/runners/direct/direct.go
##########
@@ -65,6 +65,7 @@ func Execute(ctx context.Context, p *beam.Pipeline)
(beam.PipelineResult, error)
if err != nil {
return nil, errors.Wrap(err, "translation failed")
}
+ beam.PipelineOptions.LoadOptionsFromFlags(make(map[string]bool))
Review comment:
Oh nice - that's much cleaner, thanks for calling that option out!
Updated!
##########
File path: sdks/go/pkg/beam/core/runtime/options.go
##########
@@ -117,6 +118,23 @@ func (o *Options) Export() RawOptions {
return RawOptions{Options: copyMap(o.opt)}
}
+// LoadOptionsFromFlags adds any flags not defined in excludeFlags to the
options.
+// If the key is already defnined, it panics.
Review comment:
As I think about this more, I actually think this is probably not the
behavior we want anyways. If a pipeline author tries to write to
PipelineOptions with a value that also happens to be a flag, I think we want to
let them do that - we shouldn't steamroll their value. I've updated to just
ignore any keys that are already defined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 720454)
Time Spent: 1.5h (was: 1h 20m)
> [Go SDK] Export flags to PipelineOptions on non-dataflow runners.
> -----------------------------------------------------------------
>
> Key: BEAM-13024
> URL: https://issues.apache.org/jira/browse/BEAM-13024
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Danny McCormick
> Priority: P2
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Recently the Go SDK dataflow runner code began adding explicitly set flags to
> the pipeline options. This made them available to worker code at execution
> time, if queried with `beam.PipelineOptions.Get`.
> The proposal is to generalize this code and make use of it in all runners
> (including the direct runner). This would make PipelineOptions a useful
> concept in the Go SDK.
> Flag filtering and adding here:
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/dataflow/dataflow.go#L281]
> Once implemented for all runners, it should be possible to test use of this
> in the integration tests to ensure continued function.
> Danny wrote a quick Doc:
> https://docs.google.com/document/d/1AyO5SDEd_DzyOyrz_TkyDLUv19eLsKI97vKM8I7fQ9o/edit#
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
