[
https://issues.apache.org/jira/browse/BEAM-10274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-10274:
-----------------------------------
Resolution: Fixed
Status: Resolved (was: Resolved)
Hello! Due to a bug in our Jira configuration, this issue had status:Resolved
but resolution:Unresolved.
I am bulk editing these issues to have resolution:Fixed
If a different resolution is appropriate, please change it. To do this, click
the "Resolve" button (you can do this even for closed issues) and set the
Resolution field to the right value.
> Python SDK can't parse type=json.loads pipeline options at execution time
> -------------------------------------------------------------------------
>
> Key: BEAM-10274
> URL: https://issues.apache.org/jira/browse/BEAM-10274
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, sdk-py-harness
> Reporter: Brian Hulette
> Assignee: Robert Bradshaw
> Priority: P2
> Fix For: 2.24.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> It's pretty common to use `type=json.loads` in argparse to create JSON
> formatted options, in fact we have a couple in Beam:
> https://github.com/apache/beam/blob/61b665640d6c0f91751bba59782c0ac6aceacba6/sdks/python/apache_beam/options/pipeline_options.py#L431-L443
> https://github.com/apache/beam/blob/61b665640d6c0f91751bba59782c0ac6aceacba6/sdks/python/apache_beam/options/pipeline_options.py#L577-L586
> Attempting to access these options at pipeline execution time yields an error
> (note the single quotes):
> {code}
> argparse.ArgumentError: argument --beam_services: invalid loads value:
> "{'foo': 'bar'}"
> {code}
> Why does this happen?
> - sdk_worker_main.py received these values from the PIPELINE_OPTIONS env var
> which represents them as proper JSON:
> {code}
> ..., "some_option": "some_value", "json_option": {"foo": "bar"}}, ...
> {code}
> - The json is loaded and parsed with PipelineOptions.from_dictionary:
> https://github.com/apache/beam/blob/61b665640d6c0f91751bba59782c0ac6aceacba6/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L168-L181
> - from_dictionary just [writes out the value with
> str(v)|https://github.com/apache/beam/blob/61b665640d6c0f91751bba59782c0ac6aceacba6/sdks/python/apache_beam/options/pipeline_options.py#L241-L249].
> When the option is accessed we attempt re-parse it, and it's no longer valid
> JSON, so json.loads fails.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
