[
https://issues.apache.org/jira/browse/BEAM-10275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17145953#comment-17145953
]
Ankur Goenka commented on BEAM-10275:
-------------------------------------
Thanks for identifying this issue.
Based on the reference Jira BEAM-10274
I think the root cause of the issue is not serializing and deserializing the
options correctly.
There is value to validating all the options early as it gives clear failure
signal.
Should we try to address the correct desrialization of the options?
> sdk_worker_main.py eagerly parses pipeline options
> --------------------------------------------------
>
> Key: BEAM-10275
> URL: https://issues.apache.org/jira/browse/BEAM-10275
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> sdk_worker_main.py eagerly parses pipeline options beause of the call to
> get_all_options here:
> https://github.com/apache/beam/blob/61b665640d6c0f91751bba59782c0ac6aceacba6/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L138
> This can cause the worker to crash if any option that can't be read at
> execution is time is used, even if we don't need to access it at execution
> time (e.g. json.loads arguments, described in BEAM-10274)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
