[jira] [Work logged] (BEAM-5509) Python pipeline_options doesn't handle int type
[
https://issues.apache.org/jira/browse/BEAM-5509?focusedWorklogId=149242&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-149242
]
ASF GitHub Bot logged work on BEAM-5509:
Author: ASF GitHub Bot
Created on: 28/Sep/18 17:46
Start Date: 28/Sep/18 17:46
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6514: [BEAM-5509] Fix int
pipeline option serialization in portable_runner
URL: https://github.com/apache/beam/pull/6514
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/runners/job/utils.py
b/sdks/python/apache_beam/runners/job/utils.py
index 5a247cd2729..3e347517972 100644
--- a/sdks/python/apache_beam/runners/job/utils.py
+++ b/sdks/python/apache_beam/runners/job/utils.py
@@ -27,7 +27,7 @@
def dict_to_struct(dict_obj):
- return json_format.Parse(json.dumps(dict_obj), struct_pb2.Struct())
+ return json_format.ParseDict(dict_obj, struct_pb2.Struct())
def struct_to_dict(struct_obj):
diff --git a/sdks/python/apache_beam/runners/portability/portable_runner.py
b/sdks/python/apache_beam/runners/portability/portable_runner.py
index ec537266324..15ae84d341a 100644
--- a/sdks/python/apache_beam/runners/portability/portable_runner.py
+++ b/sdks/python/apache_beam/runners/portability/portable_runner.py
@@ -136,7 +136,8 @@ def run_pipeline(self, pipeline):
del transform_proto.subtransforms[:]
# TODO: Define URNs for options.
-options = {'beam:option:' + k + ':v1': v
+# convert int values: https://issues.apache.org/jira/browse/BEAM-5509
+options = {'beam:option:' + k + ':v1': (str(v) if type(v) == int else v)
for k, v in pipeline._options.get_all_options().items()
if v is not None}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 149242)
Time Spent: 0.5h (was: 20m)
> Python pipeline_options doesn't handle int type
> ---
>
> Key: BEAM-5509
> URL: https://issues.apache.org/jira/browse/BEAM-5509
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability-flink
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The int option supplied at the command line is turned into a decimal during
> serialization and then the parser in SDK harness fails to restore it as int.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5509) Python pipeline_options doesn't handle int type
[ https://issues.apache.org/jira/browse/BEAM-5509?focusedWorklogId=148939&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148939 ] ASF GitHub Bot logged work on BEAM-5509: Author: ASF GitHub Bot Created on: 27/Sep/18 23:14 Start Date: 27/Sep/18 23:14 Worklog Time Spent: 10m Work Description: tweise opened a new pull request #6514: [BEAM-5509] Fix int/long pipeline option serialization in portable_runner URL: https://github.com/apache/beam/pull/6514 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 148939) Time Spent: 10m Remaining Estimate: 0h > Python pipeline_options doesn't handle int type > --- > > Key: BEAM-5509 > URL: https://issues.apache.org/jira/browse/BEAM-5509 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Thomas Weise >Assignee: Thomas We
[jira] [Work logged] (BEAM-5509) Python pipeline_options doesn't handle int type
[ https://issues.apache.org/jira/browse/BEAM-5509?focusedWorklogId=148940&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148940 ] ASF GitHub Bot logged work on BEAM-5509: Author: ASF GitHub Bot Created on: 27/Sep/18 23:14 Start Date: 27/Sep/18 23:14 Worklog Time Spent: 10m Work Description: tweise commented on issue #6514: [BEAM-5509] Fix int/long pipeline option serialization in portable_runner URL: https://github.com/apache/beam/pull/6514#issuecomment-425271502 R: @angoenka This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 148940) Time Spent: 20m (was: 10m) > Python pipeline_options doesn't handle int type > --- > > Key: BEAM-5509 > URL: https://issues.apache.org/jira/browse/BEAM-5509 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability-flink > Time Spent: 20m > Remaining Estimate: 0h > > The int option supplied at the command line is turned into a decimal during > serialization and then the parser in SDK harness fails to restore it as int. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
