[
https://issues.apache.org/jira/browse/BEAM-5720?focusedWorklogId=153980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153980
]
ASF GitHub Bot logged work on BEAM-5720:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Oct/18 18:29
Start Date: 12/Oct/18 18:29
Worklog Time Spent: 10m
Work Description: tvalentyn commented on a change in pull request #6659:
[BEAM-5720] Fix encoding of large python ints in Python 3.
URL: https://github.com/apache/beam/pull/6659#discussion_r224876803
##########
File path: sdks/python/apache_beam/coders/coder_impl.py
##########
@@ -293,8 +299,21 @@ def encode_to_stream(self, value, stream, nested):
if value is None:
stream.write_byte(NONE_TYPE)
elif t is int:
- stream.write_byte(INT_TYPE)
- stream.write_var_int64(value)
+ # In Python 3, an int may be larger than 64 bits.
+ # Note that an OverflowError on stream.write_var_int64 would happen
+ # *after* the marker byte is written, so we must check earlier.
+ try:
+ # This may throw an overflow error when compiled.
+ int_value = value
+ # Otherwise, we must check ourselves.
+ if not is_compiled:
+ if not fits_in_64_bits(value):
Review comment:
Thanks. Consider the following wording for comments:
```
# In Python 3, an int may be larger than 64 bits.
# We need to check whether value fits into a 64 bit integer before
# writing the marker byte.
try:
# In Cython-compiled code this will throw an overflow error
# when value does not fit into int64.
int_value = value
# If Cython is not used, we must do a (slower) check ourselves.
if not is_compiled:
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 153980)
Time Spent: 1h 10m (was: 1h)
> Default coder breaks with large ints on Python 3
> ------------------------------------------------
>
> Key: BEAM-5720
> URL: https://issues.apache.org/jira/browse/BEAM-5720
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Robert Bradshaw
> Assignee: Robert Bradshaw
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The test for `int` includes greater than 64-bit values, which causes an
> overflow error later in the code. We need to only use that coding scheme for
> machine-sized ints.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
