[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=111261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111261 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 12/Jun/18 20:27 Start Date: 12/Jun/18 20:27 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-396722360 For the record, https://github.com/apache/beam/pull/5586 improves the performance of `dict.items()` with a Cython directive to use Python 3 interpretation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 111261) Time Spent: 20.5h (was: 20h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 20.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=110284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-110284 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 08/Jun/18 21:24 Start Date: 08/Jun/18 21:24 Worklog Time Spent: 10m Work Description: robertwb closed pull request #5586: [BEAM-3981] Cleanup of coders futurization. URL: https://github.com/apache/beam/pull/5586 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/coders/coder_impl.py b/sdks/python/apache_beam/coders/coder_impl.py index 2fcd0f22a31..78d692c1f76 100644 --- a/sdks/python/apache_beam/coders/coder_impl.py +++ b/sdks/python/apache_beam/coders/coder_impl.py @@ -15,6 +15,8 @@ # limitations under the License. # +# cython: language_level=3 + """Coder implementations. The actual encode/decode implementations are split off from coders to @@ -32,7 +34,6 @@ from __future__ import absolute_import from __future__ import division -import sys from builtins import chr from builtins import object @@ -310,13 +311,7 @@ def encode_to_stream(self, value, stream, nested): dict_value = value # for typing stream.write_byte(DICT_TYPE) stream.write_var_int64(len(dict_value)) - # Use iteritems() on Python 2 instead of future.builtins.iteritems to - # avoid performance regression in Cython compiled code. - if sys.version_info[0] == 2: -items = dict_value.iteritems() # pylint: disable=dict-iter-method - else: -items = dict_value.items() - for k, v in items: + for k, v in dict_value.items(): self.encode_to_stream(k, stream, True) self.encode_to_stream(v, stream, True) elif t is bool: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 110284) Time Spent: 20h 20m (was: 20h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 20h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=110014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-110014 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 08/Jun/18 07:18 Start Date: 08/Jun/18 07:18 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5586: [BEAM-3981] Cleanup of coders futurization. URL: https://github.com/apache/beam/pull/5586#issuecomment-395671954 Thanks! LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 110014) Time Spent: 20h 10m (was: 20h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 20h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109948 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 08/Jun/18 01:19 Start Date: 08/Jun/18 01:19 Worklog Time Spent: 10m Work Description: robertwb commented on issue #5586: [BEAM-3981] Cleanup of coders futurization. URL: https://github.com/apache/beam/pull/5586#issuecomment-395615484 R: @RobbeSneyders, @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 109948) Time Spent: 20h (was: 19h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 20h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109947 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 08/Jun/18 01:18 Start Date: 08/Jun/18 01:18 Worklog Time Spent: 10m Work Description: robertwb opened a new pull request #5586: [BEAM-3981] Cleanup of coders futurization. URL: https://github.com/apache/beam/pull/5586 Specifying language_level=3 allows the dict.items() optimization. This is about a 40% speedup for 1-element dicts as well as cleaner code. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 109947) Time Spent: 19h 50m (was: 19h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 19h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109884 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 07/Jun/18 22:05 Start Date: 07/Jun/18 22:05 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-395580375 Hmm. I am pretty sure my microbenchmark uses a Cython codepath since in order for any code change to take effect I have to run `python setup.py build_ext --inplace` to recompile associated C extensions. I checked once again and I do see a 2x slowdown with `items()` once the size of the dictionary goes more than 10 elements. Here's code generated by Cython: a https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.drcukhvo4hd6. Perhaps the slowdown is related to materializing the list of keys? My microbenchmark setup is here: https://github.com/apache/beam/compare/master...tvalentyn:coders_dict_microbencmark?expand=1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 109884) Time Spent: 19h 40m (was: 19.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 19h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109883 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 07/Jun/18 22:05 Start Date: 07/Jun/18 22:05 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-395580375 Hmm. I am pretty sure my microbenchmark uses a Cython codepath since in order for any code change to take effect I have to run `python setup.py build_ext --inplace` to recompile associated C extensions. I checked once again and I do see a 2x slowdown with `items()` once the size of the dictionary goes more than 10 elements. Here's code generated by Cython: a https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.drcukhvo4hd6. Perhaps the slowdown is related to materializing the list? My microbenchmark setup is here: https://github.com/apache/beam/compare/master...tvalentyn:coders_dict_microbencmark?expand=1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 109883) Time Spent: 19.5h (was: 19h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 19.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109571 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 06/Jun/18 22:41 Start Date: 06/Jun/18 22:41 Worklog Time Spent: 10m Work Description: robertwb commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-395236495 `dict.items()` should have the same performance when compiled with Cython, as it unpacks it into item-iterating code without the function call (or intermediate object generation). Having two cases for Py2 and Py3 will break this optimization (as well as the fact that the case checking code could be (relatively) expensive for small dicts.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 109571) Time Spent: 19h 20m (was: 19h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 19h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109566 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 06/Jun/18 22:08 Start Date: 06/Jun/18 22:08 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-395229849 An update on `dict.iteritems` vs `dict.items()` vs `future.utils.iteritems(dict)` - I did more performance testing of encode-decode operation using a microbenchmark (currently in flight: https://github.com/apache/beam/pull/5565). I don't observe a difference in performance of `dict.iteritems()` and `future.utils.iteritems(dict)`. As far as `dict.items()` vs `dict.iteritems()` goes, I saw a 2x performance slowdown in coder implementation with dict.items() for dictionaries with over 10 entries, but did not observe a significant difference on dictionaries with 1 entries or less. That said I think it would not hurt to keep using `iteritems()` for Python 2 as we do now. With `future.utils.iteritems()`: ``` Median time cost: Dict[int, int], FastPrimitiveCoder : per element median time cost: 3.27529e-07 sec ``` With `dict.iteritems()`: ``` Median time cost: Dict[int, int], FastPrimitiveCoder : per element median time cost: 3.4485e-07 sec ``` With `dict.items()`: ``` Median time cost: Dict[int, int], FastPrimitiveCoder : per element median time cost: 7.3393e-07 sec ``` I also observe a 2.5x degradation in coder implementation with `builtins.range()` compared to `range()` on lists as small as 1000 - 1 elements. I did not try smaller lists. With 1 elements, python 2 `range()`: ``` Median time cost: List[int], FastPrimitiveCoder : per element median time cost: 1.17695e-07 sec ``` With `builtins.range():` ``` Median time cost: List[int], FastPrimitiveCoder : per element median time cost: 3.22402e-07 sec ``` We should try to use microbenchmarks for performance evaluations moving forward since they can provide feedback in a matter of seconds. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 109566) Time Spent: 19h 10m (was: 19h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 19h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=102758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102758 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/May/18 02:13 Start Date: 17/May/18 02:13 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r188822509 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -210,11 +216,15 @@ def as_cloud_object(self): def __repr__(self): return self.__class__.__name__ + # pylint: disable=protected-access def __eq__(self, other): -# pylint: disable=protected-access return (self.__class__ == other.__class__ and self._dict_without_impl() == other._dict_without_impl()) -# pylint: enable=protected-access + + def __hash__(self): Review comment: That's true. Although that wouldn't honor the contract between __eq__ and __hash__. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 102758) Time Spent: 19h (was: 18h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 19h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=102673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102673 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 16/May/18 21:38 Start Date: 16/May/18 21:38 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r188780116 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -210,11 +216,15 @@ def as_cloud_object(self): def __repr__(self): return self.__class__.__name__ + # pylint: disable=protected-access def __eq__(self, other): -# pylint: disable=protected-access return (self.__class__ == other.__class__ and self._dict_without_impl() == other._dict_without_impl()) -# pylint: enable=protected-access + + def __hash__(self): Review comment: We can also use the id which is guaranteed to stay the same for an object: `hash(id(self))` The default Python 2 hash also relies on id. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 102673) Time Spent: 18h 50m (was: 18h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 18h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=102672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102672 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 16/May/18 21:35 Start Date: 16/May/18 21:35 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r188780116 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -210,11 +216,15 @@ def as_cloud_object(self): def __repr__(self): return self.__class__.__name__ + # pylint: disable=protected-access def __eq__(self, other): -# pylint: disable=protected-access return (self.__class__ == other.__class__ and self._dict_without_impl() == other._dict_without_impl()) -# pylint: enable=protected-access + + def __hash__(self): Review comment: We can also explicitly implement the default python 2 hash, which is guaranteed to stay the same for an object: `hash(id(self))` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 102672) Time Spent: 18h 40m (was: 18.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 18h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=102666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102666 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 16/May/18 21:25 Start Date: 16/May/18 21:25 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r18897 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -210,11 +216,15 @@ def as_cloud_object(self): def __repr__(self): return self.__class__.__name__ + # pylint: disable=protected-access def __eq__(self, other): -# pylint: disable=protected-access return (self.__class__ == other.__class__ and self._dict_without_impl() == other._dict_without_impl()) -# pylint: enable=protected-access + + def __hash__(self): Review comment: Revisiting this in light of other PRs. I think, it would be safer to guarantee the contract that hash does not change for the same object if we compute it here based on object type, sent https://github.com/apache/beam/pull/5390. Another possibility to guarantee consistent behavior between Python 2 and 3 would be to set `__hash__ = None` if we can infer that a class is obviously non-hashable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 102666) Time Spent: 18.5h (was: 18h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Fix For: Not applicable > > Time Spent: 18.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100665
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 10/May/18 16:09
Start Date: 10/May/18 16:09
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #5053: [BEAM-3981] Futurize
coders subpackage
URL: https://github.com/apache/beam/pull/5053
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/coders/__init__.py
b/sdks/python/apache_beam/coders/__init__.py
index acca89f70f4..3192494ebbf 100644
--- a/sdks/python/apache_beam/coders/__init__.py
+++ b/sdks/python/apache_beam/coders/__init__.py
@@ -14,6 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from __future__ import absolute_import
from apache_beam.coders.coders import *
from apache_beam.coders.typecoders import registry
diff --git a/sdks/python/apache_beam/coders/coder_impl.pxd
b/sdks/python/apache_beam/coders/coder_impl.pxd
index 98dd508556a..8a85d085448 100644
--- a/sdks/python/apache_beam/coders/coder_impl.pxd
+++ b/sdks/python/apache_beam/coders/coder_impl.pxd
@@ -70,11 +70,11 @@ cdef class DeterministicFastPrimitivesCoderImpl(CoderImpl):
cdef object NoneType
cdef char UNKNOWN_TYPE, NONE_TYPE, INT_TYPE, FLOAT_TYPE, BOOL_TYPE
-cdef char STR_TYPE, UNICODE_TYPE, LIST_TYPE, TUPLE_TYPE, DICT_TYPE, SET_TYPE
+cdef char BYTES_TYPE, UNICODE_TYPE, LIST_TYPE, TUPLE_TYPE, DICT_TYPE, SET_TYPE
cdef class FastPrimitivesCoderImpl(StreamCoderImpl):
cdef CoderImpl fallback_coder_impl
- @cython.locals(unicode_value=unicode, dict_value=dict)
+ @cython.locals(dict_value=dict)
cpdef encode_to_stream(self, value, OutputStream stream, bint nested)
diff --git a/sdks/python/apache_beam/coders/coder_impl.py
b/sdks/python/apache_beam/coders/coder_impl.py
index cc7ed87c3ad..2fcd0f22a31 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -24,13 +24,17 @@
This module may be optionally compiled with Cython, using the corresponding
coder_impl.pxd file for type hints.
+Py2/3 porting: Native range is used on both python versions instead of
+future.builtins.range to avoid performance regression in Cython compiled code.
+
For internal use only; no backwards-compatibility guarantees.
"""
from __future__ import absolute_import
+from __future__ import division
-from types import NoneType
-
-import six
+import sys
+from builtins import chr
+from builtins import object
from apache_beam.coders import observable
from apache_beam.utils import windowed_value
@@ -54,10 +58,12 @@
from .slow_stream import get_varint_size
# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
-try:
- long# Python 2
-except NameError:
- long = int # Python 3
+try:# Python 2
+ long # pylint: disable=long-builtin
+ unicode # pylint: disable=unicode-builtin
+except NameError: # Python 3
+ long = int
+ unicode = str
class CoderImpl(object):
@@ -199,7 +205,7 @@ def __init__(self, coder, step_label):
self._step_label = step_label
def _check_safe(self, value):
-if isinstance(value, (str, six.text_type, long, int, float)):
+if isinstance(value, (bytes, unicode, long, int, float)):
pass
elif value is None:
pass
@@ -253,7 +259,7 @@ def decode(self, encoded):
NONE_TYPE = 0
INT_TYPE = 1
FLOAT_TYPE = 2
-STR_TYPE = 3
+BYTES_TYPE = 3
UNICODE_TYPE = 4
BOOL_TYPE = 9
LIST_TYPE = 5
@@ -279,7 +285,7 @@ def get_estimated_size_and_observables(self, value,
nested=False):
def encode_to_stream(self, value, stream, nested):
t = type(value)
-if t is NoneType:
+if value is None:
stream.write_byte(NONE_TYPE)
elif t is int:
stream.write_byte(INT_TYPE)
@@ -287,13 +293,13 @@ def encode_to_stream(self, value, stream, nested):
elif t is float:
stream.write_byte(FLOAT_TYPE)
stream.write_bigendian_double(value)
-elif t is str:
- stream.write_byte(STR_TYPE)
+elif t is bytes:
+ stream.write_byte(BYTES_TYPE)
stream.write(value, nested)
-elif t is six.text_type:
- unicode_value = value # for typing
+elif t is unicode:
+ text_value = value # for typing
stream.write_byte(UNICODE_TYPE)
- stream.write(unicode_value.encode('utf-8'), nested)
+ stream.write(text_value.encode('utf-8'), nested)
elif t is list or t is tuple or t is set:
stream.write_byte(
LIST_TYPE if t is list
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100664 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 10/May/18 16:08 Start Date: 10/May/18 16:08 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-388101551 We chatted with @tvalentyn. The test failure is unrelated and @tvalentyn will implement @robertwb's suggestions in a follow up PR. Merging this now to unblock progress. Thank you @RobbeSneyders and @tvalentyn for pushing it thus far! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100664) Time Spent: 18h 10m (was: 18h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 18h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100309 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 22:44 Start Date: 09/May/18 22:44 Worklog Time Spent: 10m Work Description: robertwb commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387897325 Regarding items vs. iteritems, let's just go with dict_value.items everywhere. There's little if any need for explicit iteritems in this case (and if dict is typed, I think Cython optimizes `for k, v in dict_value.items()` optimally without the intermediate list anyways). Though macrobenchmarks can be good for final validation, this is the kind of code that could probably benefit from some pretty simple microbenchmarks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100309) Time Spent: 18h (was: 17h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 18h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100265=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100265 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 21:28 Start Date: 09/May/18 21:28 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387881232 ValidatesRunner suite just takes time to start. It failed due to an unrelated issue, filed: https://issues.apache.org/jira/browse/INFRA-16508 to clarify. I'll do a sanity check locally. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100265) Time Spent: 17h 50m (was: 17h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 17h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100165 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 17:57 Start Date: 09/May/18 17:57 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387823211 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100165) Time Spent: 17h 40m (was: 17.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 17h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100163 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 17:53 Start Date: 09/May/18 17:53 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387821931 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100163) Time Spent: 17.5h (was: 17h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 17.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100161 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 17:53 Start Date: 09/May/18 17:53 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387821705 @RobbeSneyders Thank you for the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100161) Time Spent: 17h 20m (was: 17h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 17h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100160 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 17:52 Start Date: 09/May/18 17:52 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387821627 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100160) Time Spent: 17h 10m (was: 17h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 17h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100141 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 17:19 Start Date: 09/May/18 17:19 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387811751 I have added comments in the file and added a remark in the documentation of the used approach. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100141) Time Spent: 17h (was: 16h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 17h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100078 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 16:36 Start Date: 09/May/18 16:36 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387798512 No, apparently the linter does not check for the `range` import, it only checks for wrong usage of `range()`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100078) Time Spent: 16h 50m (was: 16h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100074=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100074 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 16:33 Start Date: 09/May/18 16:33 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387798829 Can you please add a comment in the file and/or PR description that we intentionally did not import range and use iteritems() suspecting possible performance regression in compiled Cython code? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100074) Time Spent: 16h 40m (was: 16.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100072 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 16:32 Start Date: 09/May/18 16:32 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387798512 No, appareantly the linter does not check for the `range` import, it only checks for wrong usage of `range()`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100072) Time Spent: 16.5h (was: 16h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100069 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 16:28 Start Date: 09/May/18 16:28 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387797099 Thanks Robbe, the change looks good to me. I noticed we did not whitelist coders_impl.py from linter given that import of range was missing(). So, linter does not require this import? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100069) Time Spent: 16h 20m (was: 16h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=100058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-100058 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/May/18 16:08 Start Date: 09/May/18 16:08 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387791150 Ok, great. I have pushed the new changes. Seems like the javaPreCommit failing is unrelated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 100058) Time Spent: 16h 10m (was: 16h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=99719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99719 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 08/May/18 20:50 Start Date: 08/May/18 20:50 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387538220 @RobbeSneyders I think it's fine to make one-off exceptions for coders_impl.py, and annotating them with inline exceptions from lint checker for the two changes we discussed. We don't expect these kind of exceptions for most other parts of codebase, but wanted to do our due diligence on this package since it's on critical path. I agree with you, let's remove the `range()` import. For `iteritems` how about we do something along the lines: ``` if (...python 2...): items = dict_value.iteritems() # no-py3lint: allow iteritems() else: items = dict_value.items() for k,v in items: ... ``` I think with these changes we can proceed with the merge, I will do benchmarking in parallel and we can clean this up later. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 99719) Time Spent: 16h (was: 15h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=99496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99496 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 08/May/18 12:44 Start Date: 08/May/18 12:44 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-387389750 I think we can remove the `range()` import from coders_impl.py without changing any behavior. We will have to disable the lint check for this file though. An alternative to the iteritems() import is using `for k, v in dict_value.items()`. This will have improved performance in Python 3, but will return a list on Python 2. This might however still perform better with Cython compared to the `iteritems()` import. Could you check this, @tvalentyn? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 99496) Time Spent: 15h 50m (was: 15h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=97619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97619 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/May/18 18:06 Start Date: 02/May/18 18:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-386068489 Based on the information I currently have, my best guess is that regression is caused by changes in Cython's interpretation of Py3 implementations of range() and iteritems() in coder_impl.py. We can see a difference in code generated by Cython here: https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.xkt18qn9ofn1 I am seeing 7% regression in the median on the benchmarks I've been running, and I was able to see the regression go away with these changes reverted. We can also try to benchmark these methods in isolation to confirm and rule out the variance of test setup and runner componets. I might not have time to do it this week. @robertwb , do you think anything can be done to improve Cython translation here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97619) Time Spent: 15h 40m (was: 15.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=97618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97618 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/May/18 18:05 Start Date: 02/May/18 18:05 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-386068489 Based on the information I currently have, my best guess is that regression is caused by changes in Cython's interpretation of Py3 implementations of range() and iteritems() in coder_impl.py. We can see a difference in code generated by Cython here: https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.xkt18qn9ofn1 I am seeing 7% regression in the median on the benchmarks I've been running, and I was able to see the regression go away with these changes reverted. We can also try to benchmark these methods in isolation to confirm and rule out the variance of test setup. I might not have time to do it this week. @robertwb , do you think anything can be done to improve Cython translation here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97618) Time Spent: 15.5h (was: 15h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=96733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96733 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 30/Apr/18 15:38 Start Date: 30/Apr/18 15:38 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-385437438 Ok thank you @tvalentyn. A small performance regression in Python 2 can be expected with the chosen approach. With the use of the future package, we focus more on writing native Python 3 and backporting this to Python 2. I believe this is the best choice since Python 3 will provide a performance improvement and support for Python 2 will be dropped in less than 2 years. I have applied the changes proposed by @robertwb, which will hopefully reduce the amount of regression. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96733) Time Spent: 15h 20m (was: 15h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=96548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96548 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 30/Apr/18 05:36 Start Date: 30/Apr/18 05:36 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-385316601 I have tried to revert several statements that might trigger a regression (hash functions, dropped typing annotation in cython code, explicit encode() statements, and recent commit). I also started a suite with more wide reverts. I also plan to take a look at generated cython code by robertwb's suggestion. Current suite I am running takes around 8hours to complete, which reduces feedback look significantly. If I still don't see the culprit tomorrow, I might try to reproduce the regression in a microbenchmark that specifically targets the code we are modifying code to reduce the feedback loop. There's not that many things left to revert. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96548) Time Spent: 15h 10m (was: 15h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=96283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96283 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 27/Apr/18 23:47 Start Date: 27/Apr/18 23:47 Worklog Time Spent: 10m Work Description: robertwb commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r184828723 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -253,8 +258,8 @@ def decode(self, encoded): NONE_TYPE = 0 INT_TYPE = 1 FLOAT_TYPE = 2 -STR_TYPE = 3 -UNICODE_TYPE = 4 +BYTES_TYPE = 3 +TEXT_TYPE = 4 Review comment: Also, any renaming here should be reflected in the .pxd file. That may account for some of the perf issues. Running "python -m cython -a apache_beam/coders/coder_impl.py" could be useful for debugging this too. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96283) Time Spent: 15h (was: 14h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=96282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96282 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 27/Apr/18 23:47 Start Date: 27/Apr/18 23:47 Worklog Time Spent: 10m Work Description: robertwb commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r184828680 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -253,8 +258,8 @@ def decode(self, encoded): NONE_TYPE = 0 INT_TYPE = 1 FLOAT_TYPE = 2 -STR_TYPE = 3 -UNICODE_TYPE = 4 +BYTES_TYPE = 3 +TEXT_TYPE = 4 Review comment: I would leave this as UNICODE_TYPE. TEXT_TYPE is ambiguous wrt. Python 2 vs. 3. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96282) Time Spent: 15h (was: 14h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=95281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95281 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 25/Apr/18 22:18 Start Date: 25/Apr/18 22:18 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-384451792 After collecting additional datapoints on the same benchmark, the regression seems to be on the order of 7%, I am doing more A/B testing to confirm the exact changes that contribute to it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95281) Time Spent: 14h 50m (was: 14h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=95003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95003 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 25/Apr/18 11:35 Start Date: 25/Apr/18 11:35 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-384255784 Ok, thank you. PRs of other packages are blocked by this one because of the added lint tests and the cython upgrade. I have already ported 3 extra packages which are waiting to be submitted, but I have paused extra porting work so I won't have to redo too much work if anything changes. When this PR is merged, I will reach out to the mailing list to work on other packages in parallel so we can move forward quickly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95003) Time Spent: 14h 40m (was: 14.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=94946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94946 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 25/Apr/18 06:26 Start Date: 25/Apr/18 06:26 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-384174207 @RobbeSneyders I should have an update on performance later today. We'd like to make sure we understand if there are any performance implication since this code is on the data path. I am hoping that the merge is not blocking you from other py3 efforts, but let me know if this is a hard dependency for you at the moment. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94946) Time Spent: 14.5h (was: 14h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=94350=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94350 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 23/Apr/18 22:55 Start Date: 23/Apr/18 22:55 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-383748311 Cython issue has a workaround and is no longer a blocker. I observed a 15% performance degradation on Dataflow runner with this PR & Cython upgrade, however this was a one-off benchmark run and the data is inconclusive. I am starting multiple runs to confirm or rule out the regression here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94350) Time Spent: 14h 20m (was: 14h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=93448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93448 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 20/Apr/18 21:18 Start Date: 20/Apr/18 21:18 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-383225525 I have noticed a breaking change in Cython 0.28.2 which currently affects another part of the codebase. I brought it up here: https://github.com/cython/cython/issues/2132#issuecomment-383190578. If possible, I'd like to resolve it first, since with this PR requires upgrading to Cython 0.28+ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93448) Time Spent: 14h 10m (was: 14h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=93239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93239 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 20/Apr/18 14:57 Start Date: 20/Apr/18 14:57 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-383123596 ping @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93239) Time Spent: 14h (was: 13h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92147 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 18/Apr/18 15:45 Start Date: 18/Apr/18 15:45 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382433570 Looks like this is ready to merge, @aaltay ! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92147) Time Spent: 13h 50m (was: 13h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 13h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92127=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92127 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 18/Apr/18 14:42 Start Date: 18/Apr/18 14:42 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382411484 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92127) Time Spent: 13h 40m (was: 13.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 13h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92126 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 18/Apr/18 14:41 Start Date: 18/Apr/18 14:41 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382411187 Alright, Mr. Jenkins, be kind and patient to us: Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92126) Time Spent: 13.5h (was: 13h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 13.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92049 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 18/Apr/18 06:37 Start Date: 18/Apr/18 06:37 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382278654 https://github.com/apache/beam/pull/5163 should fix Jenkins build, let's rerun the tests again once that is merged. We can continue working on other Python-3 changes in parallel, I don't anticipate changes to this PR at this point but would like to get a signal from ValidatesRunner suites. Happy to test manually if Jenkins build does not get resolve within next day. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92049) Time Spent: 13h 20m (was: 13h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 13h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92023 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 18/Apr/18 01:36 Start Date: 18/Apr/18 01:36 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382215896 @tvalentyn if the test issues are unrelated to this PR and related to the current ongoing Jenkins issues should we move forward with it? (I suggest that we run a few manual tests locally to check whether this PR introduces issues or not.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92023) Time Spent: 13h 10m (was: 13h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 13h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92020 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 18/Apr/18 01:29 Start Date: 18/Apr/18 01:29 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382214475 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92020) Time Spent: 13h (was: 12h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 13h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91944 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/Apr/18 21:44 Start Date: 17/Apr/18 21:44 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382158455 Run Python Precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91944) Time Spent: 12h 50m (was: 12h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 12h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91925 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/Apr/18 21:15 Start Date: 17/Apr/18 21:15 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382149090 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91925) Time Spent: 12h 40m (was: 12.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 12h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91916=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91916 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/Apr/18 20:35 Start Date: 17/Apr/18 20:35 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382132920 Yes, please squash the commits; I am looking into postcommit issue, hopefully that does not affect this PR. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91916) Time Spent: 12h 20m (was: 12h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 12h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91917=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91917 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/Apr/18 20:35 Start Date: 17/Apr/18 20:35 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-382133008 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91917) Time Spent: 12.5h (was: 12h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 12.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91664 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/Apr/18 08:12 Start Date: 17/Apr/18 08:12 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381892234 I've rebased the branch anyway. Is there anything else I should do before we can merge? Like squash some commits? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91664) Time Spent: 12h 10m (was: 12h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 12h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91563=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91563 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/Apr/18 01:10 Start Date: 17/Apr/18 01:10 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381797547 There is a proto generation error in the ValidatesRunner tests, which seems to be unrelated to this PR, I also see that error in other Postcommit test suites. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91563) Time Spent: 12h (was: 11h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 12h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91561=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91561 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 17/Apr/18 01:02 Start Date: 17/Apr/18 01:02 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381796316 Actually, looking at Jenkins logs I see that Jenkins already merges this PR with latest commit on master when we run a Postcommit suite: > Checking out Revision 1b8df077a60fc0188d786a146d5c5edb9eb2732f (refs/remotes/origin/pr/5053/merge) > > git config core.sparsecheckout # timeout=10 > > git checkout -f 1b8df077a60fc0188d786a146d5c5edb9eb2732f > Commit message: "Merge 6909ff3d2b1802c0712d8236692c0c7110d9b250 into e1c526d7f88add0aa44635a56223a4907b81e86b" > ... > This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91561) Time Spent: 11h 50m (was: 11h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 11h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91552 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 16/Apr/18 23:47 Start Date: 16/Apr/18 23:47 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381784452 With #5131 merged, we probably need to rebase this PR off the current master, for the ValidatesRunner tests to pass. @RobbeSneyders can we do that please? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91552) Time Spent: 11h 40m (was: 11.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 11h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91551 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 16/Apr/18 23:46 Start Date: 16/Apr/18 23:46 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381784452 With #5131 merged, we probably need to rebase this PR off the current masters, for the ValidatesRunner tests to pass. @RobbeSneyders can we do that please? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91551) Time Spent: 11.5h (was: 11h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 11.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91550 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 16/Apr/18 23:43 Start Date: 16/Apr/18 23:43 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381783927 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91550) Time Spent: 11h 20m (was: 11h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 11h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=91054=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91054 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 14/Apr/18 04:25 Start Date: 14/Apr/18 04:25 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381302368 Containers are released and PR #5131 is out to upgrade them on master, we can also apply the changes from that PR here as well and rerun the postcommit suite. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91054) Time Spent: 11h 10m (was: 11h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 11h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90914 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 18:00 Start Date: 13/Apr/18 18:00 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381215630 I see the following error in the output ``` I /tmp/pip-cVZPmP-build/setup.py:79: UserWarning: You are using version 0.27.3 of cython. However, version 0.28.1 is recommended. I_CYTHON_VERSION, REQUIRED_CYTHON_VERSION I I Error compiling Cython file: I I ... I except NameError: # Python 3 Ilong = int Iunicode = str I I I class CoderImpl(object): I ^ I I I apache_beam/coders/coder_impl.py:68:0: 'object' is not a type name I Compiling apache_beam/coders/stream.pyx because it changed. I Compiling apache_beam/runners/worker/statesampler_fast.pyx because it changed. I Compiling apache_beam/coders/coder_impl.py because it changed. I Compiling apache_beam/metrics/execution.py because it changed. I Compiling apache_beam/runners/common.py because it changed. I Compiling apache_beam/runners/worker/logger.py because it changed. I Compiling apache_beam/runners/worker/opcounters.py because it changed. I Compiling apache_beam/runners/worker/operations.py because it changed. I Compiling apache_beam/transforms/cy_combiners.py because it changed. I Compiling apache_beam/utils/counters.py because it changed. I Compiling apache_beam/utils/windowed_value.py because it changed. I [ 1/11] Cythonizing apache_beam/coders/coder_impl.py I Traceback (most recent call last): IFile "", line 1, in IFile "/tmp/pip-cVZPmP-build/setup.py", line 204, in I 'apache_beam/utils/windowed_value.py', IFile "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 1039, in cythonize I cythonize_one(*args) IFile "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 1161, in cythonize_one I raise CompileError(None, pyx_file) I Cython.Compiler.Errors.CompileError: apache_beam/coders/coder_impl.py I I ``` I assume this is because The new code requires the new Cython version however dataflow workers do not have it. @tvalentyn Could you upgrade the workers at head to use the 0.28.1 cython version? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90914) Time Spent: 11h (was: 10h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90844 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 15:27 Start Date: 13/Apr/18 15:27 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381171611 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90844) Time Spent: 10h 50m (was: 10h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90843 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 15:25 Start Date: 13/Apr/18 15:25 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381171057 Thank you Robbe! The PR and approach look good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90843) Time Spent: 10h 40m (was: 10.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90837
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 13/Apr/18 15:01
Start Date: 13/Apr/18 15:01
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on a change in pull request
#5053: [BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181416218
##
File path: sdks/python/run_pylint.sh
##
@@ -118,4 +99,4 @@ if [ -n "${TESTS_MISSING_MAIN}" ]; then
done
echo
exit 1
-fi
+fi
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90837)
Time Spent: 10.5h (was: 10h 20m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90836
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 13/Apr/18 14:50
Start Date: 13/Apr/18 14:50
Worklog Time Spent: 10m
Work Description: tvalentyn commented on a change in pull request #5053:
[BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181412862
##
File path: sdks/python/run_pylint.sh
##
@@ -118,4 +99,4 @@ if [ -n "${TESTS_MISSING_MAIN}" ]; then
done
echo
exit 1
-fi
+fi
Review comment:
Let's add a new line at the end of file here and in
sdks/python/run_pylint_2to3.sh.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90836)
Time Spent: 10h 20m (was: 10h 10m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90833 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 14:41 Start Date: 13/Apr/18 14:41 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381157301 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90833) Time Spent: 10h 10m (was: 10h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90745 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 08:46 Start Date: 13/Apr/18 08:46 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381067548 Thanks for the reviews @charlesccychen, @tvalentyn. I've tried to address all of your comments and have committed some changes based on your input. Please have another look if everything seems ok now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90745) Time Spent: 10h (was: 9h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 10h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90743 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 08:42 Start Date: 13/Apr/18 08:42 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-381067548 Thanks for the reviews @charlesccychen, @tvalentyn. I've tried to address all of your comments and have committed some changes based on your input. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90743) Time Spent: 9h 50m (was: 9h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90738
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 13/Apr/18 08:35
Start Date: 13/Apr/18 08:35
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on a change in pull request
#5053: [BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181320040
##
File path: sdks/python/apache_beam/coders/coders_test_common.py
##
@@ -129,7 +130,7 @@ def test_dill_coder(self):
def test_fast_primitives_coder(self):
coder = coders.FastPrimitivesCoder(coders.SingletonCoder(len))
-self.check_coder(coder, None, 1, -1, 1.5, 'str\0str', u'unicode\0\u0101')
+self.check_coder(coder, None, 1, -1, 1.5, b'str\0str', u'unicode\0\u0101')
self.check_coder(coder, (), (1, 2, 3))
self.check_coder(coder, [], [1, 2, 3])
self.check_coder(coder, dict(), {'a': 'b'}, {0: dict(), 1: len})
Review comment:
See comment above.
The unmarked 'a' and 'b' here represent `str` on both versions, which seems
ok for this test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90738)
Time Spent: 9h 40m (was: 9.5h)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90737 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 08:33 Start Date: 13/Apr/18 08:33 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r181319580 ## File path: sdks/python/apache_beam/coders/coders_test_common.py ## @@ -103,7 +104,7 @@ def test_custom_coder(self): self.check_coder(CustomCoder(), 1, -10, 5) self.check_coder(coders.TupleCoder((CustomCoder(), coders.BytesCoder())), - (1, 'a'), (-10, 'b'), (5, 'c')) + (1, b'a'), (-10, b'b'), (5, b'c')) Review comment: I guess this is aimed at the 'a' in line 109? The marking of the strings as bytes literals is done manually. I've only marked strings as bytes literals when it's clear that they're meant to represent bytes (when testing BytesCoder, when the content of the string are clearly bytes, ...). When a string is not marked, it represents `str` on both versions, which seems ok for the 'a' at line 109 for example. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90737) Time Spent: 9.5h (was: 9h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90733
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 13/Apr/18 08:09
Start Date: 13/Apr/18 08:09
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on a change in pull request
#5053: [BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181314010
##
File path: sdks/python/tox.ini
##
@@ -88,6 +89,21 @@ commands =
pip --version
time {toxinidir}/run_pylint.sh
+[testenv:py27-lint3]
Review comment:
py27-lint3 checks for portability issues, while py3-lint checks for python 3
issues. I'll add a comment to clarify.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90733)
Time Spent: 9h 20m (was: 9h 10m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90730 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 08:05 Start Date: 13/Apr/18 08:05 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r181312959 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -210,11 +216,15 @@ def as_cloud_object(self): def __repr__(self): return self.__class__.__name__ + # pylint: disable=protected-access def __eq__(self, other): -# pylint: disable=protected-access return (self.__class__ == other.__class__ and self._dict_without_impl() == other._dict_without_impl()) -# pylint: enable=protected-access + + def __hash__(self): Review comment: On Python 2, `__hash__` defaults to give a different value for each instance. On Python 3, `__hash__` defaults to `None` if `__eq__` is implemented. By implementing `__hash__`, we get consistent behavior on both versions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90730) Time Spent: 9h 10m (was: 9h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90711 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 06:38 Start Date: 13/Apr/18 06:38 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r181296567 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -210,11 +216,15 @@ def as_cloud_object(self): def __repr__(self): return self.__class__.__name__ + # pylint: disable=protected-access def __eq__(self, other): -# pylint: disable=protected-access return (self.__class__ == other.__class__ and self._dict_without_impl() == other._dict_without_impl()) -# pylint: enable=protected-access + + def __hash__(self): Review comment: Is this change required by Python3 migration or we are just fixing an omission that __hash__ was not previously defined, while __eq__ was? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90711) Time Spent: 9h (was: 8h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90709
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 13/Apr/18 06:38
Start Date: 13/Apr/18 06:38
Worklog Time Spent: 10m
Work Description: tvalentyn commented on a change in pull request #5053:
[BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181296766
##
File path: sdks/python/tox.ini
##
@@ -88,6 +89,21 @@ commands =
pip --version
time {toxinidir}/run_pylint.sh
+[testenv:py27-lint3]
Review comment:
Can we add a comment how this is different from py3-lint? Or perhaps we
don't need both of them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90709)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90708
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 13/Apr/18 06:38
Start Date: 13/Apr/18 06:38
Worklog Time Spent: 10m
Work Description: tvalentyn commented on a change in pull request #5053:
[BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181296673
##
File path: sdks/python/apache_beam/coders/coders_test_common.py
##
@@ -129,7 +130,7 @@ def test_dill_coder(self):
def test_fast_primitives_coder(self):
coder = coders.FastPrimitivesCoder(coders.SingletonCoder(len))
-self.check_coder(coder, None, 1, -1, 1.5, 'str\0str', u'unicode\0\u0101')
+self.check_coder(coder, None, 1, -1, 1.5, b'str\0str', u'unicode\0\u0101')
self.check_coder(coder, (), (1, 2, 3))
self.check_coder(coder, [], [1, 2, 3])
self.check_coder(coder, dict(), {'a': 'b'}, {0: dict(), 1: len})
Review comment:
Also here 'a' and 'b' are not bytestrings.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90708)
Time Spent: 8h 40m (was: 8.5h)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90710 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 13/Apr/18 06:38 Start Date: 13/Apr/18 06:38 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r181296626 ## File path: sdks/python/apache_beam/coders/coders_test_common.py ## @@ -103,7 +104,7 @@ def test_custom_coder(self): self.check_coder(CustomCoder(), 1, -10, 5) self.check_coder(coders.TupleCoder((CustomCoder(), coders.BytesCoder())), - (1, 'a'), (-10, 'b'), (5, 'c')) + (1, b'a'), (-10, b'b'), (5, b'c')) Review comment: Probably not critical, but looks like 'a' is not replaced with b'a' here - are these changes done by some tool or manually? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90710) Time Spent: 8h 50m (was: 8h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90373 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 12/Apr/18 10:55 Start Date: 12/Apr/18 10:55 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r181040977 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -304,7 +307,7 @@ def encode_to_stream(self, value, stream, nested): dict_value = value # for typing stream.write_byte(DICT_TYPE) stream.write_var_int64(len(dict_value)) - for k, v in dict_value.iteritems(): + for k, v in dict_value.items(): Review comment: `iteritems()` isn't available anymore in Python 3. Instead, `items()` returns an iterator. I've replaced it with `iteritems(dict)` from future.utils, which returns an iterator on both versions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90373) Time Spent: 8.5h (was: 8h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90349 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 12/Apr/18 09:32 Start Date: 12/Apr/18 09:32 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r180998996 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -304,7 +307,7 @@ def encode_to_stream(self, value, stream, nested): dict_value = value # for typing stream.write_byte(DICT_TYPE) stream.write_var_int64(len(dict_value)) - for k, v in dict_value.iteritems(): + for k, v in dict_value.items(): Review comment: `iteritems()` was replaced by `items()` in Python 3 dict This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90349) Time Spent: 8h 20m (was: 8h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90328 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 12/Apr/18 08:42 Start Date: 12/Apr/18 08:42 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r181005942 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -216,6 +222,9 @@ def __eq__(self, other): and self._dict_without_impl() == other._dict_without_impl()) # pylint: enable=protected-access + def __hash__(self): +return hash(type(self)) Review comment: You're right. I've changed the hash to match the `__eq__` check. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90328) Time Spent: 8h 10m (was: 8h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90327
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 12/Apr/18 08:41
Start Date: 12/Apr/18 08:41
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on a change in pull request
#5053: [BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181005567
##
File path: sdks/python/apache_beam/coders/coders.py
##
@@ -309,11 +318,6 @@ class ToStringCoder(Coder):
"""A default string coder used if no sink coder is specified."""
def encode(self, value):
-try: # Python 2
- if isinstance(value, unicode):
-return value.encode('utf-8')
Review comment:
The `unicode = str` trick won't work here on Python 3:
```
>>> str('test'.encode('utf-8'))
"b'test'"
```
I've reverted this change to the previous solution
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90327)
Time Spent: 8h (was: 7h 50m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 8h
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90326
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 12/Apr/18 08:39
Start Date: 12/Apr/18 08:39
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on a change in pull request
#5053: [BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r181005086
##
File path: sdks/python/apache_beam/coders/coder_impl.py
##
@@ -279,21 +282,21 @@ def get_estimated_size_and_observables(self, value,
nested=False):
def encode_to_stream(self, value, stream, nested):
t = type(value)
-if t is NoneType:
+if value is None:
stream.write_byte(NONE_TYPE)
elif t is int:
stream.write_byte(INT_TYPE)
stream.write_var_int64(value)
elif t is float:
stream.write_byte(FLOAT_TYPE)
stream.write_bigendian_double(value)
-elif t is str:
- stream.write_byte(STR_TYPE)
+elif t is bytes:
+ stream.write_byte(BYTES_TYPE)
stream.write(value, nested)
-elif t is six.text_type:
- unicode_value = value # for typing
- stream.write_byte(UNICODE_TYPE)
- stream.write(unicode_value.encode('utf-8'), nested)
+elif t is unicode:
+ text_value = value # for typing
Review comment:
I tried this, but it throws a 'not a type' error for `unicode`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90326)
Time Spent: 7h 50m (was: 7h 40m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90323 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 12/Apr/18 08:17 Start Date: 12/Apr/18 08:17 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r180998996 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -304,7 +307,7 @@ def encode_to_stream(self, value, stream, nested): dict_value = value # for typing stream.write_byte(DICT_TYPE) stream.write_var_int64(len(dict_value)) - for k, v in dict_value.iteritems(): + for k, v in dict_value.items(): Review comment: `iteritems()` was replaced by `items()` in Python 3 dict This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90323) Time Spent: 7h 40m (was: 7.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90256 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 12/Apr/18 01:33 Start Date: 12/Apr/18 01:33 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r180943594 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -304,7 +307,7 @@ def encode_to_stream(self, value, stream, nested): dict_value = value # for typing stream.write_byte(DICT_TYPE) stream.write_var_int64(len(dict_value)) - for k, v in dict_value.iteritems(): + for k, v in dict_value.items(): Review comment: Any particular reason for the `iteritems()` -> `items()` change? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90256) Time Spent: 7.5h (was: 7h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90253
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 12/Apr/18 01:32
Start Date: 12/Apr/18 01:32
Worklog Time Spent: 10m
Work Description: charlesccychen commented on a change in pull request
#5053: [BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r180943361
##
File path: sdks/python/apache_beam/coders/coders.py
##
@@ -309,11 +318,6 @@ class ToStringCoder(Coder):
"""A default string coder used if no sink coder is specified."""
def encode(self, value):
-try: # Python 2
- if isinstance(value, unicode):
-return value.encode('utf-8')
Review comment:
Should we do the same `unicode = str` thing here? (In the new version, we
will raise an error if the value is a non-ascii unicode string; eg: `str(u'')`
raises an error, while this worked before.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90253)
Time Spent: 7h (was: 6h 50m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90255
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 12/Apr/18 01:32
Start Date: 12/Apr/18 01:32
Worklog Time Spent: 10m
Work Description: charlesccychen commented on a change in pull request
#5053: [BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r180943388
##
File path: sdks/python/apache_beam/coders/coder_impl.py
##
@@ -279,21 +282,21 @@ def get_estimated_size_and_observables(self, value,
nested=False):
def encode_to_stream(self, value, stream, nested):
t = type(value)
-if t is NoneType:
+if value is None:
stream.write_byte(NONE_TYPE)
elif t is int:
stream.write_byte(INT_TYPE)
stream.write_var_int64(value)
elif t is float:
stream.write_byte(FLOAT_TYPE)
stream.write_bigendian_double(value)
-elif t is str:
- stream.write_byte(STR_TYPE)
+elif t is bytes:
+ stream.write_byte(BYTES_TYPE)
stream.write(value, nested)
-elif t is six.text_type:
- unicode_value = value # for typing
- stream.write_byte(UNICODE_TYPE)
- stream.write(unicode_value.encode('utf-8'), nested)
+elif t is unicode:
+ text_value = value # for typing
Review comment:
Can you use the same `try: unicode except: unicode = str` in the
corresponding part of the .pxd file so that the type annotation directive for
`text_value` is respected?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90255)
Time Spent: 7h 20m (was: 7h 10m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=90254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90254 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 12/Apr/18 01:32 Start Date: 12/Apr/18 01:32 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#discussion_r180943338 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -216,6 +222,9 @@ def __eq__(self, other): and self._dict_without_impl() == other._dict_without_impl()) # pylint: enable=protected-access + def __hash__(self): +return hash(type(self)) Review comment: Any particular reason for this change? Previously, the hash would default to `object.__hash__`, which tries to give a different hash code for each instance. This change would give the same hash code for each class, which may not be desirable, since coders in general could be parameterized. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90254) Time Spent: 7h 10m (was: 7h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=89312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89312 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 10/Apr/18 09:40 Start Date: 10/Apr/18 09:40 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-380038653 - I tried to remove all six references so we don't have 2 dependencies for py2/3 compatibility. If the six method is preferred, I can revert the changes. - The previous cython version is not compatible with future.builtin types. Without this bump, any pure python cythonized file cannot use the builtins. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89312) Time Spent: 6h 40m (was: 6.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=89268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89268 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 10/Apr/18 05:39 Start Date: 10/Apr/18 05:39 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-379980133 R: @charlesccychen cc: @tvalentyn Some high level comments: - six could be handy in some cases, for example `six.string_types` could be used instead of ``` Try: unicode # pylint: disable=unicode-builtin except NameError: unicode = str ``` What do you think about that? - Is there a reason to change cython version? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89268) Time Spent: 6.5h (was: 6h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=89033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89033 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/Apr/18 17:25 Start Date: 09/Apr/18 17:25 Worklog Time Spent: 10m Work Description: angoenka commented on issue #5053: [BEAM-3981] Futurize coders subpackage URL: https://github.com/apache/beam/pull/5053#issuecomment-379829492 R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89033) Time Spent: 6h 20m (was: 6h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88954 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 09/Apr/18 14:41 Start Date: 09/Apr/18 14:41 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#issuecomment-379776098 I ran into some problems while applying the same strategy to some more subpackages (mostly the typehints package). Most of these problems are caused by the use of future.builtins str, bytes and int. I've therefore decided to change the used approach, which can be found in the new pull request #5053. A new pull request was made because a lot of the comments in this thread are now irrelevant. The new approach is unfortunately a bit less resistant to regression, but a better fit for the beam project and I suspect we will be able to move forward more quickly. Closing this PR in favor of #5053 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88954) Time Spent: 6h (was: 5h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88955
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 09/Apr/18 14:41
Start Date: 09/Apr/18 14:41
Worklog Time Spent: 10m
Work Description: RobbeSneyders closed pull request #4990: [BEAM-3981]
[WIP] Futurize and fix python 2 compatibility for coders subpackage
URL: https://github.com/apache/beam/pull/4990
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/coders/__init__.py
b/sdks/python/apache_beam/coders/__init__.py
index acca89f70f4..4a7e509ed75 100644
--- a/sdks/python/apache_beam/coders/__init__.py
+++ b/sdks/python/apache_beam/coders/__init__.py
@@ -14,6 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
from apache_beam.coders.coders import *
from apache_beam.coders.typecoders import registry
diff --git a/sdks/python/apache_beam/coders/coder_impl.pxd
b/sdks/python/apache_beam/coders/coder_impl.pxd
index 98dd508556a..dd82a00ab1a 100644
--- a/sdks/python/apache_beam/coders/coder_impl.pxd
+++ b/sdks/python/apache_beam/coders/coder_impl.pxd
@@ -74,7 +74,7 @@ cdef char STR_TYPE, UNICODE_TYPE, LIST_TYPE, TUPLE_TYPE,
DICT_TYPE, SET_TYPE
cdef class FastPrimitivesCoderImpl(StreamCoderImpl):
cdef CoderImpl fallback_coder_impl
- @cython.locals(unicode_value=unicode, dict_value=dict)
+ @cython.locals(dict_value = dict)
cpdef encode_to_stream(self, value, OutputStream stream, bint nested)
diff --git a/sdks/python/apache_beam/coders/coder_impl.py
b/sdks/python/apache_beam/coders/coder_impl.py
index cc7ed87c3ad..ee58c3fb043 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -25,12 +25,26 @@
coder_impl.pxd file for type hints.
For internal use only; no backwards-compatibility guarantees.
+
+isort:skip_file
"""
from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+native_int = int
-from types import NoneType
+# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
+from builtins import bytes
+from builtins import chr
+from builtins import int
+from builtins import object
+from builtins import range
+from builtins import str
-import six
+from past.builtins import str as old_str
+from past.builtins import long
+from past.builtins import unicode
from apache_beam.coders import observable
from apache_beam.utils import windowed_value
@@ -38,7 +52,6 @@
from apache_beam.utils.timestamp import MIN_TIMESTAMP
from apache_beam.utils.timestamp import Timestamp
-# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
try:
from .stream import InputStream as create_InputStream
from .stream import OutputStream as create_OutputStream
@@ -54,11 +67,6 @@
from .slow_stream import get_varint_size
# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
-try:
- long# Python 2
-except NameError:
- long = int # Python 3
-
class CoderImpl(object):
"""For internal use only; no backwards-compatibility guarantees."""
@@ -199,7 +207,7 @@ def __init__(self, coder, step_label):
self._step_label = step_label
def _check_safe(self, value):
-if isinstance(value, (str, six.text_type, long, int, float)):
+if isinstance(value, (str, bytes, int, float)):
pass
elif value is None:
pass
@@ -279,18 +287,21 @@ def get_estimated_size_and_observables(self, value,
nested=False):
def encode_to_stream(self, value, stream, nested):
t = type(value)
-if t is NoneType:
+if value is None:
stream.write_byte(NONE_TYPE)
-elif t is int:
+elif t is bool:
+ stream.write_byte(BOOL_TYPE)
+ stream.write_byte(value)
+elif t is int or t is native_int or t is long:
stream.write_byte(INT_TYPE)
stream.write_var_int64(value)
elif t is float:
stream.write_byte(FLOAT_TYPE)
stream.write_bigendian_double(value)
-elif t is str:
+elif t is bytes or t is old_str:
stream.write_byte(STR_TYPE)
stream.write(value, nested)
-elif t is six.text_type:
+elif t is str or t is unicode:
unicode_value = value # for typing
stream.write_byte(UNICODE_TYPE)
stream.write(unicode_value.encode('utf-8'), nested)
@@ -304,12 +315,9 @@ def encode_to_stream(self, value, stream, nested):
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88421 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 06/Apr/18 12:41 Start Date: 06/Apr/18 12:41 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#issuecomment-379232283 I've added some changes. Most notable: - Replace bytes with memoryview in stream cython files. This also works with subtypes of bytes like the future.builtins bytes type. - Revert isinstance checks to typechecks for performance with the use of past.builtins. I've also added the applied strategy to the [Python 3 proposal document](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit). It would be great to get some feedback on this so we can start moving forward with the other subpackages. @aaltay @charlesccychen This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88421) Time Spent: 5h 40m (was: 5.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88409 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 06/Apr/18 12:00 Start Date: 06/Apr/18 12:00 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#issuecomment-379232283 I've added some changes. Most notable: - Replace bytes with memoryview in stream cython files. This also works with subtypes of bytes like the future.builtins bytes type. - Revert isinstance checks to typechecks for performance with the use of past.builtins. I've also added the applied approach to the [Python 3 proposal document](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit). It would be great to get some feedback on this so we can start moving forward with the other subpackages. @aaltay @charlesccychen This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88409) Time Spent: 5.5h (was: 5h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88259 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 05/Apr/18 22:46 Start Date: 05/Apr/18 22:46 Worklog Time Spent: 10m Work Description: asfgit commented on issue #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#issuecomment-379098999 FAILURE --none-- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88259) Time Spent: 5h 20m (was: 5h 10m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88146 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 05/Apr/18 17:23 Start Date: 05/Apr/18 17:23 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#issuecomment-378065718 Thanks for the review @charlesccychen. Some general points based on your feedback and my answers: - The imports: `from __future__ import absolute_import` `from __future__ import division` `from __future__ import print_function` were added at the top of each updated module to prevent regression before full python 3 support is added. This way no new code can be added using for instance the old python 2 division. Another benefit is the consistency of division and print across modules. - `from builtins import ...` imports from future.builtins on python 2 and has no effect on python 3. future.builtins contains a bunch of backported python 3 builtins for compatibility. - The bytes type annotation was removed in the stream cython files because it was breaking due to a mismatch between the cython bytes type and the future bytes type. However, this is not meant to be merged this way, but I wanted to submit the pull request with working code to get some feedback on this. I have tried replacing bytes with a memory view as explained [here](http://cython.readthedocs.io/en/latest/src/tutorial/strings.html#accepting-strings-from-python-code), but this resulted in a packaging error. Any help on this is appreciated. - The Cython version was upgraded from 0.26.1 to 0.28.1 because of an incompatibility between cython and future types. I have not noticed any backward incompatibility. - The is type checks were replaced by isinstance checks because the future.builtins are all subclasses of the standard python classes. However, this works a lot slower. I could revert this change if I use six again for compatibility. The drawback is that `str` and `bytes` will work different across modules. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88146) Time Spent: 5h 10m (was: 5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=86815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86815 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/Apr/18 22:25 Start Date: 02/Apr/18 22:25 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#issuecomment-378065718 Thanks for the review @charlesccychen. Some general points based on your feedback and my answers: - The imports: `from __future__ import absolute_import` `from __future__ import division` `from __future__ import print_function` were added at the top of each updated module to prevent regression before full python 3 support is added. This way no new code can be added using for instance the old python 2 division. Another benefit is the consistency of division and print across modules. - `from builtins import ...` imports from future.builtins on python 2 and has no effect on python 3. future.builtins contains a bunch of backported python 3 builtins for compatibility. I actually think it's best to add `from builtins import *` to every module. Just like the \_\_future\_\_ imports, this helps avoid regression and adds consistency. This however still breaks some tests. I'll try to fix these and update the pull request later. - The bytes type annotation was removed in the stream cython files because it was breaking due to a mismatch between the cython bytes type and the future bytes type. However, this is not meant to be merged this way, but I wanted to submit the pull request with working code to get some feedback on this. I have tried replacing bytes with a memory view as explained [here](http://cython.readthedocs.io/en/latest/src/tutorial/strings.html#accepting-strings-from-python-code), but this resulted in a packaging error. Any help on this is appreciated. - The Cython version was upgraded from 0.26.1 to 0.28.1 because of an incompatibility between cython and future types. I have not noticed any backward incompatibility. - The is type checks were replaced by isinstance checks because the future.builtins are all subclasses of the standard python classes. However, this works a lot slower. I could revert this change if I use six again for compatibility. The drawback is that `str` and `bytes` will work different across modules. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86815) Time Spent: 5h (was: 4h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=86814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86814 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/Apr/18 22:25 Start Date: 02/Apr/18 22:25 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#issuecomment-378065718 Thanks for the review @charlesccychen. Some general points based on your feedback and my answers: - The imports: `from __future__ import absolute_import` `from __future__ import division` `from __future__ import print_function` were added at the top of each updated module to prevent regression before full python 3 support is added. This way no new code can be added using for instance the old python 2 division. Another benefit is the consistency of division and print across modules. - `from builtins import ...` imports from future.builtins on python 2 and has no effect on python 3. future.builtins contains a bunch of backported python 3 builtins for compatibility. I actually think it's best to add `from builtins import *` to every module. Just like the \_\_future\_\_ imports, this helps avoid regression and adds consistency. This however still breaks some tests. I'll try to fix these and update the pull request later. - The bytes type annotation was removed in the stream cython files because it was breaking due to a mismatch between the cython bytes type and the future bytes type. However, this is not meant to be merged this way, but I wanted to submit the pull request with working code to get some feedback on this. I have tried replacing bytes with a memory view as explained [here](http://cython.readthedocs.io/en/latest/src/tutorial/strings.html#accepting-strings-from-python-code), but this resulted in a packaging error. Any help on this is appreciated. - The Cython version was upgraded from 0.26.1 to 0.28.1 because of an incompatibility between cython and future types. I have not noticed any backward incompatibility. - The is type checks were replaced by isinstance checks because the future.builtins are all subclasses of the standard python classes. However, works a lot slower. I could revert this change if I use six again for compatibility. The drawback is that `str` and `bytes` will work different across modules. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86814) Time Spent: 4h 50m (was: 4h 40m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=86801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86801 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/Apr/18 22:01 Start Date: 02/Apr/18 22:01 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#discussion_r178663095 ## File path: sdks/python/apache_beam/coders/standard_coders_test.py ## @@ -74,7 +77,7 @@ class StandardCodersTest(unittest.TestCase): lambda x: IntervalWindow( start=Timestamp(micros=(x['end'] - x['span']) * 1000), end=Timestamp(micros=x['end'] * 1000)), - 'urn:beam:coders:stream:0.1': lambda x, parser: map(parser, x), + 'urn:beam:coders:stream:0.1': lambda x, parser: list(map(parser, x)), Review comment: There's one test that fails without the list call. I'll look into it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86801) Time Spent: 4h 40m (was: 4.5h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=86799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86799 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/Apr/18 22:00 Start Date: 02/Apr/18 22:00 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#discussion_r178662863 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -278,38 +277,38 @@ def get_estimated_size_and_observables(self, value, nested=False): return out.get_count(), [] def encode_to_stream(self, value, stream, nested): -t = type(value) -if t is NoneType: +if value is None: stream.write_byte(NONE_TYPE) Review comment: I have just checked the timing myself. `type(list()) is list` is exactly as fast as `isinstance(list(), list)`, but with an average of 5 if checks for every type call, the type check is 2.5x faster. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86799) Time Spent: 4.5h (was: 4h 20m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=86797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86797
]
ASF GitHub Bot logged work on BEAM-3981:
Author: ASF GitHub Bot
Created on: 02/Apr/18 21:54
Start Date: 02/Apr/18 21:54
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on a change in pull request
#4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders
subpackage
URL: https://github.com/apache/beam/pull/4990#discussion_r178661554
##
File path: sdks/python/apache_beam/coders/typecoders_test.py
##
@@ -33,14 +39,18 @@ def __init__(self, n):
def __eq__(self, other):
return self.number == other.number
+ def __hash__(self):
+return self.number
+
class CustomCoder(coders.Coder):
def encode(self, value):
-return str(value.number)
+x = value.number
+return int(x).to_bytes((x.bit_length() + 7) // 8, 'big', signed=True)
Review comment:
That would be simpler indeed. It clashed with the cython bytes, since
`str().encode('latin-1')` will return `newbytes` with the future package.
However, seems like we need to fix this cython bytes issue anyway, so I have
changed it back. This is the drawback of working on a per module basis I guess
:)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 86797)
Time Spent: 4h 20m (was: 4h 10m)
> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix
> python 2 compatibility. This prepares the subpackage for python 3 support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=86790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86790 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/Apr/18 21:38 Start Date: 02/Apr/18 21:38 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#discussion_r178643580 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -278,38 +277,38 @@ def get_estimated_size_and_observables(self, value, nested=False): return out.get_count(), [] def encode_to_stream(self, value, stream, nested): -t = type(value) -if t is NoneType: +if value is None: stream.write_byte(NONE_TYPE) Review comment: This was changed because `is` doesn't check for subclasses. The whole future.builtins package depends on subclasses to add compatibility. This can however be changed by depending on six again if necessary. Problem with this approach is that the `str` and `bytes` types will work different across modules. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86790) Time Spent: 4h 10m (was: 4h) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package
[ https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=86783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86783 ] ASF GitHub Bot logged work on BEAM-3981: Author: ASF GitHub Bot Created on: 02/Apr/18 21:23 Start Date: 02/Apr/18 21:23 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #4990: [BEAM-3981] [WIP] Futurize and fix python 2 compatibility for coders subpackage URL: https://github.com/apache/beam/pull/4990#discussion_r178654516 ## File path: sdks/python/setup.py ## @@ -69,7 +69,7 @@ def get_version(): ) -REQUIRED_CYTHON_VERSION = '0.26.1' +REQUIRED_CYTHON_VERSION = '0.28.1' Review comment: There seems to be an incompatibility between the future package and earlier versions of cython. On version 0.26.1, cython throws a compile error because it doesn't recognize the future builtins as types. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 86783) Time Spent: 4h (was: 3h 50m) > Futurize and fix python 2 compatibility for coders package > -- > > Key: BEAM-3981 > URL: https://issues.apache.org/jira/browse/BEAM-3981 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Ahmet Altay >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Run automatic conversion with futurize tool on coders subpackage and fix > python 2 compatibility. This prepares the subpackage for python 3 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
