[
https://issues.apache.org/jira/browse/IMPALA-11974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698544#comment-17698544
]
ASF subversion and git services commented on IMPALA-11974:
----------------------------------------------------------
Commit eb66d00f9f43ddaa6a9547574a150e0b1436f4d4 in impala's branch
refs/heads/dependabot/pip/infra/python/deps/wheel-0.38.1 from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=eb66d00f9 ]
IMPALA-11974: Fix lazy list operators for Python 3 compatibility
Python 3 changes list operators such as range, map, and filter
to be lazy. Some code that expects the list operators to happen
immediately will fail. e.g.
Python 2:
range(0,5) == [0,1,2,3,4]
True
Python 3:
range(0,5) == [0,1,2,3,4]
False
The fix is to wrap locations with list(). i.e.
Python 3:
list(range(0,5)) == [0,1,2,3,4]
True
Since the base operators are now lazy, Python 3 also removes the
old lazy versions (e.g. xrange, ifilter, izip, etc). This uses
future's builtins package to convert the code to the Python 3
behavior (i.e. xrange -> future's builtins.range).
Most of the changes were done via these futurize fixes:
- libfuturize.fixes.fix_xrange_with_import
- lib2to3.fixes.fix_map
- lib2to3.fixes.fix_filter
This eliminates the pylint warnings:
- xrange-builtin
- range-builtin-not-iterating
- map-builtin-not-iterating
- zip-builtin-not-iterating
- filter-builtin-not-iterating
- reduce-builtin
- deprecated-itertools-function
Testing:
- Ran core job
Change-Id: Ic7c082711f8eff451a1b5c085e97461c327edb5f
Reviewed-on: http://gerrit.cloudera.org:8080/19589
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
> Fix range/xrange/map/filter/ifilter/zip/izip issues regarding iterators being
> lazy
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-11974
> URL: https://issues.apache.org/jira/browse/IMPALA-11974
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
> Affects Versions: Impala 4.3.0
> Reporter: Joe McDonnell
> Priority: Major
>
> Python 3 made map, range, zip, and filter lazy. That means that existing
> Python code that relies on this turning into a list immediately need
> adjustments:
> {noformat}
> # Python 2:
> range(5) == [0,1,2,3,4]
> True
> # Python 3:
> range(5) == [0,1,2,3,4]
> False
> # Python 3 fix:
> list(range(5)) == [0,1,2,3,4]
> True
> # Same deal with map, zip, filter, etc.{noformat}
> This example is basic, but this applies across the board.
> Since the ordinary operators like range, filter, etc are now lazy, the old
> lazy ones have been removed (xrange, ifilter, izip). Those need to be
> replaced, and the future package provides implementations of range, zip, map,
> filter, etc.
> See:
> [https://python-future.org/compatible_idioms.html#xrange]
> [https://python-future.org/compatible_idioms.html#range]
> [https://python-future.org/compatible_idioms.html#map]
> [https://python-future.org/compatible_idioms.html#imap]
> etc
> This corresponds to these pylint checks:
> xrange-builtin
> range-builtin-not-iterating
> map-builtin-not-iterating
> zip-builtin-not-iterating
> filter-builtin-not-iterating
> reduce-builtin
> deprecated-itertools-function
> Futurize has some tools to help:
> libfuturize.fixes.fix_xrange_with_import
> lib2to3.fixes.fix_map
> lib2to3.fixes.fix_filter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
