HyukjinKwon commented on a change in pull request #29229:
URL: https://github.com/apache/spark/pull/29229#discussion_r460371504
##########
File path: python/pyspark/shuffle.py
##########
@@ -498,7 +498,7 @@ def load(f):
if current_chunk:
chunks.append(iter(current_chunk))
- return heapq.merge(chunks, key=key, reverse=reverse)
+ return heapq.merge(*chunks, key=key, reverse=reverse)
Review comment:
There was a bit of modification when this file was ported from Python 3
because `heapq` has to be able to compile with Python 2 as well. The diffs made
are:
```diff
< def merge(iterables, key=None, reverse=False):
---
> def merge(*iterables, key=None, reverse=False):
216c218,219
< h_append([next(it), order * direction, it])
---
> next = it.__next__
> h_append([next(), order * direction, next])
223c226
< value, order, it = s = h[0]
---
> value, order, next = s = h[0]
225c228
< s[0] = next(it) # raises StopIteration when
exhausted
---
> s[0] = next() # raises StopIteration when
exhausted
231c234
< value, order, it = h[0]
---
> value, order, next = h[0]
233,234c236
< for value in it:
< yield value
---
> yield from next.__self__
239,240c241,243
< value = next(it)
< h_append([key(value), order * direction, value, it])
---
> next = it.__next__
> value = next()
> h_append([key(value), order * direction, value, next])
247c250
< key_value, order, value, it = s = h[0]
---
> key_value, order, value, next = s = h[0]
249c252
< value = next(it)
---
> value = next()
256c259
< key_value, order, value, it = h[0]
---
> key_value, order, value, next = h[0]
258,259c261
< for value in it:
< yield value
---
> yield from next.__self__
```
These differences don't look affecting any behaviours.
I think it was ported from Python 3.5:
https://github.com/python/cpython/blob/3.5/Lib/heapq.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
