Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19184
@jerryshao Actually the second half of your comment is not valid in this
case.
The PR is not targeting the merge sort in this case, but relevant when
iterating over all tuples.
`UnsafeExternalSorter` has two methods to iterate over the tuples.
You are referring to `getSortedIterator` - which uses a PriorityQueue and
requires all files to be opened at the same time (so that it can return a
sorted iterator).
The primary usecase of this PR is for `getIterator` - where we are simply
iterating over all tuples : and used in `ExternalAppendOnlyUnsafeRowArray` for
example : there is no need to sort here.
This is used by various `WindowFunctionFrame` implementations for example.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
