Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20184#discussion_r208993136
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
---
@@ -116,13 +138,18 @@ public void loadNext() throws IOException {
if (taskContext != null) {
taskContext.killTaskIfInterrupted();
}
- recordLength = din.readInt();
- keyPrefix = din.readLong();
- if (recordLength > arr.length) {
- arr = new byte[recordLength];
+ // check if the reader is closed to prevent reopen the in and din.
+ if (!hasNext()) {
+ throw new IndexOutOfBoundsException("Can not load next item when
UnsafeSorterSpillReader is closed.");
+ }
+ recordLength = getDin().readInt();
+ keyPrefix = getDin().readLong();
+ int arrLength = Math.max(1024 * 1024, recordLength);
+ if (arrLength > arr.length) {
+ arr = new byte[arrLength];
baseObject = arr;
}
- ByteStreams.readFully(in, arr, 0, recordLength);
+ ByteStreams.readFully(getIn(), arr, 0, recordLength);
--- End diff --
Is it fine if `recordLength` is greater than `1024 * 1024`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
