Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19184#discussion_r137981565
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
---
@@ -72,12 +77,27 @@ public UnsafeSorterSpillReader(
bufferSizeBytes = DEFAULT_BUFFER_SIZE_BYTES;
}
+ try (InputStream bs = new NioBufferedFileInputStream(file, (int)
bufferSizeBytes);
+ DataInputStream dataIn = new
DataInputStream(serializerManager.wrapStream(blockId, bs))) {
+ this.numRecords = dataIn.readInt();
+ this.numRecordsRemaining = numRecords;
+ }
+
+ this.buffSize = bufferSizeBytes;
+ this.file = file;
+ this.blockId = blockId;
+ this.serializerManager = serializerManager;
+
+ logger.debug("bufSize: {}, file: {}, records: {}", buffSize, file,
this.numRecords);
--- End diff --
Is this log useful? If the number of spill readers is so many, I guess we
don't want to see so many log info?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
