siknezevic commented on a change in pull request #27246:
URL: https://github.com/apache/spark/pull/27246#discussion_r436385980
##########
File path:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
##########
@@ -47,55 +47,48 @@
private int numRecords;
private int numRecordsRemaining;
- private byte[] arr = new byte[1024 * 1024];
+ private byte[] arr = new byte[1024];
Review comment:
Does this look good? Perhaps you have some suggestion.
private[spark] val UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE_RATIO =
ConfigBuilder("spark.unsafe.sorter.spill.reader.buffer.size.ratio")
.doc("The multiplication ratio is the parameter that controls the
initial read buffer " +
"size. The multiplication ratio value range is from 1 to 1024. This
parameter increases " +
"the initial read buffer size in 1KB increments. It will result in
the initial buffer " +
"size in the range from 1KB to 1MB. The read buffer size is
dynamically adjusted " +
"afterward based on data length read from the spilled file.")
.intConf
.checkValue(v => 1 <= v && v <= DEFAULT_BUFFER_SIZE_RATIO,
s"The value must be in allowed range [1,
${DEFAULT_BUFFER_SIZE_RATIO}].")
.createWithDefault(DEFAULT_BUFFER_SIZE_RATIO)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
