maropu commented on a change in pull request #27246:
URL: https://github.com/apache/spark/pull/27246#discussion_r441204984
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -1238,6 +1239,18 @@ package object config {
s"The value must be in allowed range [1,048,576,
${MAX_BUFFER_SIZE_BYTES}].")
.createWithDefault(1024 * 1024)
+ private[spark] val UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE_RATIO =
+ ConfigBuilder("spark.unsafe.sorter.spill.reader.buffer.size.ratio")
+ .doc("The multiplication ratio is the parameter that controls the
initial read buffer " +
+ "size. The multiplication ratio value range is from 1 to 1024. This
parameter changes " +
+ "the initial read buffer size in 1KB increments. It will result in the
initial buffer " +
+ "size in the range from 1KB to 1MB. The read buffer size is
dynamically adjusted " +
+ "afterward based on data length read from the spilled file.")
+ .intConf
+ .checkValue(v => 1 <= v && v <= DEFAULT_BUFFER_SIZE_RATIO,
+ s"The value must be in allowed range [1,
${DEFAULT_BUFFER_SIZE_RATIO}].")
+ .createWithDefault(DEFAULT_BUFFER_SIZE_RATIO)
Review comment:
Could you move this config into `SQLConf`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
