siknezevic commented on a change in pull request #27246:
URL: https://github.com/apache/spark/pull/27246#discussion_r441273689
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -1238,6 +1239,18 @@ package object config {
s"The value must be in allowed range [1,048,576,
${MAX_BUFFER_SIZE_BYTES}].")
.createWithDefault(1024 * 1024)
+ private[spark] val UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE_RATIO =
+ ConfigBuilder("spark.unsafe.sorter.spill.reader.buffer.size.ratio")
+ .doc("The multiplication ratio is the parameter that controls the
initial read buffer " +
+ "size. The multiplication ratio value range is from 1 to 1024. This
parameter changes " +
+ "the initial read buffer size in 1KB increments. It will result in the
initial buffer " +
+ "size in the range from 1KB to 1MB. The read buffer size is
dynamically adjusted " +
+ "afterward based on data length read from the spilled file.")
+ .intConf
+ .checkValue(v => 1 <= v && v <= DEFAULT_BUFFER_SIZE_RATIO,
+ s"The value must be in allowed range [1,
${DEFAULT_BUFFER_SIZE_RATIO}].")
+ .createWithDefault(DEFAULT_BUFFER_SIZE_RATIO)
Review comment:
Just need clarification. SQLConf is member of catalist package/project.
I looked all classes involved in this change and I do not see usage of SQLConf.
I would expect that new parameter UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE_RATIO
is together with existing
package$.MODULE$.UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE parameter in
package.scala. Also, UnsafeSorterSpillReader is in core package/project. I just
want to be sure that we want UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE_RATIO to be
in SQLConf. Could you please confirm?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
