hai-tao-1 commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876509593
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -41,6 +41,15 @@ private[spark] object History {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
+ val UPDATE_BATCHSIZE = ConfigBuilder("spark.history.fs.update.batchSize")
+ .doc("Specifies the batch size for updating new eventlog files. " +
+ "This controls each scan process to be completed within a reasonable
time, and such " +
+ "prevent the initial scan from running too long and blocking new
eventlog files to " +
+ "be scanned in time in large environments.")
+ .version("3.4.0")
+ .intConf
Review Comment:
Added checkValue for a positive. A positive value should be enough.
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -41,6 +41,15 @@ private[spark] object History {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
+ val UPDATE_BATCHSIZE = ConfigBuilder("spark.history.fs.update.batchSize")
+ .doc("Specifies the batch size for updating new eventlog files. " +
+ "This controls each scan process to be completed within a reasonable
time, and such " +
+ "prevent the initial scan from running too long and blocking new
eventlog files to " +
+ "be scanned in time in large environments.")
+ .version("3.4.0")
+ .intConf
Review Comment:
Added checkValue for a positive value. A positive value should be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
