MaxGekk commented on code in PR #59115:
URL: https://github.com/apache/spark/pull/59115#discussion_r4129713899
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorResizePlugin.scala:
##########
@@ -65,8 +65,8 @@ class ExecutorResizeDriverPlugin extends DriverPlugin with
Logging {
val interval = Utils.timeStringAsSeconds(
sc.conf.get(EXECUTOR_RESIZE_INTERVAL.key, "1m"))
Review Comment:
While we are here: `resizeInterval` is still read by raw key one line above,
so it bypasses `EXECUTOR_RESIZE_INTERVAL`'s `checkValue(_ >= 0)`, and the
hard-coded default `1m` contradicts the entry's default of `0` and its doc, "To
disable, set 0 (default)". Setting `0` as that doc says makes
`scheduleAtFixedRate(..., 0, 0, SECONDS)` throw out of `init`, so the driver
fails to start. `ExecutorPVCResizePlugin` does `val interval =
sc.conf.get(PVC_RESIZE_INTERVAL); if (interval <= 0) return` - the same shape
here would make all three knobs consistent, which is what this PR sets out to
do.
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorResizePlugin.scala:
##########
@@ -65,8 +65,8 @@ class ExecutorResizeDriverPlugin extends DriverPlugin with
Logging {
val interval = Utils.timeStringAsSeconds(
sc.conf.get(EXECUTOR_RESIZE_INTERVAL.key, "1m"))
- val threshold = sc.conf.getDouble(EXECUTOR_RESIZE_THRESHOLD.key, 0.9)
- val factor = sc.conf.getDouble(EXECUTOR_RESIZE_FACTOR.key, 0.1)
+ val threshold = sc.conf.get(EXECUTOR_RESIZE_THRESHOLD)
Review Comment:
With the entries enforced, values that 4.2.x and 4.3.x accepted
(`resizeFactor=1.5`, say) become startup failures, but neither the `.doc`
strings ("The threshold to resize.", "The factor to resize.") nor the table in
`docs/running-on-kubernetes.md` names the accepted ranges. Could the ranges
`(0, 1)` and `(0, 1]` go into both, so a user who hits
`INVALID_CONF_VALUE.REQUIREMENT` after an upgrade finds them documented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
