peter-toth commented on code in PR #59113:
URL: https://github.com/apache/spark/pull/59113#discussion_r4130623815
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorResizePlugin.scala:
##########
@@ -104,6 +104,7 @@ class ExecutorResizeDriverPlugin extends DriverPlugin with
Logging {
.inNamespace(namespace)
.withLabel(SPARK_APP_ID_LABEL, appId)
.withLabel(SPARK_ROLE_LABEL, SPARK_POD_EXECUTOR_ROLE)
+ .withoutLabel(SPARK_EXECUTOR_INACTIVE_LABEL, "true")
Review Comment:
**Finding 1.** `ExecutorPVCResizePlugin.checkAndResizePVCs` has the same pod
listing without this line (`ExecutorPVCResizePlugin.scala:130-135`), and there
it costs more than a failed API call. `latestReports` is only ever written
(`:113`), so a dead executor's last report stays forever. With
`deleteOnTermination=false`, its pod is still listed, so `podByExecId` finds
it. Each interval then runs `tryResize` again on the same stale ratio. When the
previous expansion has finished, `current == capacity`, `newSize = current *
1.1` differs from `requestedSizes`, and the PVC is patched again.
`resizeMaxStorage` defaults to `Long.MaxValue`, so nothing stops it.
I checked it in `ExecutorPVCResizePluginSuite` with an inactive-labelled
pod, one `0.95` report and three intervals, where the mocked PVC shows each
previous resize as done:
```
patches=3 sizes=1100000000,1210000000,1331000000
```
The same one-liner there closes it:
```scala
.withLabel(SPARK_ROLE_LABEL, SPARK_POD_EXECUTOR_ROLE)
.withoutLabel(SPARK_EXECUTOR_INACTIVE_LABEL, "true")
.list()
```
With it applied, plus the matching `withoutLabel` stub in the suite's
`before`, both plugin suites stay green (31 tests). Dropping the report when
the executor goes away would also cover the short window before the pod is
labelled inactive. Either fix is fine here or as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
