dongjoon-hyun opened a new pull request, #58137:
URL: https://github.com/apache/spark/pull/58137
### What changes were proposed in this pull request?
This PR hardens two pieces of executor-request bookkeeping in
`CoarseGrainedSchedulerBackend`
against huge requested totals (e.g. `sc.requestTotalExecutors(Int.MaxValue,
0, Map.empty)`):
1. `requestExecutors` now computes the new total with `Long` arithmetic and
saturates at
`Int.MaxValue` instead of overflowing to a negative value.
2. `updateExecRequestTimes` now skips an unbounded `Int.MaxValue` total, so
that no
effectively-unconsumable entry is pushed into `execRequestTimes`.
A `private[spark]` accessor `requestedTotalExecutors()` is added so the test
can observe the
requested totals.
Note: https://github.com/apache/spark/pull/58054 currently carries the same
two hunks; once
this is merged, that PR will rebase and drop them.
### Why are the changes needed?
Both defects are reproducible on `master` when a user requests a huge total
first:
1. If the recorded total is already huge (e.g. `Int.MaxValue`), a subsequent
`requestExecutors(n)` overflows `numExisting + numAdditionalExecutors` to
a negative
number, which is then sent to the cluster manager unvalidated — the
negative-value check
in `requestTotalExecutors` does not cover this path.
2. A huge total pushes an `(Int.MaxValue, timestamp)` entry into
`execRequestTimes` that is
never drained, so every executor registered afterwards reports that same
stale timestamp
as its "Request Time" in the event log (`JsonProtocol`), skewing history
data.
### Does this PR introduce _any_ user-facing change?
No. It only prevents pathological bookkeeping values in the corner cases
above.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
