dongjoon-hyun commented on PR #58054: URL: https://github.com/apache/spark/pull/58054#issuecomment-5352564335
Rebased onto the current master as 193df19, squashing the review history into a single commit (the incremental commits referenced in the threads above remain reachable on GitHub). Two fixes that were spun off from this PR landed on master separately and are now integrated instead of duplicated: - [SPARK-58886](https://github.com/apache/spark/commit/77a4f757790) (`requestExecutors` overflow): this PR keeps only the `publishTotals()` wiring on top of it. The duplicated saturating add, the `updateExecRequestTimes` sentinel guard (unnecessary now that the resume sentinel is published through `publishTotalsWithoutRecording`, which bypasses the bookkeeping entirely), and this PR's own `requestedTotalExecutors()` accessor (superseded by `getRequestedTotalExecutors()`) are dropped. - [SPARK-58887](https://github.com/apache/spark/commit/4998b7769ba) (barrier job cancellable during slot-check retries): this PR's `jobsDeferredWhileHeld` is gone. The pipelined-while-held deferral now registers in `deferredBarrierJobs` and relies on its cancellation-marker mechanism, which is more robust than what this PR carried (the drop decision is made on the event loop, and job-group/tag cancellation covers deferred jobs too). The barrier retry-budget freeze while held is unchanged. All suites touched by this PR pass on the rebased branch (DAGSchedulerSuite 224, CoarseGrainedSchedulerBackendSuite 29, ExecutorAllocationManagerSuite 45, the SparkContextSuite/UISeleniumSuite/K8s hold tests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
