dongjoon-hyun opened a new pull request, #58054: URL: https://github.com/apache/spark/pull/58054
### What changes were proposed in this pull request? This PR adds the ability to hold and resume a whole application from the Driver Web UI. - New `@DeveloperApi` methods `SparkContext.holdExecutors()` / `resumeExecutors()`. Holding stops requesting new executors (via `ExecutorAllocationManager.suspend()` when dynamic allocation is on, or `requestTotalExecutors(0)` otherwise) and gracefully decommissions all existing ones, so each executor finishes its running tasks and then exits. Resuming restores the executor requirements. - The Jobs page shows an application status line with a `(hold)` / `(resume)` control, following the existing kill button pattern, gated by a new config `spark.ui.holdEnabled` (default `true`). This requires both `spark.shuffle.service.enabled` (to keep the shuffle data of the decommissioned executors) and `spark.decommission.enabled`, and is rejected otherwise. ### Why are the changes needed? To allow a running application to temporarily give back its resources without being killed. Executors are drained gracefully with no loss of in-progress task work or shuffle output, and the application can be resumed later without recomputation. ### Does this PR introduce _any_ user-facing change? Yes, this adds new features: the `SparkContext` APIs, the Web UI control, and the `spark.ui.holdEnabled` config above. There is no behavior change for applications that do not use them. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
