TongWei1105 commented on PR #59756: URL: https://github.com/apache/spark/pull/59756#issuecomment-6466413315
> Reviewed [7e2997b](https://github.com/apache/spark/commit/7e2997bd3cb9d8408641f94f0d08afadfc9be11d) with five independent passes covering allocation/PVC accounting, concurrency and shutdown, tests, documentation/style, and design/performance. > > One reproducible shutdown race remains, described inline, so I am withholding approval. I also left a nonblocking documentation clarification about allocator scope. > > Validation: verified the exact-head CI logs show all 454 Kubernetes tests passing, including the new allocator/backend scenarios; documentation generation and lint/compatibility checks are green. Independently reproduced the queue-drain termination race with standalone JDK 17 and 21 probes, including a deterministic interleaving. I did not rerun the full Spark suite locally or run a live-cluster benchmark. @sunchao Thanks for the detailed review and the reproducer! Fixed in 6f509b852fb: after draining and cancelling queued tasks, we call shutdown() again to wake idle workers without interrupting active API requests. I added a deterministic regression test covering a request finishing during queue draining; it fails before the fix and passes afterward. Also addressed the allocator-scope clarification in 8980d0351da. Both the configuration descriptions and the Kubernetes documentation now state that these settings apply only to the direct allocator. All 455 Kubernetes tests passed locally. CI has been retriggered after the workflow detection failure and is pending. Could you please take another look when you have time? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
