ericm-db opened a new pull request, #58248: URL: https://github.com/apache/spark/pull/58248
### What changes were proposed in this pull request? This is layer 7 of the nine-PR local Connect pool stack: #57684 -> #57685 -> #57907 -> #57686 -> RECOVERY_PR -> #57687 -> PURGE_PR -> #57102 -> #57688 Until lower layers merge, GitHub shows their cumulative diff. The review unit introduced here is commit `196221a8d5a`. This layer adds the forceful escape hatch for returning the pool to a clean slate: - scans every state kind directly, including malformed and duplicate claimed records; - verifies attendants and servers before signalling reused PIDs; - SIGKILLs recoverable attendants, launch groups, and server processes; - removes all state files and member directories; and - exposes `python -m pyspark.sql.connect.local_server_pool --purge`. SparkSession integration and JIT warmup remain in later PRs. ### Why are the changes needed? Normal retirement deliberately preserves retryable state. Operators also need a bounded, destructive recovery path when the pool itself is corrupt or wedged. Keeping purge separate makes its stronger signalling and deletion semantics explicit and independently reviewable. ### Does this PR introduce _any_ user-facing change? Yes. It adds `python -m pyspark.sql.connect.local_server_pool --purge`, which force-stops every local pool process it can verify and empties the pool directory. SparkSession still does not select the pool until #57102. ### How was this patch tested? Added one focused corruption-and-process-lifecycle test at this layer, bringing the suite to 51 tests. It covers ready, pending, half-started, duplicate-claimed, retiring, and malformed members. ```bash python/run-tests --testnames pyspark.sql.tests.connect.test_connect_local_server_pool ``` The rebuilt commit passed Python AST parsing, `git diff --check`, and changed-file ASCII and 100-column checks. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Fable 5) and OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
