ericm-db opened a new pull request, #57688: URL: https://github.com/apache/spark/pull/57688
### What changes were proposed in this pull request? This is layer 6 of the local Connect pool stack: #57684 -> #57685 -> #57686 -> #57687 -> #57102 -> this PR. Until the lower layers merge, GitHub shows their cumulative diff; the new review unit here is commit `275235cd137`. This layer adds optional JIT warmup for newly booted pool members: - attendants launch a bounded child process against each published server; - fixed synthetic queries warm JVM JIT and Spark codegen paths without creating persistent catalog state; - warmup stops immediately if the member is claimed and is killed after its timeout; - `SPARK_LOCAL_CONNECT_POOL_WARMUP=0` disables it; - the internal `--warm` entry point runs the query set; and - the pool documentation explains the latency and memory trade-off. ### Why are the changes needed? A booted server avoids JVM and SparkContext startup, but its first real query can still pay JIT and code-generation costs. Keeping warmup separate from acquisition makes this optimization and its subprocess failure boundary optional and independently reviewable. ### Does this PR introduce _any_ user-facing change? Yes, only for the experimental pool opt-in introduced by #57102. Pool members warm themselves by default before use. Set `SPARK_LOCAL_CONNECT_POOL_WARMUP=0` to retain the booted-server behavior without synthetic queries. Behavior is unchanged when the pool opt-in is disabled. ### How was this patch tested? Added focused tests for environment parsing, termination of a warmup child after a member is claimed, and execution plus cleanup of the fixed query set. The full pool suite also covers sequential and concurrent clients against real local Connect servers. ```bash python -m unittest -v pyspark.sql.tests.connect.test_connect_local_server_pool ``` All 22 tests passed. Ruff check, Ruff format check, and `git diff --check` also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Fable 5) and OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
