juliuszsompolski opened a new pull request, #53384: URL: https://github.com/apache/spark/pull/53384
### What changes were proposed in this pull request? Add testing helpers to SparkConnectServerTest to enable using connect Spark SQL APIs in tests using that helper. ### Why are the changes needed? In Spark 3.5, a testing trait SparkConnectServerTest was introduced that helped test Spark Connect Service with a SparkConnectClient in the same JVM proccess, which tested real Spark Connect code paths (SparkConnectClient communicating with the server over actual connection to the localhost server). Before that, using RemoteSparkSession, server was started in a separate process. It helped testability: can trigger stuff from the client, then have verification code checking stuff server side. Can also do some more internal server side setup to test specific things. debugging, as both client and server can be easily connected to by a debugger. At that time, it was impossible to test Spark Connect client SQL APIs (SparkSession, Dataset) this way, because they were in the same namespace as server, and hence couldn't be classloaded together. Since Spark 4.0, there is a new API layer that makes it possible for connect and classic implementation of the interfaces to coexist. With that, testing can be extended to use actual SparkSession and other APIs, instead of having to construct tests using more raw APIs. ### Does this PR introduce _any_ user-facing change? No. It's testing only. ### How was this patch tested? Added SparkConnectServerTestSuite showcasing the new APIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code opus 4.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
