HyukjinKwon opened a new pull request, #42464: URL: https://github.com/apache/spark/pull/42464
### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/41013 that adds a check for `SPARK_CONNECT_MODE_ENABLED` when we create a Spark session via `pyspark.sql.SparkSession.getOrCreate()` so the next session can be consistently returned. ### Why are the changes needed? Currently, it returns non Spark connect session if you call `SparkSession.builder.getOrCreate` after creating a Spark Connect session if `SPARK_REMOTE` is not globally set (that is currently automatically set if you launch `./bin/pyspark --remote local`). So this can only be reproducible when you Spark Connect as a library. See how it's tested below. ### Does this PR introduce _any_ user-facing change? The change has not been released yet so there's no behaviour changes to the end users. ### How was this patch tested? Manually tested as below: ```bash cd python python ``` ```python from pyspark.sql import SparkSession SparkSession.builder.remote("local").getOrCreate() SparkSession.builder.getOrCreate() ``` ``` <pyspark.sql.connect.session.SparkSession object at 0x7fa9807fd790> <pyspark.sql.session.SparkSession object at 0x7fd97890e730> ``` ``` <pyspark.sql.connect.session.SparkSession object at 0x7fa9807fd790> <pyspark.sql.connect.session.SparkSession object at 0x7fa9807fd790> ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
