cdkrot opened a new pull request, #41987:
URL: https://github.com/apache/spark/pull/41987
### What changes were proposed in this pull request?
ML and other uses rely on _active_spark_session to find spark session.
Sessions created using getOrCreate method set this variable, but sessions
created with create don't.
Update create method to set _active_spark_session.
### Why are the changes needed?
This breaks spark connect customers, such as pyspark.ml and pandas from
finding created session if it was created with create.
### Does this PR introduce _any_ user-facing change?
Sessions created by create are set as current session. This is slightly
different behavior then before. We can hide it under configuration flag, e.g.
`create(set_active_session=False)` if undesired. In this case clients who use
`create` and want to use pyspark.ml/pandas will need to update to set it to
True.
### How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
