HyukjinKwon commented on a change in pull request #34757:
URL: https://github.com/apache/spark/pull/34757#discussion_r769248275
##########
File path: python/pyspark/sql/session.py
##########
@@ -304,8 +306,15 @@ def __init__(
and not
self._jvm.SparkSession.getDefaultSession().get().sparkContext().isStopped()
):
jsparkSession =
self._jvm.SparkSession.getDefaultSession().get()
+ getattr(getattr(self._jvm, "SparkSession$"),
"MODULE$").applyModifiableSettings(
Review comment:
If we can't make it in Spark 3.3, I think maybe it's just safer to
revert https://github.com/apache/spark/pull/34757
https://github.com/apache/spark/pull/34732 and
https://github.com/apache/spark/pull/34559 for now because each patch here will
introduce either:
1. Unexpected configuration propagation of static SQL configuration, or
2. Too much warnings
Separately, I still feel
https://github.com/apache/spark/commit/8424f552293677717da7411ed43e68e73aa7f0d6
is inefficient. We don't know which configurations don't take affect, or why it
keeps complaining (see the example above) for which configuration. We should
probably at least print out the keys or lower the level of log.
cc @AngersZhuuuu @yaooqinn @maropu @dongjoon-hyun FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
