HyukjinKwon commented on a change in pull request #31598:
URL: https://github.com/apache/spark/pull/31598#discussion_r581776970
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##########
@@ -914,7 +933,34 @@ object SparkSession extends Logging {
*/
def getOrCreate(): SparkSession = synchronized {
val sparkConf = new SparkConf()
- options.foreach { case (k, v) => sparkConf.set(k, v) }
+ val (nonEffectConfigurations, mayEffectConfigurations) =
+ options.partition((DRIVER_RELATED_LAUNCHER_CONFIG).contains)
+
+ if (nonEffectConfigurations.nonEmpty) {
+ logWarning(
+ s"""Since spark has been started, such configuration
+ | `${nonEffectConfigurations.mkString(", ")}` may not be affected
when setting
+ | programmatically through SparkConf in runtime, or the behavior
is depending
+ | on which cluster manager and deploy mode you choose, so it
would be suggested to
+ | set through configuration file or spark-submit command line
options.
+ |""".stripMargin)
+ }
+
+ val (scAlreadySetConfigurations, normalConfigs) =
+ mayEffectConfigurations.partition(EXECUTOR_LAUNCHER_CONFIG.contains)
+
+ if (SparkContext.getActive.isDefined) {
Review comment:
I think `SparkConf` is immutable and we cannot change configuration once
`SparkContext` is launched?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
