[
https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro resolved SPARK-31532.
--------------------------------------
Fix Version/s: 2.4.6
Assignee: Kent Yao
Resolution: Fixed
Resolved by https://github.com/apache/spark/pull/28316
> SparkSessionBuilder shoud not propagate static sql configurations to the
> existing active/default SparkSession
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-31532
> URL: https://issues.apache.org/jira/browse/SPARK-31532
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.5, 3.0.0, 3.1.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Major
> Fix For: 2.4.6
>
>
> Clearly, this is a bug.
> {code:java}
> scala> spark.sql("set spark.sql.warehouse.dir").show
> +--------------------+--------------------+
> | key| value|
> +--------------------+--------------------+
> |spark.sql.warehou...|file:/Users/kenty...|
> +--------------------+--------------------+
> scala> spark.sql("set spark.sql.warehouse.dir=2");
> org.apache.spark.sql.AnalysisException: Cannot modify the value of a static
> config: spark.sql.warehouse.dir;
> at
> org.apache.spark.sql.RuntimeConfig.requireNonStaticConf(RuntimeConfig.scala:154)
> at org.apache.spark.sql.RuntimeConfig.set(RuntimeConfig.scala:42)
> at
> org.apache.spark.sql.execution.command.SetCommand.$anonfun$x$7$6(SetCommand.scala:100)
> at
> org.apache.spark.sql.execution.command.SetCommand.run(SetCommand.scala:156)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3644)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3642)
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)
> at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
> at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
> ... 47 elided
> scala> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.SparkSession
> scala> SparkSession.builder.config("spark.sql.warehouse.dir", "xyz").get
> getClass getOrCreate
> scala> SparkSession.builder.config("spark.sql.warehouse.dir",
> "xyz").getOrCreate
> 20/04/23 23:49:13 WARN SparkSession$Builder: Using an existing SparkSession;
> some configuration may not take effect.
> res7: org.apache.spark.sql.SparkSession =
> org.apache.spark.sql.SparkSession@6403d574
> scala> spark.sql("set spark.sql.warehouse.dir").show
> +--------------------+-----+
> | key|value|
> +--------------------+-----+
> |spark.sql.warehou...| xyz|
> +--------------------+-----+
> scala>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
