Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21299#discussion_r187787208
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala ---
@@ -68,16 +68,27 @@ object SQLExecution {
// sparkContext.getCallSite() would first try to pick up any call
site that was previously
// set, then fall back to Utils.getCallSite(); call
Utils.getCallSite() directly on
// streaming queries would give us call site like "run at
<unknown>:0"
- val callSite = sparkSession.sparkContext.getCallSite()
+ val callSite = sc.getCallSite()
-
sparkSession.sparkContext.listenerBus.post(SparkListenerSQLExecutionStart(
+ // Set all the specified SQL configs to local properties, so that
they can be available at
+ // the executor side.
+ val allConfigs = sparkSession.sessionState.conf.getAllConfs
+ allConfigs.foreach {
+ // Excludes external configs defined by users.
+ case (key, value) if key.startsWith("spark") =>
sc.setLocalProperty(key, value)
--- End diff --
Only propagate config values that have been set other than default value?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
