ajithme commented on a change in pull request #27267: [SPARK-30556][SQL] Copy
sparkContext.localproperties to child thread inSubqueryExec.executionContext
URL: https://github.com/apache/spark/pull/27267#discussion_r369343656
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
##########
@@ -164,4 +166,19 @@ object SQLExecution {
}
}
}
+
+ /**
+ * Wrap passed function to ensure necessary thread-local variables like
+ * SparkContext local properties are forwarded to execution thread
+ */
+ def withThreadLocalCaptured[T](sparkSession: SparkSession, exec:
ExecutionContext)(
+ body: => T): Future[T] = {
+ val activeSession = sparkSession
+ val localProps =
Utils.cloneProperties(sparkSession.sparkContext.getLocalProperties)
+ Future {
+ SparkSession.setActiveSession(activeSession)
Review comment:
Do you mean i should clear the thread locals.? as these are worker threads,
they do not have any old state as such. For example, when the thread is created
first time, it will inherit from parent thread, but when resued, restoring
would cause the previous properties to be set which is stale. IMO, setting this
on every execution should suffice. Please correct me if wrong
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
