Github user squito commented on the issue:
https://github.com/apache/spark/pull/21185
> Yea I think this is the root cause. I'm making a PR to ban SQLConf.get at
executor side, shall we do the same thing for SparkSession? And fixes all the
places that mistakenly access SparkSession in executor.
ah that makes sense. All the failures I observed w/ the executor accessing
the active session were actually via `SQLConf.get`. I don't know of any others.
I think it makes sense to actively ban `SparkSession.getActiveSession` the
same way. I can include that in this PR, or make another one (or I guess you
can put it into your PR), whatever you prefer. I don't think there will be any
existing uses that need to be fixed, as I'm pretty sure if you did access it,
you'd get the exception we've seen in tests related to the listener bus:
```
08:36:34.694 Executor task launch worker for task 436 ERROR Executor:
Exception in task 0.0 in stage 402.0 (TID 436)
java.lang.IllegalStateException: LiveListenerBus is stopped.
at
org.apache.spark.scheduler.LiveListenerBus.addToQueue(LiveListenerBus.scala:97)
at
org.apache.spark.scheduler.LiveListenerBus.addToStatusQueue(LiveListenerBus.scala:80)
at
org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:93)
at
org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:117)
...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
