Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5277#discussion_r27442341
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1394,28 +1396,37 @@ class SparkContext(config: SparkConf) extends
Logging with ExecutorAllocationCli
/** Shut down the SparkContext. */
def stop() {
- SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
- if (!stopped) {
- stopped = true
- postApplicationEnd()
- ui.foreach(_.stop())
- env.metricsSystem.report()
- metadataCleaner.cancel()
- cleaner.foreach(_.stop())
- dagScheduler.stop()
- dagScheduler = null
- listenerBus.stop()
- eventLogger.foreach(_.stop())
- env.actorSystem.stop(heartbeatReceiver)
- progressBar.foreach(_.stop())
- taskScheduler = null
- // TODO: Cache.stop()?
- env.stop()
- SparkEnv.set(null)
- logInfo("Successfully stopped SparkContext")
- SparkContext.clearActiveContext()
- } else {
- logInfo("SparkContext already stopped")
+ // The shutdown logic can create a deadlock scenario when an external
thread triggers an
+ // error in the handling of which it shuts down the SparkContext.
Meanwhile, this thread
+ // will wait for the external thread to join during the
"dagScheduler.stop()" while that
+ // thread waits on the SPARK_CONTEXT_CONSTRUCTOR_LOCK. We avoid this
by surrounding this
+ // logic in a try catch.
+ while (!stopped) {
+ if (SparkContext.shutdownLock.tryLock(10, TimeUnit.SECONDS)) {
--- End diff --
I think we need a `finally` to ensure that the lock acquired here is
eventually released.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
