Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/1931#discussion_r16464399
--- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala ---
@@ -66,10 +66,15 @@ private[spark] class ContextCleaner(sc: SparkContext)
extends Logging {
/**
* Whether the cleaning thread will block on cleanup tasks.
- * This is set to true only for tests.
+ *
+ * Due to SPARK-3015, this is set to true by default. This is intended
to be only a temporary
+ * workaround for the issue, which is ultimately caused by the way the
BlockManager actors
+ * issue inter-dependent blocking Akka messages to each other at high
frequencies. This happens,
+ * for instance, when the driver performs a GC and cleans up all
broadcast blocks that are no
+ * longer in scope.
*/
private val blockOnCleanupTasks = sc.conf.getBoolean(
- "spark.cleaner.referenceTracking.blocking", false)
--- End diff --
The changes will not solve the problem here. see.
[BlockManagerMasterActor.scala#L165](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala#L165)
```scala
private def removeShuffle(shuffleId: Int): Future[Seq[Boolean]] = {
// Nothing to do in the BlockManagerMasterActor data structures
import context.dispatcher
val removeMsg = RemoveShuffle(shuffleId)
Future.sequence(
blockManagerInfo.values.map { bm =>
// Here has set the akkaTimeout
bm.slaveActor.ask(removeMsg)(akkaTimeout).mapTo[Boolean]
}.toSeq
)
}
````
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
