Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/6648#discussion_r33624446
--- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala
---
@@ -62,4 +88,31 @@ private[spark] trait ShuffleManager {
/** Shut down this ShuffleManager. */
def stop(): Unit
+
+ private[this] val shuffleToAttempts = new ConcurrentHashMap[Int,
CopyOnWriteArraySet[Int]]()
+
+ /**
+ * Register a stage attempt for the given shuffle, so we can clean up
all attempts when
+ * the shuffle is unregistered
+ */
+ protected def addShuffleAttempt(shuffleId: Int, stageAttemptId: Int):
Unit = {
+ shuffleToAttempts.putIfAbsent(shuffleId, new
CopyOnWriteArraySet[Int]())
+ shuffleToAttempts.get(shuffleId).add(stageAttemptId)
+ }
+
+ /**
+ * Get all stage attempts a shuffle, so they can all be cleaned up.
+ *
+ * Calling this also cleans up internal state which tracks attempts for
each shuffle, so calling
+ * this again for the same shuffleId will always yield an empty Iterable.
+ */
+ @VisibleForTesting
+ private[shuffle] def stageAttemptsForShuffle(shuffleId: Int):
Iterable[Int] = {
--- End diff --
I'd prefer that the name reflect the mutative/clean-up nature of this
method -- it's certainly not obvious at the call sites in unregisterShuffle
that this what is happening.
Can there ever be a legitimate use for a non-clean-up
stageAttemptsForShuffle?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
