mridulm commented on code in PR #37533:
URL: https://github.com/apache/spark/pull/37533#discussion_r956329357
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler(
}
override def onShuffleMergeFailure(e: Throwable): Unit = {
+ if (e.isInstanceOf[IOException]) {
+ logInfo(s"Failed to connect external shuffle service "
+
+ s"${shuffleServiceLoc.hostPort}")
+
blockManagerMaster.removeShufflePushMergerLocation(shuffleServiceLoc.host)
+ }
}
})
}
}
}, 0, TimeUnit.SECONDS)
} else {
- stage.shuffleDep.getMergerLocs.zipWithIndex.foreach {
- case (shuffleServiceLoc, index) =>
- // Sends async request to shuffle service to finalize shuffle
merge on that host
- // TODO: SPARK-35536: Cancel finalizeShuffleMerge if the stage is
cancelled
- // TODO: during shuffleMergeFinalizeWaitSec
- shuffleClient.finalizeShuffleMerge(shuffleServiceLoc.host,
- shuffleServiceLoc.port, shuffleId, shuffleMergeId,
- new MergeFinalizerListener {
- override def onShuffleMergeSuccess(statuses: MergeStatuses):
Unit = {
- assert(shuffleId == statuses.shuffleId)
- eventProcessLoop.post(RegisterMergeStatuses(stage,
MergeStatus.
- convertMergeStatusesToMergeStatusArr(statuses,
shuffleServiceLoc)))
- results(index).set(true)
- }
+ shuffleMergeFinalizeScheduler.schedule(new Runnable {
Review Comment:
Agree, and that is why driver makes a best case effort to send the message.
If we are unable to do so in reasonable time due to network issues (n/w
partition, etc), failures, and so on - there could be files which are not yet
closed.
On other hand, keeping this pending state around in driver for extended
periods of time will cause its own issues.
Note that in general, there will always be some dropped messages (for ex
when failure handler is getting invoked) - so some variant of what you
described can occur even if we make this specific case robust - what we have to
evaluate is how much of an impact it has on shuffle service itself in context
of whether it will negatively impact NM stability.
In our environment, this has been in prod for a while now - and we have not
seen this specific issue. Having said that, for a different application and
network characteristics, it could possibly be an issue : more information will
help understand it.
On option could be to evaluate if we move failed sends and cancelled tasks
to a different threadpool and retry a few times to the send the message and
mitigate the issue (thoughts on adding a cancel instead of merge for that
shuffleid-shuffleattemptid ?). This will have no performance/functional impact
on driver, but can mitigate ESS load in terms of open files.
Thoughts @wankunde, @otterc ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
