otterc commented on code in PR #36141:
URL: https://github.com/apache/spark/pull/36141#discussion_r849742513
##########
core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala:
##########
@@ -142,7 +142,7 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf)
extends Logging {
* VisibleForTesting
*/
protected def submitTask(task: Runnable): Unit = {
- if (BLOCK_PUSHER_POOL != null) {
+ if (BLOCK_PUSHER_POOL != null && !BLOCK_PUSHER_POOL.isShutdown) {
Review Comment:
I think in addition we should catch all kinds exception from the pusher
threads and not bubble it up. I am suggesting to change below to catch all
throwables as well.
```
private[shuffle] def tryPushUpToMax(): Unit = {
try {
pushUpToMax()
} catch {
case e: FileNotFoundException =>
logWarning("The shuffle files got deleted when this
shuffle-block-push-thread " +
"was reading from them which could happen when the job finishes
and the driver " +
"instructs the executor to cleanup the shuffle. In this case, push
of the blocks " +
"belonging to this shuffle will stop.", e)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
