vanzin commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter URL: https://github.com/apache/spark/pull/25342#discussion_r318221805
########## File path: core/src/main/scala/org/apache/spark/shuffle/ShufflePartitionPairsWriter.scala ########## @@ -60,17 +60,43 @@ private[spark] class ShufflePartitionPairsWriter( } override def close(): Unit = { + var thrownException: Option[IOException] = None if (objOut != null) { - // Closing objOut should propagate close to all inner layers - objOut.close() + thrownException = tryCloseOrAddSuppressed(objOut, thrownException) + objOut = null } - objOut = null - wrappedStream = null - partitionStream = null + + if (wrappedStream != null) { + thrownException = tryCloseOrAddSuppressed(wrappedStream, thrownException) + wrappedStream = null + } + + if (partitionStream != null) { + thrownException = tryCloseOrAddSuppressed(partitionStream, thrownException) + partitionStream = null + } + + thrownException.foreach(throw _) + isOpen = false updateBytesWritten() } + private def tryCloseOrAddSuppressed( + closeable: Closeable, prevException: Option[IOException]): Option[IOException] = { + var resolvedException = prevException + try { + closeable.close() + } catch { + case e: IOException => + resolvedException = prevException.map(presentPrev => { + presentPrev.addSuppressed(e) + presentPrev + }).orElse(Some(e)) + } + resolvedException Review comment: I also prefer Imran's approach. I'm just a tiny bit worried about bad stream implementations that don't have an idempotent `close()`, since both your code and Imran's are calling it multiple times on certain streams. Probably ok not to deal with that though. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org