squito commented on a change in pull request #23453: [SPARK-26089][CORE] Handle
corruption in large shuffle blocks
URL: https://github.com/apache/spark/pull/23453#discussion_r263602188
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -335,6 +336,44 @@ private[spark] object Utils extends Logging {
}
}
+ /**
+ * Copy all data from an InputStream to an OutputStream upto maxSize and
+ * close the input stream if all data is read.
+ * @return A tuple of boolean, which is whether the stream was fully copied,
and an InputStream,
+ * which is a combined stream of read data and any remaining data
+ */
+ def copyStreamUpTo(in: InputStream, maxSize: Long): (Boolean, InputStream) =
{
+ var count = 0L
+ val out = new ChunkedByteBufferOutputStream(64 * 1024, ByteBuffer.allocate)
+ val streamCopied = tryWithSafeFinally {
Review comment:
nit: I think a better name for `streamCopied` would be `fullyCopied` ...
everytime I come back to this code I always get a bit confused that maybe no
copying is happening in some situations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
