squito commented on a change in pull request #23453: [SPARK-26089][CORE] Handle
corruption in large shuffle blocks
URL: https://github.com/apache/spark/pull/23453#discussion_r245427177
##########
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -453,14 +454,21 @@ final class ShuffleBlockFetcherIterator(
input = streamWrapper(blockId, in)
// Only copy the stream if it's wrapped by compression or
encryption, also the size of
// block is small (the decompressed block is smaller than
maxBytesInFlight)
- if (detectCorrupt && !input.eq(in) && size < maxBytesInFlight / 3)
{
+ if (detectCorrupt && !input.eq(in)) {
isStreamCopied = true
+ streamCompressedOrEncrypted = true
Review comment:
I wonder if we should do the inputStream checks even if `detectCorrupt` is
false, and perhaps even change that default. Or add another flag just for
doing this memory copy, as its more expensive.
Also I think the comments around here need to be updated to explain what is
happening now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
