tgravescs commented on issue #23453: [SPARK-26089][CORE] Handle corruption in large shuffle blocks URL: https://github.com/apache/spark/pull/23453#issuecomment-459499602 I think you are talking about: > but always do the on-the-fly check, even if detectCorrupt is false. You could add another option for this also, and have it default true, if really want to allow users to turn off all detection. so the only place detectCorrupt is used after this change is passed into BufferReleasingInputStream for the read, and its default is true. If we want to make the exception handling there always present, are you saying just remove the detectCorrupt config or are you suggesting something with the streamCompressedOrEncrypted?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
