Github user winningsix commented on the pull request:
https://github.com/apache/spark/pull/8880#issuecomment-163466926
>>Yes, I agree that we should encrypt the compressed stream when writing.
However, BlockStoreShuffleReader should be opposite to the write path
>Sorry, I don't follow. Why does that mean the change cannot be made? Just
invert the streams ordering in both places. Writer wraps the compressed stream
with an encrypting stream, reader wraps the unencrypting stream with the
uncompressing stream.
Thanks @vanzin. The current order of wrapping the streams do exactly what
you originally suggested. For encryption, we compress the data and then encrypt
the compressed data. For decryption, we decrypt the data and then decompress
it.
To understand how it works needs a little further understanding of the
stream read and write process. Let's explain as following. You are right in
"Writer wraps the compressed stream with an encrypting stream" and the current
code follows the wrap order. For reading, we know we need to do the processes
in reverse order of writing. But this isn't achieved through reversing the
order of the streams. This is because when writing, the outer stream does its
work (compressing) first and then call the inner (wrapped) stream to do its
work (encrypting). While for reading, the outer stream first call the inner
(wrapped) stream to do the work (decrypting) and then the outer stream does its
work (decompressing). As a result, the order of wrapping the steams keeps the
same but the process of work actually reversed for reading and writing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
