[
https://issues.apache.org/jira/browse/SPARK-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827100#comment-15827100
]
Marcelo Vanzin commented on SPARK-19266:
----------------------------------------
So I started down the path of looking at exactly what's going on here and it's
a little more complicated than I first thought (and not as serious as I first
thought, too). There are 3 cases I can see:
* MEMORY_AND_DISK_SER with enough free memory: data is encrypted on memory. If
the block is later evicted to disk, the data on disk is encrypted.
* MEMORY_AND_DISK_SER without enough free memory: a chunk of the block may be
stored in memory, encrypted, but whatever does not fit will be spilled to disk,
unencrypted. This is in {{BlockManager.doPutIterator}} (see call to
{{partiallySerializedValues.finishWritingToStream}}).
* DISK_SER: data is encrypted on disk (see call to {{diskStore.put}} in
{{BlockManager.doPutIterator}} for the deserialized case)
So it seems the only gap is in the second case above, and it should be easy to
fix. The read part should already be handled, but it would be nice to add unit
tests.
> DiskStore does not encrypt serialized RDD data
> ----------------------------------------------
>
> Key: SPARK-19266
> URL: https://issues.apache.org/jira/browse/SPARK-19266
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Marcelo Vanzin
>
> {{DiskStore.putBytes()}} writes serialized RDD data directly to disk, without
> encrypting (or compressing) it. So any cached blocks that are evicted to disk
> when using {{MEMORY_AND_DISK_SER}} will not be encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
