[
https://issues.apache.org/jira/browse/MAPREDUCE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-5890:
-----------------------------------
Attachment: MAPREDUCE-5890.4.patch
[~chris.douglas],
Updating patch..
As per your suggestion, I've refactored out the need for another file to store
the IV.
But I still could not find a consistent way for seamlessly handling the IV :
* The IV for the Spill file created at the end of the Map phase is now added to
the spill {{IndexRecord}} and is transmitted to the {{Fetcher}} via the
{{ShuffleHeader}} in the {{ShuffleHandler}}
* Unfortunately, the intermediate files created by the OnDisk mergers do not
have an index file associated with them. I was thus forced to write the IV as a
prefix into the stream. This happens when I "wrap" the outputStream, before
passing it to the {{IFile}} writer.
> Support for encrypting Intermediate data and spills in local filesystem
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-5890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5890
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: security
> Affects Versions: 2.4.0
> Reporter: Alejandro Abdelnur
> Assignee: Arun Suresh
> Labels: encryption
> Attachments: MAPREDUCE-5890.3.patch, MAPREDUCE-5890.4.patch,
> org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-output.txt,
> syslog.tar.gz
>
>
> For some sensitive data, encryption while in flight (network) is not
> sufficient, it is required that while at rest it should be encrypted.
> HADOOP-10150 & HDFS-6134 bring encryption at rest for data in filesystem
> using Hadoop FileSystem API. MapReduce intermediate data and spills should
> also be encrypted while at rest.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
