[
https://issues.apache.org/jira/browse/MAPREDUCE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042451#comment-14042451
]
Chris Douglas commented on MAPREDUCE-5890:
------------------------------------------
The repeated config lookup and unit test are not blockers, but they're places
where the patch could be improved.
bq. The ShuffleHandler is a private class of MapReduce, if other frameworks use
it, it is at their own risk.
Every version of the patch has broken compatibility with existing versions of
_MapReduce_. Other frameworks may rely on functionality we don't guarantee, but
breaking them is avoidable.
bq. Regarding adding new abstractions, I’m OK if they are small and
non-intrusive. I just don’t want to send Arun chasing a goose a wild goose and
when he finally does we backtrack because the changes are too pervasive in the
core of MapReduce
Adding a new file just to pass 16 bytes to the {{ShuffleHandler}} will harm
performance; breaking backwards compatibility is not OK, and not necessary for
this feature. Aside from those, I've asked for some formatting fixes and that
the code not return an IV that doesn't match the hard-coded 16-byte size. These
are reasonable, limited requests and bug fixes, and I've suggested two possible
implementations that would address them. These would be blockers during the
merge, too.
> Support for encrypting Intermediate data and spills in local filesystem
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-5890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5890
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: security
> Affects Versions: 2.4.0
> Reporter: Alejandro Abdelnur
> Assignee: Arun Suresh
> Labels: encryption
> Attachments: MAPREDUCE-5890.3.patch, MAPREDUCE-5890.4.patch,
> org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-output.txt,
> syslog.tar.gz
>
>
> For some sensitive data, encryption while in flight (network) is not
> sufficient, it is required that while at rest it should be encrypted.
> HADOOP-10150 & HDFS-6134 bring encryption at rest for data in filesystem
> using Hadoop FileSystem API. MapReduce intermediate data and spills should
> also be encrypted while at rest.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
