[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300522#comment-17300522
]
Ahmed Hussein commented on MAPREDUCE-7322:
------------------------------------------
Thanks [~Jim_Brennan] for the reviews.
h4. Unit test Failures:
* TestConfTest failure is unrelated. It is caused by HADOOP-17571. I see that
the failure is being discussed there
h4. List of Changes
* [^MAPREDUCE-7322.009.patch]:
** Address checkstyle error by adding javadoc comment to {{package-info.java}}
** change log level in {{SpillCallBackPathsFinder}} to debug-level.
* [^MAPREDUCE-7322.008.patch]:
** Address checkstyle errors: add {{package-info.java}}, fix lines more than
80 characters.
** Address findbugs warning about ignored return from {{mkdirs}}.
* [^MAPREDUCE-7322.007.patch]:
** Moved {{TestMRIntermediateDataEncryption}} to
{{org.apache.hadoop.mapreduce.}}
** Change the implementation of {{TestMRIntermediateDataEncryption}}
generating textInput, then counting the words. The configuration are set in a
way to guarantee that Spilling is triggered.
** I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers",
2, 4, false}} ) because of the bug reported in MAPREDUCE-7325.
** I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}}
and to expose the random sentence generator.
** Created {{org.apache.hadoop.mapreduce.security.SpillCallBackInjector}}
injecting callbacks while spilling files.
** Instead of changing {{CryptoUtils}} - which can be tricky for backward
compatibility and ecosystem- I preferred adding a new wrapper
{{org.apache.hadoop.mapreduce.security.IntermediateEncryptedStream}}.
** capturing the Spilled files implied changing the calls in: {{MapTask}},
{{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}},
{{BackupStore}}, {{MergeManagerImpl}}
** Added some utility methods in
{{org.apache.hadoop.mapreduce.util.MRJobConfUtil}} to organize output folders:
{{setLocalDirectoriesConfigForTesting}}, and
{{initEncryptedIntermediateConfigsForTesting}}.
** Applied initializations to other unit tests related to intermediate data:
{{TestMROpportunisticMaps}}, {{TestLocalJobSubmission}}, {{TestMerger}},
{{TestRecovery}}.
** Fixed a bug in the {{JarFinder}} where temporary file was not deleted in
test-dir
** Fixed a bug in {{TestMerger}} which was closing the dfs before Yarn cluster.
> revisiting TestMRIntermediateDataEncryption
> --------------------------------------------
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
> Reporter: Ahmed Hussein
> Assignee: Ahmed Hussein
> Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
