[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated MAPREDUCE-7325:
-----------------------------------
Fix Version/s: 3.2.3
2.10.2
3.3.1
3.1.5
3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the contribution [~ahussein]! I have committed this to trunk -
branch-2.10.
> Intermediate data encryption is broken in LocalJobRunner
> --------------------------------------------------------
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
> Reporter: Ahmed Hussein
> Assignee: Ahmed Hussein
> Priority: Major
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
