Ahmed Hussein created MAPREDUCE-7325: ----------------------------------------
Summary: Intermediate data encryption is broken in LocalJobRunner Key: MAPREDUCE-7325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security Reporter: Ahmed Hussein With enabling intermediate encryption, running a job using LocalJobRunner with multiple reducers fails with the following error: {code:bash} 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1344328673_0004 java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#5 at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#5 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error at org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) at org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) at org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) {code} The bug can be reproduced with any test unit like TestLocalJobSubmission. Another way to reproduce the bug is to run {{LargeSorter}} with multiple reducers. {code:java} Configuration config = new Configuration(); // set all the necessary configurations config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); String[] args = new String[] {"output-dir"}; int res = ToolRunner.run(config, new LargeSorter(), args); {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org