[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306260#comment-17306260
]
Jim Brennan commented on MAPREDUCE-7325:
Thanks for the fix [~ahussein]! +1 This change looks good to me. I will
commit later today.
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306172#comment-17306172
]
Ahmed Hussein commented on MAPREDUCE-7325:
--
[~Jim_Brennan] Can you please take a look at the patch?
The bug was due to an incorrect seek in the local fetcher. It works only if
encryption was disabled.
The changes are:
# The bug fix is just two lines change in LocalFetcher.java
# In {{TestMRIntermediateDataEncryption}}, I added an extra step to verify the
output of the jobs. This is done by comparing the checksum of the output files.
The reference checksum comes from a job that starts at the beginning of the
unit test with encryption disabled.
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305068#comment-17305068
]
Hadoop QA commented on MAPREDUCE-7325:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 48s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m
36s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
14s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 58s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304567#comment-17304567
]
Hadoop QA commented on MAPREDUCE-7325:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
6s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 17s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 18m
58s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
18s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/70/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt{color}
| {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch
generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291002#comment-17291002
]
Ahmed Hussein commented on MAPREDUCE-7325:
--
This bug was not detected before because the unit tests did not cover spill
encryption.
Example of these unit test is reported in MAPREDUCE-7322
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Priority: Major
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
