[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307115#comment-17307115 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan]! > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch, > MAPREDUCE-7322.branch-3.2.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302882#comment-17302882 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 55s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} branch-3.2 Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 2s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 27s{color} | {color:green}{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 39s{color} | {color:green}{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 59s{color} | {color:green}{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 46s{color} | {color:green}{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 28s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 7s{color} | {color:green}{color} | {color:green} branch-3.2 passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 28m 22s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 2m 21s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/68/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html{color} | {color:red} hadoop-common-project/hadoop-common in branch-3.2 has 2 extant spotbugs warnings. {color} | | {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 1m 30s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/68/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in branch-3.2 has 2 extant spotbugs warnings. {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 36s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 23s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 23s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 48s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/68/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 1 new + 1175 unchanged - 18 fixed = 1176 total (was 1193) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 35s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 55s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 40s{color} |
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302867#comment-17302867 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan] for the reviews and committing the patches. I ran the related tests locally and they were successful. The Yetus failure has been raised in YARN-10501. Hopefully, it will be fixed soon. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch, > MAPREDUCE-7322.branch-3.2.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302735#comment-17302735 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 15s{color} | {color:red}{color} | {color:red} Unprocessed flag(s): --spotbugs-strict-precheck {color} | \\ \\ || Subsystem || Report/Notes || | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/69/artifact/out/Dockerfile | | JIRA Issue | MAPREDUCE-7322 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13022403/MAPREDUCE-7322.branch-2.10.009.patch | | Console output | https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/69/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.3.1 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch, > MAPREDUCE-7322.branch-3.2.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302591#comment-17302591 ] Jim Brennan commented on MAPREDUCE-7322: [~ahussein], -MAPREDUCE-7320- has already been backported all the way to branch-2.10. Let me know if you want to provide additional patches for this one. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.3.1 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302092#comment-17302092 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan] for the review and merging the patch. In order to port this into branch-3.2 and branch-2.10, we will need to port MAPREDUCE-7320 because to pull {{GenericTestUtils}} changes. I can provide patches for both MAPREDUCE-7320 branch-3.2 and branch-2.10 if they do not pull clean. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.3.1 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302010#comment-17302010 ] Jim Brennan commented on MAPREDUCE-7322: Thanks for the contribution [~ahussein]! I have committed this to trunk and branch-3.3, but the patch does not apply to branch-3.2. Can you please provide a patch for 3.2 and any earlier branches you want this pulled back to? > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.3.1 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300646#comment-17300646 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 45s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 4s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 50s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 8s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 3s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 48s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 34m 16s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m 55s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 27s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 3s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 3s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 59s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 59s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 4m 14s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/67/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 1 new + 1179 unchanged - 18 fixed = 1180 total (was 1197) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 3s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green}
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300597#comment-17300597 ] Jim Brennan commented on MAPREDUCE-7322: Thanks for all the work on this [~ahussein]! I am +1 on patch 009, pending precommit build results. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300522#comment-17300522 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan] for the reviews. h4. Unit test Failures: * TestConfTest failure is unrelated. It is caused by HADOOP-17571. I see that the failure is being discussed there h4. List of Changes * [^MAPREDUCE-7322.009.patch]: ** Address checkstyle error by adding javadoc comment to {{package-info.java}} ** change log level in {{SpillCallBackPathsFinder}} to debug-level. * [^MAPREDUCE-7322.008.patch]: ** Address checkstyle errors: add {{package-info.java}}, fix lines more than 80 characters. ** Address findbugs warning about ignored return from {{mkdirs}}. * [^MAPREDUCE-7322.007.patch]: ** Moved {{TestMRIntermediateDataEncryption}} to {{org.apache.hadoop.mapreduce.}} ** Change the implementation of {{TestMRIntermediateDataEncryption}} generating textInput, then counting the words. The configuration are set in a way to guarantee that Spilling is triggered. ** I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2, 4, false}} ) because of the bug reported in MAPREDUCE-7325. ** I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and to expose the random sentence generator. ** Created {{org.apache.hadoop.mapreduce.security.SpillCallBackInjector}} injecting callbacks while spilling files. ** Instead of changing {{CryptoUtils}} - which can be tricky for backward compatibility and ecosystem- I preferred adding a new wrapper {{org.apache.hadoop.mapreduce.security.IntermediateEncryptedStream}}. ** capturing the Spilled files implied changing the calls in: {{MapTask}}, {{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}}, {{BackupStore}}, {{MergeManagerImpl}} ** Added some utility methods in {{org.apache.hadoop.mapreduce.util.MRJobConfUtil}} to organize output folders: {{setLocalDirectoriesConfigForTesting}}, and {{initEncryptedIntermediateConfigsForTesting}}. ** Applied initializations to other unit tests related to intermediate data: {{TestMROpportunisticMaps}}, {{TestLocalJobSubmission}}, {{TestMerger}}, {{TestRecovery}}. ** Fixed a bug in the {{JarFinder}} where temporary file was not deleted in test-dir ** Fixed a bug in {{TestMerger}} which was closing the dfs before Yarn cluster. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300034#comment-17300034 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 50s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 22s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 44s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 59s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 4s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 3s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 47s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 34m 23s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 25s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 11s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 11s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 0s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 0s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 4m 12s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/66/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 1 new + 1179 unchanged - 18 fixed = 1180 total (was 1197) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 1s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green}
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299887#comment-17299887 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 32s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 32s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 34s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 43s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 9s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 3s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 43s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 34m 8s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 22s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 1s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 2s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 56s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 56s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 4m 10s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/65/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 2 new + 1179 unchanged - 18 fixed = 1181 total (was 1197) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 6s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green}
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299219#comment-17299219 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 21s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 2s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 26s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 51s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 22m 51s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 48s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 16s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 1s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 59s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 58s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m 58s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 1s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 1s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 4m 29s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/64/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 5 new + 1864 unchanged - 18 fixed = 1869 total (was 1882) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} |
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299154#comment-17299154 ] Jim Brennan commented on MAPREDUCE-7322: A couple more comments. * Instead of putting the new files in mapreduce/lib/spill, I think putting them in mapreduce/security might make more sense? * Is it possible in TestMRIntermediateDataEncryption to run the test once without spill encryption enabled and verify that it produces the same output? > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299144#comment-17299144 ] Jim Brennan commented on MAPREDUCE-7322: {quote} I think that I should add a configuration to set the class implementation of the callback injector. This would allow starting the runtime on a cluster with the injector enabled. I ruled out making the configuration per Job as this would imply parsing the JobConf everytime (not a no-op for production). {quote} I'm not sure this is a good idea. Sounds like a potential security hole. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299031#comment-17299031 ] Jim Brennan commented on MAPREDUCE-7322: Great work [~ahussein]. Overall this looks excellent. Just a few minor comments: GenericTestUtils * I’m not sure why we need convertCamelToHyphen(). I can understand that the directory names would be more readable, but they would no longer match the test case method name, making it harder to match up the test dirs with the test cases? Am I missing something? * The methods relating to initializing spill encryption configs do not belong in Common. These need to be somewhere in the MAPREDUCE project. initEncryptedIntermediateConfigs(), setLocalDirectoriesConfig() MapTask * remove TODO comment (as you noted) TestGenericTestUtils * not sure about need for testConvertCamelCase() (see above) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17298877#comment-17298877 ] Ahmed Hussein commented on MAPREDUCE-7322: -- I think that I should add a configuration to set the class implementation of the callback injector. This would allow starting the runtime on a cluster with the injector enabled. I ruled out making the configuration per Job as this would imply parsing the JobConf everytime (not a no-op for production). > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297703#comment-17297703 ] Hadoop QA commented on MAPREDUCE-7322: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 47s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 10 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 47s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 31s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 4s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 2s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 48s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 59s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 25s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 3s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 3s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 54s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 54s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 4s{color} | {color:green}{color} | {color:green} root: The patch generated 0 new + 1163 unchanged - 18 fixed = 1163 total (was 1181) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 4s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 1s{color} | {color:green}{color} | {color:green}
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297487#comment-17297487 ] Ahmed Hussein commented on MAPREDUCE-7322: -- List of changes: * Moved {{TestMRIntermediateDataEncryption}} to {{org.apache.hadoop.mapreduce}}. * Change the implementation of {{TestMRIntermediateDataEncryption}} generating textInput, then counting the words. The configuration are set in a way to guarantee that Spilling is triggered. * I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2, 4, false}} ) because of the bug reported in MAPREDUCE-7325. * I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and to expose the random sentence generator. * Created {{SpillCallBackInjector}} injecting callbacks while spilling files. * Instead of changing {{CryptoUtils}} - which can be tricky for backward compatibility and ecosystem- I preferred adding a new wrapper {{IntermediateEncryptedStream}}. * capturing the Spilled files implied changing the calls in: {{MapTask}}, {{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}}, {{BackupStore}}, {{MergeManagerImpl}} * Added some utility methods in {{GenericTestUtils}} to organize output folders. * Applied initializations to other unit tests related to intermediate data. * Fixed a bug in the {{JarFinder}} where temporary file was not deleted in {{test-dir}} > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297490#comment-17297490 ] Ahmed Hussein commented on MAPREDUCE-7322: -- I submitted [MAPREDUCE-7322.005.patch|https://issues.apache.org/jira/secure/attachment/13021872/MAPREDUCE-7322.005.patch] fixing the check styles warnings. [~Jim_Brennan] Can you please take a look at the patch? > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296395#comment-17296395 ] Hadoop QA commented on MAPREDUCE-7322: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 10 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 51s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 45s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 24s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 32s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 6s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 23m 30s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 0s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 29s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 4s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 4s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 59s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 59s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 4m 9s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/61/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 2 new + 1163 unchanged - 18 fixed = 1165 total (was 1181) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 5s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} |
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294905#comment-17294905 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 8 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 43s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 23s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 51s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 4s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 26s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 6s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 1s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 44s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 26s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 57s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m 57s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 55s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 55s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 50s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/60/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 3 new + 630 unchanged - 15 fixed = 633 total (was 645) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 4s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} |
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293996#comment-17293996 ] Hadoop QA commented on MAPREDUCE-7322: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 40s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 20s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 2s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 19s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 3s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 58s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 42s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 26s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 52s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m 52s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 54s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 54s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 53s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/59/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 11 new + 630 unchanged - 13 fixed = 641 total (was 643) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 3s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} |