[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300034#comment-17300034 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 50s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 22s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 44s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 59s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 4s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 3s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 47s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 34m 23s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 25s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 11s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 11s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 0s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 0s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 4m 12s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/66/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 1 new + 1179 unchanged - 18 fixed = 1180 total (was 1197) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 1s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green}
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.008.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299887#comment-17299887 ] Hadoop QA commented on MAPREDUCE-7322: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 32s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 32s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 34s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 43s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 9s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 3s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 43s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 34m 8s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 22s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 1s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 2s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 56s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 56s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 4m 10s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/65/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 2 new + 1179 unchanged - 18 fixed = 1181 total (was 1197) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 6s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green}
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.007.patch Status: Patch Available (was: Open) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.007.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: (was: MAPREDUCE-7322.007.patch) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7182) MapReduce input format/record readers to support S3 select queries
[ https://issues.apache.org/jira/browse/MAPREDUCE-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7182. --- Resolution: Won't Fix > MapReduce input format/record readers to support S3 select queries > -- > > Key: MAPREDUCE-7182 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7182 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Major > > HADOOP-15229 adds S3 select through the (new) async openFile API, but the > classic RecordReader can't handle it because > # the files are shorter than they are in a getFileStatus, and the readers > assume that an EOFException is an error in that situation > # everything assumes plain text is splitable > # if a file has a gz extension, the gunzip codec should be used. So breaks > transcoded/uncompressed data > to handle s3 select data sources we need to be able to address them, either > through changes to the existing code (danger?) or some new readers -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org