[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143859#comment-15143859 ] Hudson commented on HADOOP-12699: - FAILURE: Integrated in Hadoop-trunk-Commit #9287 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9287/]) HADOOP-12699. TestKMS#testKMSProvider intermittently fails during 'test (wang: rev 8fdef0bd9d1ece560ab4e1a1ec7fc77c46a034bb) * hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java * hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/EagerKeyGeneratorKeyProviderCryptoExtension.java > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Fix For: 2.8.0 > > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch, > HADOOP-12699.10.1.patch, HADOOP-12699.10.patch, HADOOP-12699.repro.2, > HADOOP-12699.repro.patch, generated.10.html > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143907#comment-15143907 ] Xiao Chen commented on HADOOP-12699: Thanks all for the reviews and discussions, and Andrew for committing! I definitely learned a lot in this. :) > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Fix For: 2.8.0 > > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch, > HADOOP-12699.10.1.patch, HADOOP-12699.10.patch, HADOOP-12699.repro.2, > HADOOP-12699.repro.patch, generated.10.html > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142152#comment-15142152 ] Hadoop QA commented on HADOOP-12699: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 48s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 52s {color} | {color:green} trunk passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 57s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} trunk passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 56s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 53s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 55s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_72. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 42m 6s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12787391/HADOOP-12699.10.1.patch | | JIRA Issue | HADOOP-12699 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9a59bc737941 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / aeb13ef | | Default Java | 1.7.0_95 | | Multi-JDK versions |
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141924#comment-15141924 ] Andrew Wang commented on HADOOP-12699: -- Looks great! +1 committing this shortly. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch, > HADOOP-12699.10.patch, HADOOP-12699.repro.2, HADOOP-12699.repro.patch, > generated.10.html > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141938#comment-15141938 ] Andrew Wang commented on HADOOP-12699: -- Woops, looks like the patch went stale because of HADOOP-12764, and we didn't get a new jenkins run. Xiao, mind rebasing and posting a new patch? Thanks much. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch, > HADOOP-12699.10.patch, HADOOP-12699.repro.2, HADOOP-12699.repro.patch, > generated.10.html > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141953#comment-15141953 ] Andrew Wang commented on HADOOP-12699: -- FYI that Zhe is also working on HDFS-9644 which will again change that doc. Fingers crossed there aren't any conflicts, but you might want to hold off on rebasing until after 9644 goes in. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch, > HADOOP-12699.10.patch, HADOOP-12699.repro.2, HADOOP-12699.repro.patch, > generated.10.html > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142071#comment-15142071 ] Xiao Chen commented on HADOOP-12699: Thanks [~andrew.wang]. Looks like HDFS-9644 will change another document, so I went ahead and attached patch 10 against latest trunk. It was actually my fault in creating patch 10 - it was based on patch 9 change, so applying itself fails. I guess that somehow confused jenkins since there's no pre-commit comment. Now the 10.1 patch is generated by applying patch 9, then patch 10. I double checked that the generated html is identical to the attached one. Sorry for the confusion! > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch, > HADOOP-12699.10.1.patch, HADOOP-12699.10.patch, HADOOP-12699.repro.2, > HADOOP-12699.repro.patch, generated.10.html > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136428#comment-15136428 ] Andrew Wang commented on HADOOP-12699: -- Overall looks great, I checked again in the code and I agree with you about the KMSClientProvider. Not sure why that test worked before then. Building the docs is mentioned in BUILDING.txt, basically you call "mvn clean site -Preleasedocs", so can try that. A few review comments: * We have two kinds of caching, CachingKeyProvider (which caches keys) and EagerKeyGeneratorKPCE (which caches EEKs). It'd be good to add some introductory text explaining this, before talking about key caching (covered in the existing text) and EEK caching (the new text). In the content, it would also be good to say "encryption key" and "EEK" rather than just "key" to differentiate between the two. * Nit: "server-side" and "client-side" are hyphenated * Nit: "following property" -> "following properties" > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch, > HADOOP-12699.repro.2, HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127896#comment-15127896 ] Hadoop QA commented on HADOOP-12699: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 7s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 56s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 55s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 47s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 41m 16s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12785731/HADOOP-12699.09.patch | | JIRA Issue | HADOOP-12699 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux e51690e26d21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127042#comment-15127042 ] Hadoop QA commented on HADOOP-12699: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m 15s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 59s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 26s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 21s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 8s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 14s {color} | {color:red} hadoop-kms in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 91m 30s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.7.0_91 Failed junit tests | hadoop.crypto.key.kms.server.TestKMS | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12785579/HADOOP-12699.08.patch | | JIRA Issue | HADOOP-12699 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127206#comment-15127206 ] Xiao Chen commented on HADOOP-12699: The failure in {{TestACLs}} seems unrelated to this patch. I don't understand the NPEs in there though... > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.repro.2, > HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127351#comment-15127351 ] Wei-Chiu Chuang commented on HADOOP-12699: -- The NPE is unrelated to Hadoop at all. It's in Apache MINA. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.repro.2, > HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127350#comment-15127350 ] Andrew Wang commented on HADOOP-12699: -- Thanks for all the work here [~xiaochen], LGTM overall, just doc notes basically: * Need a {{}} tag to get a line break in Javadoc * Using single line comment in TestKMS rather than {{/**}} means that the {{@see}} doesn't work, so need to make it into actual Javadoc if you want this to work. I'll also note a little discrepancy in the unit test vs. actual usage, which is that we're using a CryptoExtension in the test rather than a KMSClientProvider. KMSClientProvider has another level of caching via ValueQueue, so it makes our story to users even more complicated. The KMS documentation is available here: {{hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}} and has a cache section. We could update it to mention the server and client side caching, and what kind of behavior can be expected. Basically, I as a user might have the following questions which the docs should answer: * What caches are present on the system? KMS and client. NN could be called out specifically as a KMS client. * What config keys control these caches? * What is my window of staleness after I roll a key? I'd expect server cache timeout + client cache timeout, but I don't know how eagerly Guava caches expire items. * Anything else you can think of > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.repro.2, > HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120323#comment-15120323 ] Zhe Zhang commented on HADOOP-12699: Great analysis [~xiaochen]! I think the above sequence does cause an EDEK generated from the old EZ key to be returned. The approach in the patch is reasonable. I wonder if there's anything we can do on the entrance of {{keyQueue}}? E.g. compare {{keyProviderCryptoExtension.getCurrentKey()}} and the {{encryptionKeyVersionName}} of the added key? > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.repro.2, HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111838#comment-15111838 ] Xiao Chen commented on HADOOP-12699: Zhe talked to me offline today regarding the sequence to reproduce the failure. Below is the order of the events happened to result in an old key being retrieved. Note that the race is only in the {{EagerKeyGeneratorKeyProviderCryptoExtension}} in {{KMSWebApp}}, so steps are only focused on the {{ValueQueue}} here. # Caller {{generateEncryptedKey}}, resulting in the server side KMSProvider's {{ValueQueue}} fired up an async filling task - a {{NamedRunnable}} is added into the {{UniqueKeyBlockingQueue}}(queue). # The {{NamedRunnable}} in step 1 is executed, filling in the local {{retEdeks}}, but not yet {{keyQueue.addAll}}. Code [here|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/EagerKeyGeneratorKeyProviderCryptoExtension.java#L83]. # Caller {{rollNewVersion}}, {{ValueQueue}} is drained. That is, the queue of the being rolled name in {{LoadingCache}}(keyQueues) is cleared. # The {{NamedRunnable}} continues to run, adding the old keys from local variable into {{keyQueue}}. # Caller side {{generateEncryptedKey}}, poll from the keyQueue, get old key. As a side note, simply not using the local variable wouldn't help, because even if > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch, > HADOOP-12699.07.patch, HADOOP-12699.repro.2, HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104215#comment-15104215 ] Hadoop QA commented on HADOOP-12699: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 1s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 56s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 35s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 52s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 36s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 15s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 33s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 48s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 39s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 57s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.7.0_91 Failed junit tests |
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104123#comment-15104123 ] Hadoop QA commented on HADOOP-12699: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 6s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 6s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 37s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 9s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 36s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 36s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s {color} | {color:red} hadoop-common-project/hadoop-common: patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 16s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 22s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 69m 8s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12782798/HADOOP-12699.06.patch | | JIRA Issue | HADOOP-12699 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux f06d0da1c8e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | |
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103845#comment-15103845 ] Arun Suresh commented on HADOOP-12699: -- Thanks you for digging into this [~xiaochen] !! I agree with your analysis that it is probably due to the async threads refilling the queue with old version of the key. you mentioned that : bq. I tried to 1) cancel all running threads for the key when draining, I would assume that would work.. can you point me to which patch had that fix ? I was thinking.. in the drain call, do an {{executors.shutdownnow()}}, clear all the queues and restart the executor threads. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.repro.2, > HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093610#comment-15093610 ] Hadoop QA commented on HADOOP-12699: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 51s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 32s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 53s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m 40s {color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 8s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 47s {color} | {color:green} the patch passed with JDK v {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 35s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 0s {color} | {color:red} hadoop-kms in the patch failed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 52s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 19s {color} | {color:red} hadoop-kms in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 99m 32s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_66 Failed junit tests | hadoop.crypto.key.kms.server.TestKMS | | JDK v1.8.0_66 Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle | | JDK v1.7.0_91 Failed junit tests | hadoop.crypto.key.kms.server.TestKMS | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL |
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094615#comment-15094615 ] Hadoop QA commented on HADOOP-12699: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 48s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 40s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 24s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 48s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 50s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 54s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 30s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 49s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 21s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.7.0_91 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12781869/HADOOP-12699.04.patch | | JIRA Issue | HADOOP-12699 | | Optional Tests | asflicense compile
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095032#comment-15095032 ] Andrew Wang commented on HADOOP-12699: -- Hi [~xiaochen], thanks for the investigation here, the analysis sounds correct to me. My question though is if we should fix the provider rather than changing the test. IMO from a user point of view, it's very weird to roll a key and then still see the old key version. It violates sequential consistency within a single thread. Have you looked at all at fixing the provider / value queue? Also ping [~zhz] for his thoughts. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095090#comment-15095090 ] Zhe Zhang commented on HADOOP-12699: Yes I'm having a similar thought while reading the patch and the discussions under HADOOP-11071. Agreed that we should fix the behavior of rollover. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094390#comment-15094390 ] Xiao Chen commented on HADOOP-12699: Patch 4 fixes the javac warnings. Last build seems to be problematic. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch, > HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093269#comment-15093269 ] Hadoop QA commented on HADOOP-12699: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 48s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 14s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 51s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 1s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 2s {color} | {color:red} root-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 730, now 731). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 44s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 21m 46s {color} | {color:red} root-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 724, now 725). {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 9s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 7s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 90m 24s {color} | {color:black}
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092714#comment-15092714 ] Hadoop QA commented on HADOOP-12699: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 21s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 40s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 51s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 5s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 5s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 34s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 34s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 52s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 32s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12781630/HADOOP-12699.02.patch | | JIRA Issue | HADOOP-12699 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux efeb5d2ea494
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092539#comment-15092539 ] Hadoop QA commented on HADOOP-12699: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 57s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 47s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 31s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 1s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 17s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s {color} | {color:red} Patch generated 4 new checkstyle issues in hadoop-common-project (total was 18, now 22). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 15s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 35s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 44s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 39s {color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 94m 8s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_66 Failed junit tests | hadoop.fs.TestLocalFsFCStatistics | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12781364/HADOOP-12699.01.patch | | JIRA
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090080#comment-15090080 ] Xiao Chen commented on HADOOP-12699: A sample failure is pasted here, but I don't think there's much information. Error Message {noformat} Values should be different. Actual: k6@0 {noformat} Stacktrace {noformat} java.lang.AssertionError: Values should be different. Actual: k6@0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failEquals(Assert.java:185) at org.junit.Assert.assertNotEquals(Assert.java:161) at org.junit.Assert.assertNotEquals(Assert.java:175) at org.apache.hadoop.crypto.key.kms.server.TestKMS$2.call(TestKMS.java:649) at org.apache.hadoop.crypto.key.kms.server.TestKMS$2.call(TestKMS.java:413) at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:130) at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:112) at org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSProvider(TestKMS.java:413) {noformat} Standard Output {noformat} Test KMS running at: http://localhost:50665/kms 2016-01-05 15:02:01,885 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.io.IOException: java.io.IOException: Exeption while contacting value generator at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.io.IOException: Exeption while contacting value generator at sun.reflect.GeneratedConstructorAccessor68.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:84) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:135) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more 2016-01-05 15:02:01,886 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371) at org.apache.hadoop.crypto.key.kms.server.EagerKeyGeneratorKeyProviderCryptoExtension$CryptoExtension$EncryptedQueueRefiller.fillQueueForKey(EagerKeyGeneratorKeyProviderCryptoExtension.java:77) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more 2016-01-05 15:02:01,886 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252) at
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090106#comment-15090106 ] Xiao Chen commented on HADOOP-12699: This can be reproduced by running the following codes in a loop (plus the assert). I can reproduce it within a handful of runs usually. {code} EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); kpce.rollNewVersion("k6"); EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); {code} This test is added in HADOOP-11071. From my understanding, the problem is from the async thread(s) in {{ValueQueue}}. (Probably what [~andrew.wang] said in [the comment in HADOOP-11071|https://issues.apache.org/jira/browse/HADOOP-11071?focusedCommentId=14125826=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14125826]). If I comment out the line {{submitRefillTask(keyName, keyQueue);}} in {{ValueQueue#getAtMost}}, the looped run can easily pass 10k runs without failing. I'm unsure whether this should be considered as a test only issue, or a bug. I'll have more readings first, and work on a solution. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[ https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090374#comment-15090374 ] Xiao Chen commented on HADOOP-12699: >From [~tucu00]'s [comment in >HADOOP-11071|https://issues.apache.org/jira/browse/HADOOP-11071?focusedCommentId=14125860=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14125860], > I feel this should be a test-only problem, and not a real scenario. Andrew >and Alejandro, please correct me if I'm wrong about this. Patch 1 is attached to fix the failure for the test. My thought is to disable the async filler tasks, since we just want to test the functionality of cached keys before and after rolling. I added a test-only configuration, to disable the async completely. Please review. Thanks! FYI - I initially wanted to change the configurations around the water mark to disable it, since {{ValueQueue#getAtMost}} has: {code} if (i <= (int) (lowWatermark * numValues)) { {code} But {{ValueQueue}} validates the parameters in constructor, so that's not doable. > TestKMS#testKMSProvider intermittently fails during 'test rollover draining' > > > Key: HADOOP-12699 > URL: https://issues.apache.org/jira/browse/HADOOP-12699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-12699.01.patch, HADOOP-12699.repro.patch > > > I've seen several failures of testKMSProvider, all failed in the following > snippet: > {code} > // test rollover draining > KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension. > createKeyProviderCryptoExtension(kp); > . > EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6"); > kpce.rollNewVersion("k6"); > EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6"); > Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(), > ekv2.getEncryptionKeyVersionName()); > {code} > with error message > {quote}Values should be different. Actual: k6@0{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)