[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655596#comment-16655596 ] Xiao Chen commented on HADOOP-15414: Thanks for following up [~ste...@apache.org]. I didn't test to verify my statement, but I believe HADOOP-14445 addresses this. [~hexiaoqiao] would you be able to verify? > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch, > HADOOP-15414-trunk.002.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at >
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655540#comment-16655540 ] Hadoop QA commented on HADOOP-15414: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} HADOOP-15414 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-15414 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12921113/HADOOP-15414-trunk.002.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15388/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch, > HADOOP-15414-trunk.002.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655538#comment-16655538 ] Steve Loughran commented on HADOOP-15414: - Does HADOOP-14445 address this? > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch, > HADOOP-15414-trunk.002.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) > at >
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458962#comment-16458962 ] Daryn Sharp commented on HADOOP-15414: -- I'm chasing KMS load issues, will check this out soon. I need to reacquaint myself with the history of the design (it took many people a lot of work). I do know one of the intentions is a filesystem subclasses should not be directly modifying the credentials. This patch circumvents the goal. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch, > HADOOP-15414-trunk.002.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457483#comment-16457483 ] genericqa commented on HADOOP-15414: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 56s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 34m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 31s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 35m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 35m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 37s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 27s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 43s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}158m 45s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | HADOOP-15414 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12921113/HADOOP-15414-trunk.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ede90908615a 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4 19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4844406 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14534/testReport/ | | Max. process+thread count | 1355 (vs. ulimit of 1) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14534/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Job submit not work
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457401#comment-16457401 ] He Xiaoqiao commented on HADOOP-15414: -- submit v2 patch and just delete one line code which is duplicated logic. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch, > HADOOP-15414-trunk.002.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) > at >
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455997#comment-16455997 ] He Xiaoqiao commented on HADOOP-15414: -- [~daryn],[~xiaochen], suggest that fix it using current patch (or another more graceful solution to quick fix it) since this problem exist in almost all versions. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at >
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454555#comment-16454555 ] genericqa commented on HADOOP-15414: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 45s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 28m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 48s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}124m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f | | JIRA Issue | HADOOP-15414 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12920816/HADOOP-15414-trunk.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 97bbc341417f 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4 19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 583a2f4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14528/testReport/ | | Max. process+thread count | 1467 (vs. ulimit of 1) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14528/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Job submit not work
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454482#comment-16454482 ] He Xiaoqiao commented on HADOOP-15414: -- [~daryn] {quote}That patch is not going to work correctly. It's likely to cause unintended recursive issues and just make token collection even messier.{quote} I test this patch using hadoop-2.7.1 with HDFS Federation in security mode where mount points are all HDFS cluster, it works well. The initial patch exactly is not the best solution (there may be more graceful solution) since current token mechanism is confused and maybe there are some historical reasons I think, and also trigger to get delegation token many times (it is positive correlated to number of sub cluster) in the initial patch and could bring some unnecessary overhead to KMS especially in large cluster. {quote}That design would allow the kms client to implement TokenIssuer, and DistributedFileSystem#getTokenIssuers to return the kms client, and for kms client to delete a big chunk of code. I think we can still do this. The default getTokenIssuers would return getChildFileSystems. I can mock up a POC if this doesn't make sense.{quote} It sounds a good idea and I look forward to your solution. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at >
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454452#comment-16454452 ] Wei-Chiu Chuang commented on HADOOP-15414: -- CC: [~xiaochen] [~shahrs87] I was just wondering how at-rest encryption works in Federation yesterday :) > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) > at
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454363#comment-16454363 ] Daryn Sharp commented on HADOOP-15414: -- That patch is not going to work correctly. It's likely to cause unintended recursive issues and just make token collection even messier. The root cause of this jira is {{DistributedFileSystem}} bent the (undocumented) rules for acquiring the kms token by overriding {{FileSystem#addDelegationTokens}}. The way it's supposed to work is a filesystem implements {{getDelegationToken}} to acquire its specific token. If the filesystem needs additional ancillary tokens, {{getChildFileSystems}} should return a list of those filesystems which in turn may implement {{getDelegationToken}} and maybe {{getChildFileSystems}}. This pattern was added specifically to support viewfs. So why is EZ broken through viewfs? {{FileSystem#addDelegationTokens}} via {{collectDelegationTokens}} will recursively invoke {{getDelegationToken}} on each filesystem and its children if it doesn't already have a token for the filesystem – _not_ {{addDelegationTokens}} which is what {{DistributedFileSystem}} has overridden to add the kms token. This in turn caused the kms client provider to copy the chunk of code that determined if a token was required and how to insert in the credentials. Back when I added these methods to support viewfs, here's what I really wanted to do: * make {{FileSystem#addDelegationTokens}} a final method (would have prevented this bug, but too late now) * add a {{TokenIssuer}} interface for get/cancel/renew/canonical service methods * {{getChildFileSystems}} should have been {{getTokenIssuers}} to allow filesystems to acquire non-filesystem tokens. ie. the kms client! That design would allow the kms client to implement {{TokenIssuer}}, and {{DistributedFileSystem#getTokenIssuers}} to return the kms client, and for kms client to delete a big chunk of code. I think we can still do this. The default {{getTokenIssuers}} would return {{getChildFileSystems}}. I can mock up a POC if this doesn't make sense. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at >
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454344#comment-16454344 ] He Xiaoqiao commented on HADOOP-15414: -- resubmit init patch V1 since the earlier version not rebase. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15414-trunk.001.patch > > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) > at >
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454323#comment-16454323 ] He Xiaoqiao commented on HADOOP-15414: -- hi [~shahrs87], On HDFS Federation, Job Client create ViewFileSystem instance, and it invoke fs#addDelegationTokens to obtain delegation tokens, when {{fs}} is instance of DistributedFileSystem, it collects delegation token for NameNode through FileSystem#addDelegationTokens firstly, then request delegation token for KMS if turn on HDFS Transparent Encryption, but if {{fs}} is instance of ViewFileSystem, it does not implement {{addDelegationTokens}} so invoke FileSystem#addDelegationTokens and it only collects delegation tokens for NameNode, so when job submit to ResourceManager it also not contain delegation token for KMS and Task is similar. In my opinion, maybe we can solve it which replace {{collectDelegationTokens}} with {{addDelegationTokens}} in FileSystem#collectDelegationTokens. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[ https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454183#comment-16454183 ] Rushabh S Shah commented on HADOOP-15414: - bq. IF on HDFS Federation, all delegation tokens for NameNode can collection correctly BUT delegation token for KMS not collect reference You need to debug why the job client was not able to collect kms delegation token. You need to have kms delegation token to talk to kms to decrypt edeks. > Job submit not work well on HDFS Federation with Transparent Encryption > feature > --- > > Key: HADOOP-15414 > URL: https://issues.apache.org/jira/browse/HADOOP-15414 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: He Xiaoqiao >Priority: Major > > When submit sample MapReduce job WordCount which read/write path under > encryption zone on HDFS Federation in security mode to YARN, task throws > exception as below: > {code:java} > 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, > Status : FAILED > Error: java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306) > at > org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300) > at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161) > at > org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478) > ... 21 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at >