[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298789#comment-15298789
]
Hadoop QA commented on HADOOP-13155:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 3s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 25s
{color} | {color:red} root: The patch generated 3 new + 315 unchanged - 6 fixed
= 318 total (was 321) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 7s {color} |
{color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 32s
{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 25s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
| | hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.TestAsyncDFSRename |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805935/HADOOP-13155.03.patch
|
| JIRA Issue | HADOOP-13155 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 0ba26c401938 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297874#comment-15297874
]
Hadoop QA commented on HADOOP-13155:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 40s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 34s {color}
| {color:red} root generated 1 new + 696 unchanged - 1 fixed = 697 total (was
697) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 23s
{color} | {color:red} root: The patch generated 3 new + 315 unchanged - 6 fixed
= 318 total (was 321) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 44s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 33s
{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 56s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805818/HADOOP-13155.02.patch
|
| JIRA Issue | HADOOP-13155 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux c48de7f763ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294728#comment-15294728
]
Wei-Chiu Chuang commented on HADOOP-13155:
--
Hi [~xiaochen] thanks for contributing this patch.
# Could you also update kms documentation to include these new APIs? It would
be nice to have that, though I am ok without it.
# If you want to move the property {{dfs.encryption.key.provider.uri}} to
CommonConfigurationKeys, I think you should also move this property from
hdfs-default.xml to core-default.xml.
> Implement TokenRenewer in KMS and HttpFS
>
>
> Key: HADOOP-13155
> URL: https://issues.apache.org/jira/browse/HADOOP-13155
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13155.01.patch, HADOOP-13155.pre.patch
>
>
> Service DelegationToken (DT) renewal is done in Yarn by
> {{org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer}},
> where it calls {{Token#renew}} and uses ServiceLoader to find the renewer
> class
> ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java#L382]),
> and invokes the renew method from it.
> We seem to miss the token renewer class in KMS / HttpFSFileSystem, and hence
> Yarn defaults to {{TrivialRenewer}} for DT of such kinds, resulting in the
> token not being renewed.
> As a side note, {{HttpFSFileSystem}} does have a {{renewDelegationToken}}
> API, but I don't see it invoked in hadoop code base. KMS does not have any
> renew hook.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293018#comment-15293018
]
Hadoop QA commented on HADOOP-13155:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 31s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 20s {color}
| {color:red} root generated 1 new + 696 unchanged - 1 fixed = 697 total (was
697) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 23s
{color} | {color:red} root: The patch generated 16 new + 315 unchanged - 5
fixed = 331 total (was 320) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 55s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 30s
{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m 25s
{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 116m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805160/HADOOP-13155.01.patch
|
| JIRA Issue | HADOOP-13155 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ffc261e5d524 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9325845 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| javac |
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291689#comment-15291689
]
Hadoop QA commented on HADOOP-13155:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HADOOP-13155 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804893/HADOOP-13155.pre.patch
|
| JIRA Issue | HADOOP-13155 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9521/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Implement TokenRenewer in KMS and HttpFS
>
>
> Key: HADOOP-13155
> URL: https://issues.apache.org/jira/browse/HADOOP-13155
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13155.pre.patch
>
>
> Service DelegationToken (DT) renewal is done in Yarn by
> {{org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer}},
> where it calls {{Token#renew}} and uses ServiceLoader to find the renewer
> class
> ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java#L382]),
> and invokes the renew method from it.
> We seem to miss the token renewer class in KMS / HttpFSFileSystem, and hence
> Yarn defaults to {{TrivialRenewer}} for DT of such kinds, resulting in the
> token not being renewed.
> As a side note, {{HttpFSFileSystem}} does have a {{renewDelegationToken}}
> API, but I don't see it invoked in hadoop code base. KMS does not have any
> renew hook.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291654#comment-15291654
]
Arun Suresh commented on HADOOP-13155:
--
Thanks for taking this up [~xiaochen]..
IIRC, the reason why we had not implemented the renewer for KMS was that we
were catering to {{FileSystem}} and {{DistributedFileSystem}} API which at that
point had only an {{addDelegationTokens()}}.
So, I guess it is a good idea to add a Renewer for KMS-DTs. Will go over you
patch in more detail over the weekend.
bq. Since looks like MiniKMS is setup with PseudoAuthenticationHandler by
default (didn't find a way to configure auth type before starting MiniKMS.
I remember testing KMS via {{TestKMS}} using the
{{KerberosDelegationTokenAuthenticationHandler}}. Take a look at
{{TestKMS::testStartStop}} method where we do a
{{conf.set("hadoop.kms.authentication.type", "kerberos")}} before running the
KMS server.
> Implement TokenRenewer in KMS and HttpFS
>
>
> Key: HADOOP-13155
> URL: https://issues.apache.org/jira/browse/HADOOP-13155
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13155.pre.patch
>
>
> Service DelegationToken (DT) renewal is done in Yarn by
> {{org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer}},
> where it calls {{Token#renew}} and uses ServiceLoader to find the renewer
> class
> ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java#L382]),
> and invokes the renew method from it.
> We seem to miss the token renewer class in KMS / HttpFSFileSystem, and hence
> Yarn defaults to {{TrivialRenewer}} for DT of such kinds, resulting in the
> token not being renewed.
> As a side note, {{HttpFSFileSystem}} does have a {{renewDelegationToken}}
> API, but I don't see it invoked in hadoop code base. KMS does not have any
> renew hook.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284948#comment-15284948
]
Xiao Chen commented on HADOOP-13155:
Following a quick chat and advice from [~yzhangal], here's more details on how
other tokens are handled:
- {{DFSClient$Renewer}} handles {{HDFS_DELEGATION_TOKEN}}
- {{TokenAspect$TokenManager}} handles {{HFTP delegation}}, {{HSFTP
delegation}}, {{WEBHDFS delegation}}, {{SWEBHDFS delegation}}. (No httpfs, but
didn't find httpfs-specific kind either)
- There are other similar classes in yarn/mr to handle corresponding tokens
- As mentioned above, no renewer class handles {{kms-dt}}
> Implement TokenRenewer in KMS and HttpFS
>
>
> Key: HADOOP-13155
> URL: https://issues.apache.org/jira/browse/HADOOP-13155
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>
> DelegationToken is done in Yarn by {{DelegationTokenRenewer}}, where it calls
> {{Token#renew}} and uses ServiceLoader to get the renewer class
> ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java#L382])
> We seem to miss the token renewer class in KMS/HttpFSFileSystem, and hence
> Yarn defaults to {{TrivialRenewer}} for DT of these kinds, resulting in the
> token not renewed.
> As a side note, {{HttpFSFileSystem}} does have a {{renewDelegationToken}}
> API, but I don't see it invoked in hadoop code base. KMS does not have any
> renew hook.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284861#comment-15284861
]
Xiao Chen commented on HADOOP-13155:
Hm, seems I can't edit the description. Sorry for message flooding, but here's
more details:
I found this problem when debugging an issue with KMS DT renewal.
Token's code snippet to get renewer is like this:
{code}
private static ServiceLoader renewers =
ServiceLoader.load(TokenRenewer.class);
private synchronized TokenRenewer getRenewer() throws IOException {
if (renewer != null) {
return renewer;
}
renewer = TRIVIAL_RENEWER;
synchronized (renewers) {
for (TokenRenewer canidate : renewers) {
if (canidate.handleKind(this.kind)) {
renewer = canidate;
return renewer;
}
}
}
LOG.warn("No TokenRenewer defined for token kind " + this.kind);
return renewer;
{code}
And {{META-INF/services/org.apache.hadoop.security.token.TokenRenewer}} defines
each implementation. I didn't find any TokenRenewer implementation that handles
kind {{kms-dt}}. I wanted to look at {{HttpFSFileSystem}} for a reference since
they reuse the same auth codes in hadoop-common, but found out there's none.
(There's no TokenKind in HttpFS either, so may be that's not needed...) The
main intention of this jira is to allow KMS DTs to be able to get renewed.
> Implement TokenRenewer in KMS and HttpFS
>
>
> Key: HADOOP-13155
> URL: https://issues.apache.org/jira/browse/HADOOP-13155
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>
> DelegationToken is done in Yarn by {{DelegationTokenRenewer}}, where it calls
> {{Token#renew}} and uses ServiceLoader to get the renewer class
> ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java#L382])
> We seem to miss the token renewer class in KMS/HttpFSFileSystem, and hence
> Yarn defaults to {{TrivialRenewer}} for DT of these kinds, resulting in the
> token not renewed.
> As a side note, {{HttpFSFileSystem}} does have a {{renewDelegationToken}}
> API, but I don't see it invoked in hadoop code base. KMS does not have any
> renew hook.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13155) Implement TokenRenewer in KMS and HttpFS
[
https://issues.apache.org/jira/browse/HADOOP-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284837#comment-15284837
]
Xiao Chen commented on HADOOP-13155:
Ping [~tucu00] and [~asuresh] for advice. Am I correct that we're missing the
hook?
I see {{AuthenticationFilter#doFilter}} can handle DT operation requests using
its {{authHandler}}. So I imagine we can just add renewer implementations to
call {{DelegationTokenAuthenticatedURL#renewDelegationToken}}.
> Implement TokenRenewer in KMS and HttpFS
>
>
> Key: HADOOP-13155
> URL: https://issues.apache.org/jira/browse/HADOOP-13155
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>
> DelegationToken is done in Yarn by {{DelegationTokenRenewer}}, where it calls
> {{Token#renew}} and uses ServiceLoader to get the renewer class
> ([code|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java#L382])
> We seem to miss the token renewer class in KMS/HttpFSFileSystem, and hence
> Yarn defaults to {{TrivialRenewer}} for DT of these kinds, resulting in the
> token not renewed.
> As a side note, {{HttpFSFileSystem}} does have a {{renewDelegationToken}}
> API, but I don't see it invoked in hadoop code base. KMS does not have any
> renew hook.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
