[
https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973802#comment-14973802
]
Yi Liu edited comment on HDFS-7984 at 10/26/15 6:27 AM:
{quote}
there is no way for an end user to create a token file with multiple tokens
inside it, short of building custom code to do it..
{quote}
No, I think we have. When using existing
{{Credentials#writeTokenStorageFile}}, all tokens of the credentials will be
persisted, and {{Credentials#readTokenStorageStream}} will read all tokens too.
So what we need to do is to add different tokens to the Credentials, use your
example, there are two hdfs, we can get the delegation tokens for each of them,
the {{service}} field of the two delegation tokens should be different, we can
add them to one {{Credentials}} or through the UGI api to add them into one
{{Credentials}}.
Actually even if we have multiple token files which contain only one token in
each, we can read them separately through
{{Credentials#writeTokenStorageFile}}, and add them to one {{Credentials}}.
Back to the original purpose of the JIRA, I don't know why we need to specify
multiple delegation tokens in one webhdfs://, the delegation token is used in
some service to access HDFS on behalf of user, so one hdfs only needs one
delegation token for one user. For the distcp example you said, I think
correct behavior is: user specify delegation token in each webhdfs://, and the
MR task will add the two delegation tokens to the UGI Credentials of that user.
I think this is already supported, I have not tried the distcp on two
different secured hdfs, if there is some bug, the correct fix is as I said,
it's not to support multiple delegation tokens in one webhdfs://.
We also should not use HADOOP_TOKEN_FILE_LOCATION to solve the problem.
was (Author: hitliuyi):
{quote}
there is no way for an end user to create a token file with multiple tokens
inside it, short of building custom code to do it..
{quote}
No, I think we have. When using existing
{{Credentials#writeTokenStorageFile}}, all tokens of the credentials will be
persisted, and {{Credentials#readTokenStorageStream}} will read all tokens too.
So what we need to do is to add different tokens to the Credentials, use your
example, there are two hdfs, we can get the delegation tokens for each of them,
the {{service}} filed of the two delegation tokens should be different, we can
add them to one {{Credentials}} or through the UGI api to add them into one
{{Credentials}}.
Actually even if we have multiple token files which contain only one token in
each, we can read them separately through
{{Credentials#writeTokenStorageFile}}, and add them to one {{Credentials}}.
Back to the original purpose of the JIRA, I don't know why we need to specify
multiple delegation tokens in one webhdfs://, the delegation token is used in
some service to access HDFS on behalf of user, so one hdfs only needs one
delegation token for one user. For the distcp example you said, I think
correct behavior is: user specify delegation token in each webhdfs://, and the
MR task will add the two delegation tokens to the UGI Credentials of that user.
I think this is already supported, I have not tried the distcp on two
different secured hdfs, if there is some bug, the correct fix is as I said,
it's not to support multiple delegation tokens in one webhdfs://.
We also should not use HADOOP_TOKEN_FILE_LOCATION to solve the problem.
> webhdfs:// needs to support provided delegation tokens
> --
>
> Key: HDFS-7984
> URL: https://issues.apache.org/jira/browse/HDFS-7984
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: HeeSoo Kim
>Priority: Blocker
> Attachments: HDFS-7984.patch
>
>
> When using the webhdfs:// filesystem (especially from distcp), we need the
> ability to inject a delegation token rather than webhdfs initialize its own.
> This would allow for cross-authentication-zone file system accesses.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
