[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101356#comment-13101356
]
Daryn Sharp commented on MAPREDUCE-2764:
----------------------------------------
bq. Daryn, the service is set by the DelegationTokenFetcher, since it has to
happen over there anyways.
Ah, I see, however {{getDTFromRemote}} isn't the right place to set the
service. Forcing all tokens to be converted to hftp tokens in this outside
method is artificially limiting -- there's no reason why other tokens couldn't
be retrieved via the http protocol. It would equivalent to the rpc client
forcing all tokens to be a particular type.
{{HftpFileSystem}} is the correct place to stamp a token as an hftp token.
bq. The service isn't a generic field that is required to be <host>:<port>, it
is for the use of each kind of token. The current tokens use that format, but
it isn't a requirement. Obviously a shared library of methods to help are a
good thing and should be used.
Fully agreed. Tokens like job tokens are free to use and interpret the field
as they see fit.
However, connection based tokens must use {{setTokenService}} to abstract the
format of the service field. The method was added to eliminate the rampant
copy-n-paste code that made it very difficult to enable something as simple as
ip->host change. The abstraction also eliminated the extremely tight coupling
between the token selectors and a connection-based token's service.
Given the abstraction to encode the field, an equivalent function is needed to
abstract the decoding the field. Otherwise, n-many places will need to be
changed if the encoding or desired behavior (ex. ip->host!) needs to be
universally changed.
bq. That said, the current SecurityUtil class needs cleanup to remove all of
the redundant methods and provide a much cleaner and better documented
abstraction.
Also fully agreed!! But it's what we have for the sustaining release to avoid
incompatible API changes.
> Fix renewal of dfs delegation tokens
> ------------------------------------
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Daryn Sharp
> Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch,
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch,
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
> The problem is the JT's delegation token renewal code is built on brittle
> assumptions. The token's service field contains only the "ip:port" pair.
> The renewal process assumes that the scheme must be hdfs. If that fails due
> to a {{VersionMismatchException}}, it tries https based on another assumption
> that it must be hftp if it's not hdfs. A number of other exceptions, most
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since
> it won't fallback to https.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
