[
https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420808#comment-13420808
]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
Would you please explain further? I'm not we should tie whether a fs has a
token to whether it has children. That assumption may be true for viewfs, but
isn't a requirement.
bq. FileSystems which embed others such as ViewFS will only need to implement
getChildFileSystems()
I think the proposals run counter to that goal. The methods
{{getCanonicalService}} and {{getDelegation}} are the simple atoms used by the
implementation of {{addDelegationTokens}} that an fs either does or does not
implement.
bq. the default impl of getCanaonicalName will return null if
getChildFileSystems() returns null
What would be the behavior when {{getChildFileSystems}} _does not_ return null?
Note that viewfs has to override it to always return null instead of the uri's
authority (mount table in this case). If the default is changed to always
return null, which I can do, it becomes another incompatible change...
bq. the defaul impl of getDelegationToken will return null if
getChildFileSystems() returns null
What would be the behavior when {{getChildFileSystems}} _does not_ return null?
The default is/was return null and the fs overrides only if it has a token.
The logic is currently: see if the fs itself has a token, yes, get it.
Irrespective, see if the fs has children, yes, repeat the process for each
child.
{code}
collectDelegationTokens(....) {
....
if (getCanonicalService() != null) { // get a token for myself
getDelegationToken(...);
}
foreach childFs : getChildFileSystems() { // ask each of my children for a
token
childFs.collectDelegationTokens(...);
}
}
{code}
It has to be recursive to allow for arbitrarily stacked filesystems. Only
allowing the top-level fs to have multiple children will not allow mergefs to
work.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch,
> HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch,
> HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR
> {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the
> knowledge to know if the tokens for a filesystem are available, which it
> can't possibly know for multi-token filesystems. Filtered filesystems are
> also problematic, such as har on viewfs. When mergeFs is implemented, it too
> will become a problem with the current implementation. Currently
> {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all
> the way down into the {{FileSystem}} level. The {{TokenCache}} should be
> ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
