[
https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202844#comment-13202844
]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
We should continue the discussion on MAPREDUCE-3825. There is no dispute that
multi-token support exists. However, it's broken in a number of ways, and is
incompatible with a {{TokenCache}} that assumes a 1 to 1 service/token mapping.
Regarding efficiency, leaking duplicate tokens everywhere increases the memory
load on the namenode, fills the edit logs, increases the txn burden, increases
startup time, increases GC time, leaked tokens aren't cancelled so they live
until they expire, and it increases the load on the secret manager.
It may be debated if duplicate tokens are a bug or performance, but fixing the
duplicate tokens is only incidentally fixed when fixing the bugs such as the
broken viewfs mount table.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch,
> HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR
> {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the
> knowledge to know if the tokens for a filesystem are available, which it
> can't possibly know for multi-token filesystems. Filtered filesystems are
> also problematic, such as har on viewfs. When mergeFs is implemented, it too
> will become a problem with the current implementation. Currently
> {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all
> the way down into the {{FileSystem}} level. The {{TokenCache}} should be
> ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
