[
https://issues.apache.org/jira/browse/HDFS-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736017#comment-16736017
]
Erik Krogen commented on HDFS-13923:
------------------------------------
{quote}
The cache never returns the same file system object in the cache, unfortunately.
{quote}
[~jojochuang], please correct me if I'm wrong, but I think this statement is
only true if you assume that a new UGI is created (e.g. via
{{createProxyUser()}}) between the two instantiations. In applications which
use the login user, or create the proxy user once and then use it for a long
period of time, the FileSystem cache is actually effective. I have personal
experience with applications that have disabled the cache and then experienced
massive performance degradation.
> Add a configuration to turn on/off observer reads
> -------------------------------------------------
>
> Key: HDFS-13923
> URL: https://issues.apache.org/jira/browse/HDFS-13923
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Chao Sun
> Assignee: Shweta
> Priority: Major
>
> In some situations having a config to turn on/off observer reads dynamically
> may be useful. For instance, some applications may be sensitive for data
> freshness and want to always reach directly to active NN. In a more complex
> scenario, services such as Presto may want to apply observer reads for
> different types of queries. In this case, simply change
> {{dfs.client.failover.proxy.provider.<nameservice>}} may not be enough, since
> with FileSystem cache (which is usually turned on) will ignore the change and
> still use the same FileSystem object.
> Here I'm proposing to add a flag in {{HdfsClientConfigKeys}}, such as
> {{dfs.client.observer.reads.enabled}}, that can be used to dynamically turn
> on/off observer reads. The FileSystem cache key should also take account of
> this flag in its {{hashCode}} and {{equals}} impl, so that different
> FileSystem objects will be used depending on the flag.
>
> cc [~shv], [~xkrogen], [~vagarychen], [~zero45] for discussion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
