[
https://issues.apache.org/jira/browse/HDFS-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16734684#comment-16734684
]
Wei-Chiu Chuang commented on HDFS-13923:
----------------------------------------
Just like to pile up on the previous comments regarding the FileSystem cache --
it actually doesn't work the way you expected: see HADOOP-13971. The cache
never returns the same file system object in the cache, unfortunately. This is
why several downstreamer disable file system cache.
With that in mind, what approach should we take?
> Add a configuration to turn on/off observer reads
> -------------------------------------------------
>
> Key: HDFS-13923
> URL: https://issues.apache.org/jira/browse/HDFS-13923
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Chao Sun
> Priority: Major
>
> In some situations having a config to turn on/off observer reads dynamically
> may be useful. For instance, some applications may be sensitive for data
> freshness and want to always reach directly to active NN. In a more complex
> scenario, services such as Presto may want to apply observer reads for
> different types of queries. In this case, simply change
> {{dfs.client.failover.proxy.provider.<nameservice>}} may not be enough, since
> with FileSystem cache (which is usually turned on) will ignore the change and
> still use the same FileSystem object.
> Here I'm proposing to add a flag in {{HdfsClientConfigKeys}}, such as
> {{dfs.client.observer.reads.enabled}}, that can be used to dynamically turn
> on/off observer reads. The FileSystem cache key should also take account of
> this flag in its {{hashCode}} and {{equals}} impl, so that different
> FileSystem objects will be used depending on the flag.
>
> cc [~shv], [~xkrogen], [~vagarychen], [~zero45] for discussion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
