[
https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634578#comment-15634578
]
Zhe Zhang commented on HDFS-10702:
----------------------------------
Thanks [~mackrorysd] for taking over this work! It's exciting to see this
feature moving forward.
IIUC, if a client issues a read with a "stale bound" which is fresher than the
SbNN's current latest TxId, the SbNN will tail edit logs from ANN, right?
My concern is that if a significant portion of read requests follow this
scenario (needs a fresher TxId), that will cause a high writeLock contention on
SbNN. At lease in our production cluster, over 99% of RPC requests are reads,
and NN relies on parallelism between the read requests to catch up with
incoming requests. If say 50% of those requests are sent to SbNN and 50% of
those request a fresher TxId (considering the edit log tailing frequency, this
is quite likely), SbNN is taking much higher writeLock contention than current
ANN.
Maybe we can consider combining this SbNN read logic with the ideas discussed
under HDFS-8940, to somehow combine the SbNN state with events from
{{inotify}}? Pinging [~drankye], [~mingma] and [~HuafengWang] for thoughts.
> Add a Client API and Proxy Provider to enable stale read from Standby
> ---------------------------------------------------------------------
>
> Key: HDFS-10702
> URL: https://issues.apache.org/jira/browse/HDFS-10702
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jiayi Zhou
> Assignee: Jiayi Zhou
> Priority: Minor
> Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch,
> HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch,
> HDFS-10702.006.patch, StaleReadfromStandbyNN.pdf
>
>
> Currently, clients must always talk to the active NameNode when performing
> any metadata operation, which means active NameNode could be a bottleneck for
> scalability. One way to solve this problem is to send read-only operations to
> Standby NameNode. The disadvantage is that it might be a stale read.
> Here, I'm thinking of adding a Client API to enable/disable stale read from
> Standby which gives Client the power to set the staleness restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
