[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902219#comment-16902219
]
Hudson commented on HDFS-14370:
-------------------------------
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17058 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17058/])
HDFS-14370. Add exponential backoff to the edit log tailer to avoid (xkrogen:
rev 827dbb11e24be294b40088a8aa46086ba8ca4ba8)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestEditLogTailer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Edit log tailing fast-path should allow for backoff
> ---------------------------------------------------
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
> Affects Versions: 3.3.0
> Reporter: Erik Krogen
> Assignee: Erik Krogen
> Priority: Major
> Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch,
> HDFS-14370.002.patch, HDFS-14370.003.patch, HDFS-14370.004.patch,
> HDFS-14370.005.patch
>
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
