[
https://issues.apache.org/jira/browse/HDFS-14276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889635#comment-16889635
]
Ayush Saxena commented on HDFS-14276:
-------------------------------------
Thanx Everyone for the discussion here. Seems something good to have. Checked
the test failures, All but {{TestNameNodeMXBean}} passed at my local.
{{TestNameNodeMXBean}} checks the name dir size and expects it be updated as
part of tailing edits, which we changed to be update as part of log rolling, so
we need to reduce the time period for log rolling, and minimum a transaction,
to get log roll triggered, to get the test going.
> [SBN read] Reduce tailing overhead
> ----------------------------------
>
> Key: HDFS-14276
> URL: https://issues.apache.org/jira/browse/HDFS-14276
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, namenode
> Affects Versions: 3.3.0
> Environment: Hardware: 4-node cluster, each node has 4 core, Xeon
> 2.5Ghz, 25GB memory.
> Software: CentOS 7.4, CDH 6.0 + Consistent Reads from Standby, Kerberos, SSL,
> RPC encryption + Data Transfer Encryption.
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Attachments: HDFS-14276.000.patch, Screen Shot 2019-02-12 at 10.51.41
> PM.png, Screen Shot 2019-02-14 at 11.50.37 AM.png
>
>
> When Observer sets {{dfs.ha.tail-edits.period}} = {{0ms}}, it tails edit log
> continuously in order to fetch the latest edits, but there is a lot of
> overhead in doing so.
> Critically, edit log tailer should _not_ update NameDirSize metric every
> time. It has nothing to do with fetching edits, and it involves lots of
> directory space calculation.
> Profiler suggests a non-trivial chunk of time is spent for nothing.
> Other than this, the biggest overhead is in the communication to
> serialize/deserialize messages to/from JNs. I am looking for ways to reduce
> the cost because it's burning 30% of my CPU time even when the cluster is
> idle.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
