[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966423#comment-15966423
]
Andrew Wang commented on HDFS-9260:
-----------------------------------
Hi Daryn,
I talked to our people who help run this large customer cluster. It's at about
350 million blocks, so a pretty good size, but also a lot denser than the last
published stats I saw about the 4500-node Yahoo cluster. We don't have
historical GC metrics going back a year when we put this into CDH, but they
haven't seen anything abnormal in terms of GC.
They were quite interested your balancer settings though, since we haven't seen
it stressing the NN. Could you provide the following?
{noformat}
dfs.datanode.balance.bandwidthPerSec
dfs.datanode.balance.max.concurrent.moves
dfs.namenode.replication.work.multiplier.per.iteration
dfs.namenode.replication.max-streams-hard-limit
{noformat}
I believe we're running it with mostly default settings like this:
{noformat}
hdfs balancer -Ddfs.datanode.balance.max.concurrent.moves=200 -threshold 10
{noformat}
> Improve the performance and GC friendliness of NameNode startup and full
> block reports
> --------------------------------------------------------------------------------------
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
> Affects Versions: 2.7.1
> Reporter: Staffan Friberg
> Assignee: Staffan Friberg
> Fix For: 3.0.0-alpha1
>
> Attachments: FBR processing.png, HDFS-7435.001.patch,
> HDFS-7435.002.patch, HDFS-7435.003.patch, HDFS-7435.004.patch,
> HDFS-7435.005.patch, HDFS-7435.006.patch, HDFS-7435.007.patch,
> HDFS-9260.008.patch, HDFS-9260.009.patch, HDFS-9260.010.patch,
> HDFS-9260.011.patch, HDFS-9260.012.patch, HDFS-9260.013.patch,
> HDFS-9260.014.patch, HDFS-9260.015.patch, HDFS-9260.016.patch,
> HDFS-9260.017.patch, HDFS-9260.018.patch, HDFSBenchmarks2.zip,
> HDFSBenchmarks.zip, HDFS Block and Replica Management 20151013.pdf
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
