[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951459#comment-15951459
]
Daryn Sharp commented on HDFS-9260:
-----------------------------------
I have no doubt this patch "works". If you are collecting GC metrics check if
you see elevation or spikes in frequency or cpu over historic levels. Roughly
what are the specs for the "highest scale" you have observed?
This blocks HDFS-7967 which is a critical feature for our clusters - It's been
a deployment blocker since 2.6. The balancer is unusable with 500 million to
over 1 billion replicas on dense storages. I have detailed in a comment on
HDFS-7967 that getBlocks will take hundreds of ms. Sustaining 20-40k average
ops/sec is impossible.
Rolling upgrades also already place very high GC pressure on the NN, some
narrowly escaping a full GC. I'm very worried if the NN has to re-sort FBRs
during the upgrade.
I'd like to stop internally maintaining HDFS-7967 and let the community
benefit. May we please revert?
> Improve the performance and GC friendliness of NameNode startup and full
> block reports
> --------------------------------------------------------------------------------------
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
> Affects Versions: 2.7.1
> Reporter: Staffan Friberg
> Assignee: Staffan Friberg
> Fix For: 3.0.0-alpha1
>
> Attachments: FBR processing.png, HDFS-7435.001.patch,
> HDFS-7435.002.patch, HDFS-7435.003.patch, HDFS-7435.004.patch,
> HDFS-7435.005.patch, HDFS-7435.006.patch, HDFS-7435.007.patch,
> HDFS-9260.008.patch, HDFS-9260.009.patch, HDFS-9260.010.patch,
> HDFS-9260.011.patch, HDFS-9260.012.patch, HDFS-9260.013.patch,
> HDFS-9260.014.patch, HDFS-9260.015.patch, HDFS-9260.016.patch,
> HDFS-9260.017.patch, HDFS-9260.018.patch, HDFSBenchmarks2.zip,
> HDFSBenchmarks.zip, HDFS Block and Replica Management 20151013.pdf
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
