[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815889#comment-15815889
]
Daryn Sharp commented on HDFS-9260:
-----------------------------------
I'd like to propose this patch be reverted on the NN, but perhaps not the DN.
Partially because it complicates porting HDFS-11310, but also due to
performance concerns.
Synopsis of the design change: Reference-dense datastructures with high
mutation rates are the enemy of young gen gc pauses. When a tenured reference
is updated, a dirty card tracks the memory region of the changed reference.
The next young gen gc checks traceability to references in the dirty card
regions. FBR processing rewrites triplets references to identity unreported
blocks, causing a spike in young gen gc load. A folded/sorted tree is a clever
means to avoid rewriting the triplets pointer during a FBR, thus reducing young
gen gc pressure.
That said, here's the main issues I see:
# FBRs: DNs must send sorted block reports, else the NN creates a temporary
sorted/folded tree for iteration. A new block report encoding was designed (by
me) for in-place iteration to reduce high object allocation rates that made FBR
processing unacceptable at scale. Building the sorted tree undoes the benefit
by exploding the entire report into a highly fragmented tree (out of order
insertion). During a rolling upgrade this will place extreme pressure on the
NN until all DNs are upgraded.
# IBRs: The performance of the tree is predicated on increasing block ids to
avoid fragmentation while filling the tree. However, block churn and organic
replication from dead nodes, decommissioning nodes, failed storages, block
balancing, etc will pepper a node with random blocks. The IBRs will cause
trees to quickly fragment. The tree mutations are likely to cause more dirty
cards than simply linking/unlinking a block into the triplets.
# Tree compaction: Every 10 mins all sufficiently fragmented storage trees
will be compacted. In practice this may be a large portion of the cluster
storages due to IBRs, translating to bursts of very heavy gc load. Heap growth
will increase due to defunct tree nodes.
# The CMS remark time reduction is not compelling when cycles should occur
every few days or a week if the heap is adequately sized.
The doc primarily focuses on FBRs with a footnote of 4X increase in IBR
processing and negative impacts to balancing. Impacts to balancing are
equivalent to replication from dead nodes, failed storages, decommissioning
nodes, invalidations when dead nodes rejoin, etc. FBR savings are great, but
not at the expense of increased processing and gc load from IBRs.
Unless there are real-world metrics with a large cluster under load that
dispute my concerns, I think we should revert.
> Improve the performance and GC friendliness of NameNode startup and full
> block reports
> --------------------------------------------------------------------------------------
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
> Affects Versions: 2.7.1
> Reporter: Staffan Friberg
> Assignee: Staffan Friberg
> Fix For: 3.0.0-alpha1
>
> Attachments: FBR processing.png, HDFS Block and Replica Management
> 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch,
> HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch,
> HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch,
> HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch,
> HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch,
> HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch,
> HDFSBenchmarks.zip, HDFSBenchmarks2.zip
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
