[
https://issues.apache.org/jira/browse/HDFS-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994067#comment-15994067
]
Yongjun Zhang commented on HDFS-11661:
--------------------------------------
HI [~jojochuang],
Thanks for working on this, it seems a very good fix to solve the issue
reported here. I did a review and below are my comments.
1. There are the following scenarios:
a. mv fileX fileY (fileY is a child of the same snapshot root)
b. mv fileX fileY (fileY is not a child of the same snapshot root)
c. mv fileX fileY, take snapshot, mv fileY fileX
d. delete fileX
where b/d can be handled same way, and a/c can be handled same way. HDFS-10797
already has a unit test, I'm not sure whether it covers all scenario, if not,
we can extend the test to cover these scenarios. So to trace the behavior of
the changed code to see if that the code works as expected for different
scenarios.
2. The following code covers scenario c, it seems the calculation
{{computeContentSummary}} done in this {{if}} branch is already done before
this code is reached, and we should not call it again here. Let's prove that
with the unit test.
{code}
if (nodePath.equals(inodeInMapPath)) {
LOG.info("snapshot inode " + nodePath + " was deleted or " +
"was renamed+renamed back.");
inodeInMap.computeContentSummary(Snapshot.CURRENT_STATE_ID, this);
} else {
{code}
Thanks.
> GetContentSummary uses excessive amounts of memory
> --------------------------------------------------
>
> Key: HDFS-11661
> URL: https://issues.apache.org/jira/browse/HDFS-11661
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.8.0, 3.0.0-alpha2
> Reporter: Nathan Roberts
> Assignee: Wei-Chiu Chuang
> Priority: Blocker
> Attachments: HDFS-11661.001.patch, HDFs-11661.002.patch, Heap
> growth.png
>
>
> ContentSummaryComputationContext::nodeIncluded() is being used to keep track
> of all INodes visited during the current content summary calculation. This
> can be all of the INodes in the filesystem, making for a VERY large hash
> table. This simply won't work on large filesystems.
> We noticed this after upgrading a namenode with ~100Million filesystem
> objects was spending significantly more time in GC. Fortunately this system
> had some memory breathing room, other clusters we have will not run with this
> additional demand on memory.
> This was added as part of HDFS-10797 as a way of keeping track of INodes that
> have already been accounted for - to avoid double counting.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
