Namenode performance degradation over time
------------------------------------------
Key: HADOOP-2576
URL: https://issues.apache.org/jira/browse/HADOOP-2576
Project: Hadoop
Issue Type: Bug
Components: dfs
Affects Versions: 0.16.0
Reporter: Christian Kunz
We have a cluster running the same applications again and again with a high
turnover of files.
The performance of these applications seem to be correlated to the lifetime of
the namenode:
After starting the namenode, the applications need increasingly more time to
complete, with about 50% more time after 1 week.
During that time the namenode average cpu usage increases from typically 10% to
30%, memory usage nearly doubles (although the average amount of data on dfs
stays the same), and the average load factor increases by a factor of 2-3
(although not significantly high, <2).
When looking at the namenode and datanode logs, I see a lot of asks to delete
blocks coming from the namenode for blocks not in the blockmap of the
datanodes, repeatedly for the same blocks.
When I counted the number of blocks asked by the namenode to be deleted, I
noticed a noticeable increase with the lifetime of the namenode (a factor of
2-3 after 1 week).
This makes me wonder whether the namenode does not purge the list of invalid
blocks from non-existing blocks.
But independently, the namenode has a degradation issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
