[
https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HADOOP-1117:
-------------------------------------
Attachment: CpuPendingTransfer2.patch
pendingTranfer does not cause any replication if the replication factor has
already been achieved. Also, addStoredBlock() removes blocks from
neededReplication if the replicationfactor has already been achieved.
> DFS Scalability: When the namenode is restarted it consumes 80% CPU
> -------------------------------------------------------------------
>
> Key: HADOOP-1117
> URL: https://issues.apache.org/jira/browse/HADOOP-1117
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.0
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Priority: Blocker
> Fix For: 0.12.1
>
> Attachments: CpuPendingTransfer2.patch
>
>
> When the namenode is restarted, the datanodes register and each block is
> inserted into neededReplication. When the namenode exists, safemode it sees
> starts processing neededReplication. It picks up a block from
> neededReplication, sees that it has already has the required number of
> replicas, and continues to the next block in neededReplication. The blocks
> remain in neededReplication permanentlyhe namenode worker thread to scans
> this huge list of blocks once every 3 seconds. This consumes plenty of CPU on
> the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
