[
https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HADOOP-2755:
-------------------------------------------
Attachment: 2755_20080131.patch
2755_20080131.patch: Fixed two reasons that fsck is slow:
- totalRacks and totalDatanodes, which are global system properties, are
computed for each file.
- When computing totalDatanodes, it calls
NameNode.getDatanodeReport(DatanodeReportType.LIVE) which requires super user
privilege.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after
> 6 hours (not finished yet). With the '-files' option turned on, it lists
> about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly
> (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
