[
https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564628#action_12564628
]
Christian Kunz commented on HADOOP-2755:
----------------------------------------
The patch helps for fsck -- seems to run as fast as in previous releases.
Thank you, Nicholas.
I noticed that before the patch I needed to be the owner, now any user can run
fsck.
Als, the namenode is now up more than for 1 hour, and fsck shows that there are
still 25 blocks replicated with a single copy, and the replication monitor
consistently times out:
2008-02-01 03:41:24,184 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.pendingTransfer: ask datanode to replicate blk_2984271423661664080
to datanode(s) datanode1 datanode2
2008-02-01 03:51:14,104 WARN org.apache.hadoop.fs.FSNamesystem:
PendingReplicationMonitor timed out block blk_2984271423661664080
2008-02-01 03:51:22,303 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.pendingTransfer: ask datanode to replicate blk_2984271423661664080
to datanode(s) datanode3 datanode4
2008-02-01 04:01:14,150 WARN org.apache.hadoop.fs.FSNamesystem:
PendingReplicationMonitor timed out block blk_2984271423661664080
2008-02-01 04:01:19,344 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.pendingTransfer: ask datanode to replicate blk_2984271423661664080
to datanode(s) datanode5 datanode6
...
The datanode seems to be successfully transmitting the blocks:
2008-02-01 03:42:06,284 INFO org.apache.hadoop.dfs.DataNode: datanode Starting
thread to transfer block blk_2984271423661664080 to datanode1, datannode2
2008-02-01 03:42:09,535 INFO org.apache.hadoop.dfs.DataNode:
datanode:Transmitted block blk_2984271423661664080 to /datanode1
2008-02-01 03:42:06,284 INFO org.apache.hadoop.dfs.DataNode: datanode Starting
thread to transfer block blk_2984271423661664080 to datanode3,datanode4
2008-02-01 03:42:09,535 INFO org.apache.hadoop.dfs.DataNode:
datanode:Transmitted block blk_2984271423661664080 to /datanode3
Is this related?
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after
> 6 hours (not finished yet). With the '-files' option turned on, it lists
> about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly
> (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
