[
https://issues.apache.org/jira/browse/HADOOP-4810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654943#action_12654943
]
Hairong Kuang commented on HADOOP-4810:
---------------------------------------
The data got lost when we upgraded our cluster from 0.17 to 0.18. The root
cause of the problem is HADOOP-4702 which introduced a lot of corrupted blocks
into dfs and thus polluted the data.
> Data lost at cluster startup time
> ---------------------------------
>
> Key: HADOOP-4810
> URL: https://issues.apache.org/jira/browse/HADOOP-4810
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.2
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
>
> hadoop dfs -cat file1 returns
> dfs.DFSClient: Could not obtain block blk_XX_0 from any node:
> java.io.IOException: No live nodes contain current block
> Tracing the history of the block from NN log, we found
> WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block
> blk_-6160940519231606858_0 reported from A1.A2.A3.A4:50010 current size is
> 9303872 reported size is 262144
> WARN org.apache.hadoop.fs.FSNamesystem: Deleting block
> blk_-6160940519231606858_0 from A1.A2.A3.A4:50010
> INFO org.apache.hadoop.dfs.StateChange: DIR* NameSystem.invalidateBlock:
> blk_-6160940519231606858_0 on A1.A2.A3.A4:50010
> WARN org.apache.hadoop.fs.FSNamesystem: Error in deleting bad block
> blk_-6160940519231606858_0 org.apache.hadoop.dfs.SafeModeException: Cannot
> invalidate block blk_-6160940519231606858_0. Name node is in safe mode.
> WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block
> blk_-6160940519231606858_0 reported from B1.B2.B3.B4:50010 current size is
> 9303872 reported size is 306688
> WARN org.apache.hadoop.fs.FSNamesystem: Deleting block
> blk_-6160940519231606858_0 from B1.B2.B3.B4:50010
> INFO org.apache.hadoop.dfs.StateChange: DIR* NameSystem.invalidateBlock:
> blk_-6160940519231606858_0 on B1.B2.B3.B4:50010
> WARN org.apache.hadoop.fs.FSNamesystem: Error in deleting bad block
> blk_-6160940519231606858_0 org.apache.hadoop.dfs.SafeModeException: Cannot
> invalidate block blk_-6160940519231606858_0. Name node is in safe mode.
> INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.chooseExcessReplicates: (C1.C2.C3.C4:50010,
> blk_-6160940519231606858_0) is added to recentInvalidateSets
> INFO org.apache.hadoop.dfs.StateChange: BLOCK*
> NameSystem.chooseExcessReplicates: (D1.D2.D3.D4:50010,
> blk_-6160940519231606858_0) is added to recentInvalidateSets
> INFO org.apache.hadoop.dfs.StateChange: BLOCK* ask C1.C2.C3.C4:50010 to
> delete blk_-6160940519231606858_0
> INFO org.apache.hadoop.dfs.StateChange: BLOCK* ask D1.D2.D3.D4:50010 to
> delete blk_-6160940519231606858_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
