[
https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348545#comment-14348545
]
GAO Rui commented on HDFS-7864:
-------------------------------
[~jingzhao]
{quote}
The current code's calculation is thus inconsistent: each striped block is only
counted as 1 expected block, while each of its member block may increase the
number of received blocks by 1.
{quote}
In above description, it looks like that DataNode will send information of both
StripedBlock and it's member blocks in BlockReport. I have checked
BlockManager.java, BlockInfoStriped.java and other related source codes.
In BlockReport, DataNode send StorageBlockReport[ ] to NameNode.
StorageBlockReport[ ] contains Maps of <DataNodeStrotage, long[ ]> , this long[
] array contains block Id (or StripedBlock id) which are stored in
DataNodeStrotage. The method getStoredBlock(Block block) in class
BlockManager.java can deal with StripedBlock. I am not sure wether DataNode
will send information of the StripedBlock's member blocks as well as
information of the StripedBlock itself. And, how can a DataNode send a report
about a StripedBlock while only one of its member blocks could be stored in
this DataNode? This question is based on the my image of EC logical process
below:
For example, a file (name:F, size: FS) is going to be stored in to HDFS with
EC(striping and ReedSolomon (6,3) schema).
If FS=128*6MB, F is truncated into 1024*6 slices(name: s1,s2,……s1024*6; each
size is 128kb) . So at last, F is stored in 6 date blocks (name: b1,b2…..b6)
with 3 parity blocks(name: pb1,pb2,pb3). And, b1 is consist of s1,s7,s13……. ;
b2 is consist of s2,s8,s14……….
So, in FSImage of HDFS, file F is a INodeFile with one BlockGroup, and this
BlockGroup is consist of b1,b2,b3,b4,b5,b6,pb1,pb2,pb3.
Did I get the correct image of EC logical process? Please let me know if I have
any misunderstanding.
> Erasure Coding: Update safemode calculation for striped blocks
> --------------------------------------------------------------
>
> Key: HDFS-7864
> URL: https://issues.apache.org/jira/browse/HDFS-7864
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Jing Zhao
> Assignee: GAO Rui
>
> We need to update the safemode calculation for striped blocks. Specifically,
> each striped block now consists of multiple data/parity blocks stored in
> corresponding DataNodes. The current code's calculation is thus inconsistent:
> each striped block is only counted as 1 expected block, while each of its
> member block may increase the number of received blocks by 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
