[
https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349509#comment-14349509
]
Jing Zhao commented on HDFS-7864:
---------------------------------
Hi [~demongaorui], thanks for the comments.
In our current implementation, DataNodes does not know any difference between a
contiguous block and a block belonging to a striped blockgroup. For example,
let's say a file consists of one striped block: b1 with block id
0x8000000000000000L. For a 6+3 encoding, b1 includes 6 data blocks and 3 parity
blocks, with block id 0x8000000000000000L, 0x8000000000000001L,
0x8000000000000002L, ..., 0x8000000000000008L, accordingly. DataNodes then only
see these data blocks and parity blocks, and the concept of striped block
(which is actually a block group) is only known by the NameNode (and maybe
client). In FSImage, for the striped block, we can still treat it as a normal
block except we need to store its number of data/parity blocks.
This is mainly to save NameNode memory. More details are in "NameNode Memory
Usage Reduction" section in the [design doc |
https://issues.apache.org/jira/secure/attachment/12697210/HDFSErasureCodingDesign-20150206.pdf].
> Erasure Coding: Update safemode calculation for striped blocks
> --------------------------------------------------------------
>
> Key: HDFS-7864
> URL: https://issues.apache.org/jira/browse/HDFS-7864
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Jing Zhao
> Assignee: GAO Rui
>
> We need to update the safemode calculation for striped blocks. Specifically,
> each striped block now consists of multiple data/parity blocks stored in
> corresponding DataNodes. The current code's calculation is thus inconsistent:
> each striped block is only counted as 1 expected block, while each of its
> member block may increase the number of received blocks by 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
