[
https://issues.apache.org/jira/browse/HDFS-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liuguanghua updated HDFS-17732:
-------------------------------
Description:
Now when a block commit , it should add expectedStorages into
PendingReconstructionBlocks if the ibr is not sent from expected locations. And
for a ec block group , only blkStriped.getRealTotalBlockNum() ==
expectedStorages.length will be added into PendingReconstructionBlocks.
The problem is that a samll file with ec, it may always
blkStriped.getRealTotalBlockNum() < expectedStorages.length if the file size <
ecPolicy.getNumDataUnits()*ecPolicy.getCellSize().
The following situations need to be considered：
(1) A EC file with XOR-2-1-1024k，when we write lower than 1024k into the
file，close it.
(2) Now The file has only one ec block group, indics=(0,1,2) and suppose only 0
replica ibr is received and 2 replica ibr is delayed. The ec block group
reaches minstorage and commit .
(3) When block group commit , blkStriped.getRealTotalBlockNum()=1
expectedStorages.length=3 , so it can not add this ec block group
expectedStorages into PendingReconstructionBlocks.
(4) The ec block group complete and only has minstorage and the the namenode
generates ec reconstruncation task to datanode.
(5) The 2 replicas ibr is recived.
(6) Datanode reconstruction task finished and ibr sends.
(7) (5) and (6) may in a same datanode
For other ec policy, in (7) , on a same datanode will has two different ec
block indic. For ec this is not expected and if it happens , in namenode will
be AddBlockResult.REPLACED replaced the former ibr. And it maybe increases the
ec reconstruncation task again.
was:
Now when a block is commit , it should add expectedStorages into
PendingReconstructionBlocks if the ibr is not sent from expected locations.
And for ec groups , only blkStriped.getRealTotalBlockNum() ==
expectedStorages.length will be added into PendingReconstructionBlocks.
But the following situations need to be considered：
(1)A EC file with XOR-2-1-1024k，when we write lower than 1024k into the
file，close it.
(2)Now The file has only one ec block group, indics=(0,1,2) and suppose only 0
replica ibr is received. Then the block group commit and will be completed.
(3) When block group commit , blkStriped.getRealTotalBlockNum()=1
expectedStorages.length=3 , so it can not add EC block expectedStorages in
PendingReconstructionBlocks
(4) This will result in the ec group reconstruction
(5) The 1 and 2 replicas ibr is recived.
(6) This maybe lead two same block group indics on same datanode if
reconstruction choose the same datanode
> addExpectedReplicasToPending should add EC block expectedStorages in
> PendingReconstructionBlocks
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-17732
> URL: https://issues.apache.org/jira/browse/HDFS-17732
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: liuguanghua
> Priority: Major
> Labels: pull-request-available
>
> Now when a block commit , it should add expectedStorages into
> PendingReconstructionBlocks if the ibr is not sent from expected locations.
> And for a ec block group , only blkStriped.getRealTotalBlockNum() ==
> expectedStorages.length will be added into PendingReconstructionBlocks.
> The problem is that a samll file with ec, it may always
> blkStriped.getRealTotalBlockNum() < expectedStorages.length if the file size
> < ecPolicy.getNumDataUnits()*ecPolicy.getCellSize().
> The following situations need to be considered：
> (1) A EC file with XOR-2-1-1024k，when we write lower than 1024k into the
> file，close it.
> (2) Now The file has only one ec block group, indics=(0,1,2) and suppose only
> 0 replica ibr is received and 2 replica ibr is delayed. The ec block group
> reaches minstorage and commit .
> (3) When block group commit , blkStriped.getRealTotalBlockNum()=1
> expectedStorages.length=3 , so it can not add this ec block group
> expectedStorages into PendingReconstructionBlocks.
> (4) The ec block group complete and only has minstorage and the the namenode
> generates ec reconstruncation task to datanode.
> (5) The 2 replicas ibr is recived.
> (6) Datanode reconstruction task finished and ibr sends.
> (7) (5) and (6) may in a same datanode
> For other ec policy, in (7) , on a same datanode will has two different ec
> block indic. For ec this is not expected and if it happens , in namenode will
> be AddBlockResult.REPLACED replaced the former ibr. And it maybe increases
> the ec reconstruncation task again.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
