[
https://issues.apache.org/jira/browse/HDFS-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353844#comment-14353844
]
Arpit Agarwal commented on HDFS-7820:
-------------------------------------
Hi [~andreina],
Sorry about the delay in responding. I see what you are saying. The blocks are
not deleted right away since {{processFirstBlockReport}} does not schedule
deletion of the unnecessary blocks.
Rather than add a hard-coded increment we can reserve the msb of the blockid
for an epoch. The epoch is toggled on every rollback. The low order 63 bits are
incremented sequentially as before on block allocation. Now we will not hit any
collisions with block ids allocated before the rollback.
The only drawback I can see is if a second rollback is attempted before all
excess blocks from the previous rollback were deleted. We could increase the
number of epoch bits to avoid collisions in that pathological case.
> Client Write fails after rolling upgrade rollback with "<block_id> already
> exist in finalized state"
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-7820
> URL: https://issues.apache.org/jira/browse/HDFS-7820
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: J.Andreina
> Assignee: J.Andreina
> Attachments: HDFS-7820.1.patch
>
>
> Steps to Reproduce:
> ===================
> Step 1: Prepare rolling upgrade using "hdfs dfsadmin -rollingUpgrade prepare"
> Step 2: Shutdown SNN and NN
> Step 3: Start NN with the "hdfs namenode -rollingUpgrade started" option.
> Step 4: Executed "hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT>
> upgrade" and restarted Datanode
> Step 5: Write 3 files to hdfs ( block id assigned are : blk_1073741831_1007,
> blk_1073741832_1008,blk_1073741833_1009 )
> Step 6: Shutdown both NN and DN
> Step 7: Start NNs with the "hdfs namenode -rollingUpgrade rollback" option.
> Start DNs with the "-rollback" option.
> Step 8: Write 2 files to hdfs.
> Issue:
> =======
> Client write failed with below exception
> {noformat}
> 2015-02-23 16:00:12,896 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Receiving BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008 src:
> /XXXXXXXXXXX:48545 dest: /XXXXXXXXXXX:50010
> 2015-02-23 16:00:12,897 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> opWriteBlock BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008
> received exception
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008 already exists in
> state FINALIZED and thus cannot be created.
> {noformat}
> Observations:
> =============
> 1. At Namenode side block invalidate is been sent only to 2 blocks.
> {noformat}
> 15/02/23 14:59:56 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
> blk_1073741833_1009 to XXXXXXXXXXX:50010
> 15/02/23 14:59:56 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
> blk_1073741831_1007 to XXXXXXXXXXX:50010
> {noformat}
> 2. fsck report does not show information on blk_1073741832_1008
> {noformat}
> FSCK started by Rex (auth:SIMPLE) from /XXXXXXXXXXX for path / at Mon Feb 23
> 16:17:57 CST 2015
> /File1: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741825_1001. Target Replicas
> is 3 but found 1 replica(s).
> /File11: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741827_1003. Target Replicas
> is 3 but found 1 replica(s).
> /File2: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741826_1002. Target Replicas
> is 3 but found 1 replica(s).
> /AfterRollback_2: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741831_1007. Target Replicas
> is 3 but found 1 replica(s).
> /Test1: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741828_1004. Target Replicas
> is 3 but found 1 replica(s).
> Status: HEALTHY
> Total size: 31620 B
> Total dirs: 7
> Total files: 6
> Total symlinks: 0
> Total blocks (validated): 5 (avg. block size 6324 B)
> Minimally replicated blocks: 5 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 5 (100.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 1.0
> Corrupt blocks: 0
> Missing replicas: 10 (66.666664 %)
> Number of data-nodes: 1
> Number of racks: 1
> FSCK ended at Mon Feb 23 16:17:57 CST 2015 in 3 milliseconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
