[
https://issues.apache.org/jira/browse/HDFS-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354572#comment-14354572
]
Vinayakumar B commented on HDFS-7820:
-------------------------------------
bq. Rather than add a hard-coded increment we can reserve the msb of the
blockid for an epoch. The epoch is toggled on every rollback. The low order 63
bits are incremented sequentially as before on block allocation. Now we will
not hit any collisions with block ids allocated before the rollback.
This Idea is good, but the problem I am seeing is same as you mentioned.
Multiple Rollbacks.
During the testing of a upgrade, there could be possibility of multiple
rollbacks. With each rollback if toggle the msb, then block id range will be
reduced by half since the actual no bits involved is reduced. And in any case
we cannot re-toggle the bit.
What if blocks are scheduled for deletion on the first report itself only in
case of RollBack?
> Client Write fails after rolling upgrade rollback with "<block_id> already
> exist in finalized state"
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-7820
> URL: https://issues.apache.org/jira/browse/HDFS-7820
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: J.Andreina
> Assignee: J.Andreina
> Attachments: HDFS-7820.1.patch
>
>
> Steps to Reproduce:
> ===================
> Step 1: Prepare rolling upgrade using "hdfs dfsadmin -rollingUpgrade prepare"
> Step 2: Shutdown SNN and NN
> Step 3: Start NN with the "hdfs namenode -rollingUpgrade started" option.
> Step 4: Executed "hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT>
> upgrade" and restarted Datanode
> Step 5: Write 3 files to hdfs ( block id assigned are : blk_1073741831_1007,
> blk_1073741832_1008,blk_1073741833_1009 )
> Step 6: Shutdown both NN and DN
> Step 7: Start NNs with the "hdfs namenode -rollingUpgrade rollback" option.
> Start DNs with the "-rollback" option.
> Step 8: Write 2 files to hdfs.
> Issue:
> =======
> Client write failed with below exception
> {noformat}
> 2015-02-23 16:00:12,896 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Receiving BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008 src:
> /XXXXXXXXXXX:48545 dest: /XXXXXXXXXXX:50010
> 2015-02-23 16:00:12,897 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> opWriteBlock BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008
> received exception
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008 already exists in
> state FINALIZED and thus cannot be created.
> {noformat}
> Observations:
> =============
> 1. At Namenode side block invalidate is been sent only to 2 blocks.
> {noformat}
> 15/02/23 14:59:56 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
> blk_1073741833_1009 to XXXXXXXXXXX:50010
> 15/02/23 14:59:56 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
> blk_1073741831_1007 to XXXXXXXXXXX:50010
> {noformat}
> 2. fsck report does not show information on blk_1073741832_1008
> {noformat}
> FSCK started by Rex (auth:SIMPLE) from /XXXXXXXXXXX for path / at Mon Feb 23
> 16:17:57 CST 2015
> /File1: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741825_1001. Target Replicas
> is 3 but found 1 replica(s).
> /File11: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741827_1003. Target Replicas
> is 3 but found 1 replica(s).
> /File2: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741826_1002. Target Replicas
> is 3 but found 1 replica(s).
> /AfterRollback_2: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741831_1007. Target Replicas
> is 3 but found 1 replica(s).
> /Test1: Under replicated
> BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741828_1004. Target Replicas
> is 3 but found 1 replica(s).
> Status: HEALTHY
> Total size: 31620 B
> Total dirs: 7
> Total files: 6
> Total symlinks: 0
> Total blocks (validated): 5 (avg. block size 6324 B)
> Minimally replicated blocks: 5 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 5 (100.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 1.0
> Corrupt blocks: 0
> Missing replicas: 10 (66.666664 %)
> Number of data-nodes: 1
> Number of racks: 1
> FSCK ended at Mon Feb 23 16:17:57 CST 2015 in 3 milliseconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
