Bram Schuur created HDFS-17984:
----------------------------------

             Summary: Block recovery deletes the block when a restarting 
DataNode reports no replica before its replicas are loaded
                 Key: HDFS-17984
                 URL: https://issues.apache.org/jira/browse/HDFS-17984
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode, hdfs
    Affects Versions: 3.5.0
         Environment: Hadoop hdfs 3.5.0
            Reporter: Bram Schuur


When a DataNode starts, {{DataNode#initBlockPool}} creates the FsDataset 
({{{}initStorage{}}}) and only afterwards loads the block pool's replicas 
({{{}FsDatasetImpl#addBlockPool{}}}), which first registers an empty replica 
map ({{{}volumeMap.initBlockPool{}}}) and then fills it 
({{{}getAllVolumesMap{}}}). The IPC server is already running at that point. An 
{{initReplicaRecovery}} call from a block recovery primary in this window 
passes {{{}checkStorageState{}}}, finds no replica in the not yet filled map 
and returns null, i.e. "this DataNode does not have the block".

{{BlockRecoveryWorker#recover}} does not count a null answer as an error. If 
the other replicas fail, the sync list is empty and {{syncBlock}} commits with 
{{{}deleteBlock=true{}}}: the block is removed and the file is closed without 
it, although the restarting DataNode holds a valid replica, which it then 
deletes as an orphan once it has loaded.

Observed on 3.5.0, writing an HBase WAL (replication 2) during a rolling node 
restart. dn0 restarted at 11:22:43; dn1 (primary) ran the recovery at 11:22:46 
and dn0 answered without an error:
{noformat}
11:22:43     [dn0] STARTUP_MSG: Starting DataNode
11:22:46,494 [dn1] java.io.IOException: Block length mismatch, len=7308711 but 
r=ReplicaBeingWritten, blk_1074458450_718705, RBW
11:22:46,494 [dn1] BlockRecoveryWorker: block=BP-...:blk_1074458450_718705 
(length=0), isTruncateRecovery=false, syncList=[]
11:22:46,495 [nn]  
commitBlockSynchronization(oldBlock=BP-...:blk_1074458450_718705, 
newgenerationstamp=718982, newlength=0, newtargets=[], closeFile=true, 
deleteBlock=true)
11:22:55,864 [dn0] Scheduling blk_1074458450_718705 replica 
ReplicaWaitingToBeRecovered, blk_1074458450_718705, RWR ... for deletion
{noformat}
A few hundred milliseconds earlier (11:22:46.19) dn0 answered another recovery 
with "Storage not yet initialized for initReplicaRecovery"; that case is 
already an error. The gap is between {{initStorage}} and the end of 
{{{}addBlockPool{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to