Bram Schuur created HDFS-17983:
----------------------------------
Summary: Lease recovery rejects an RBW replica with "Block length
mismatch" after an interrupted packet write
Key: HDFS-17983
URL: https://issues.apache.org/jira/browse/HDFS-17983
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, hdfs
Affects Versions: 3.5.0
Environment: Hadoop hdfs version 3.5.0
Reporter: Bram Schuur
When a packet write on a DataNode fails after the data has reached the block
file but before bytesOnDisk is updated, the RBW replica is left with a block
file longer than its in-memory bytesOnDisk. One way this happens:
*BlockReceiver#receivePacket* writes the data and checksum, the PacketResponder
interrupts the receiver because the client went away, and the following fsync
throws *ClosedByInterruptException*. The extra bytes were never acknowledged.
Pipeline recovery has handled this case since HDFS-11472:
*FsDatasetImpl#recoverRbwImpl* resets bytesOnDisk and truncates to the
acknowledged length. Lease/block recovery does not.
*FsDatasetImpl#initReplicaRecoveryImpl* calls *checkReplicaFiles*, which
requires *bytesOnDisk == blockDataLength* and throws otherwise. The replica is
excluded from recovery even though it holds all acknowledged data:
- if it is the only replica, recovery keeps failing ("All datanodes failed")
until the DataNode restarts and reloads the replica as RWR with a
checksum-validated length;
- if the other replicas cannot be used either, the block can be lost.
Observed on 3.5.0, writing an HBase WAL (hsync, replication 2) during a rolling
node restart:
{code}
11:21:40,998 INFO datanode.DataNode: opWriteBlock BP-...:blk_1074458450_718705
received exception java.nio.channels.ClosedByInterruptException
11:22:46,494 INFO impl.FsDatasetImpl: initReplicaRecovery:
blk_1074458450_718705, recoveryId=718982, replica=ReplicaBeingWritten,
blk_1074458450_718705, RBW
11:22:46,494 WARN protocol.InterDatanodeProtocol: Failed to recover block
(block=BP-...:blk_1074458450_718705,
datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
java.io.IOException: Block length mismatch, len=7308711 but
r=ReplicaBeingWritten, blk_1074458450_718705, RBW
11:22:46,494 INFO datanode.DataNode: BlockRecoveryWorker:
block=BP-...:blk_1074458450_718705 (length=0), isTruncateRecovery=false,
syncList=[]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]