Lei (Eddy) Xu created HDFS-7996:
-----------------------------------
Summary: After swapping a volume, BlockReceiver reports
ReplicaNotFoundException
Key: HDFS-7996
URL: https://issues.apache.org/jira/browse/HDFS-7996
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Critical
When removing a disk from an actively writing DataNode, the BlockReceiver
working on the disk throws {{ReplicaNotFoundException}} because the replicas
are removed from the memory:
{code}
2015-03-26 08:02:43,154 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removed
volume: /data/2/dfs/dn/current
2015-03-26 08:02:43,163 INFO org.apache.hadoop.hdfs.server.common.Storage:
Removing block level storage:
/data/2/dfs/dn/current/BP-51301509-10.20.202.114-1427296597742
2015-03-26 08:02:43,163 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
IOException in BlockReceiver.run():
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Cannot append
to a non-existent replica
BP-51301509-10.20.202.114-1427296597742:blk_1073742979_2160
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getReplicaInfo(FsDatasetImpl.java:615)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:1362)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.finalizeBlock(BlockReceiver.java:1281)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1241)
at java.lang.Thread.run(Thread.java:745)
{code}
{{FsVolumeList#removeVolume}} waits all threads release {{FsVolumeReference}}
on the volume to be removed, however, in {{PacketResponder#finalizeBlock()}},
it calls
{code}
private void finalizeBlock(long startTime) throws IOException {
BlockReceiver.this.close();
final long endTime = ClientTraceLog.isInfoEnabled() ? System.nanoTime()
: 0;
block.setNumBytes(replicaInfo.getNumBytes());
datanode.data.finalizeBlock(block);
{code}
The {{FsVolumeReference}} was released in {{BlockReceiver.this.close()}} before
calling {{datanode.data.finalizeBlock(block)}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
