[
https://issues.apache.org/jira/browse/HDFS-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099884#comment-14099884
]
Yongjun Zhang commented on HDFS-6833:
-------------------------------------
Hi [~sinchii], thanks for addressing my earlier comments.
A second thought. WIth the patch you worked out, the block is added to
deletingBlock right after it is removed from volumeMap, why we don't remove the
block from the deletingBlock right after FsDatasetAsyncDiskService remove it
from disk?. Another thing is, more items are kept being added to the
deletingBlockId, but never cleaned up.
If we remove the block from the deletingBlock whenever
FsDatasetAsyncDiskService removes a block from disk, then there is no need for
deletingBlockId data structure, right?
Thanks.
> DirectoryScanner should not register a deleting block with memory of DataNode
> -----------------------------------------------------------------------------
>
> Key: HDFS-6833
> URL: https://issues.apache.org/jira/browse/HDFS-6833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.0.0
> Reporter: Shinichi Yamashita
> Assignee: Shinichi Yamashita
> Attachments: HDFS-6833-6.patch, HDFS-6833.patch, HDFS-6833.patch,
> HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch
>
>
> When a block is deleted in DataNode, the following messages are usually
> output.
> {code}
> 2014-08-07 17:53:11,606 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:11,617 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> However, DirectoryScanner may be executed when DataNode deletes the block in
> the current implementation. And the following messsages are output.
> {code}
> 2014-08-07 17:53:30,519 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:31,426 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata
> files:0, missing block files:0, missing blocks in memory:1, mismatched
> blocks:0
> 2014-08-07 17:53:31,426 WARN
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added
> missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
> getNumBytes() = 21230663
> getBytesOnDisk() = 21230663
> getVisibleLength()= 21230663
> getVolume() = /hadoop/data1/dfs/data/current
> getBlockFile() =
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> unlinked =false
> 2014-08-07 17:53:31,531 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> Deleting block information is registered in DataNode's memory.
> And when DataNode sends a block report, NameNode receives wrong block
> information.
> For example, when we execute recommission or change the number of
> replication, NameNode may delete the right block as "ExcessReplicate" by this
> problem.
> And "Under-Replicated Blocks" and "Missing Blocks" occur.
> When DataNode run DirectoryScanner, DataNode should not register a deleting
> block.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
