[
https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753866#action_12753866
]
dhruba borthakur commented on HDFS-611:
---------------------------------------
> The datanode knows the number of volumes and could use that instead of a
> config option
This makes sense to me.
> But do you really think a single thread would get so far behind that the
> queue of blocks to delete would exhaust the datanode's heap
I will test how fact a file deletion on ext3 is. will report back with some
numbers.
Another thing that I am keeping in mind is that if a thread asyncronously
processes block file deletions while freeing the heartbeat thread to continue
hearbeating to the namenode might introduce new race conditions. Let's assume
that a blockfile deletion request is queud up. Then the datanode decides to
send a block report and will report that block to the NN. Then the async trhead
will start prcessing items from the queue and delete the block file. This could
be bad! The Nameode will think that the datanode has this block whereas
actually it is now gone from the datanode.
> Heartbeats times from Datanodes increase when there are plenty of blocks to
> delete
> ----------------------------------------------------------------------------------
>
> Key: HDFS-611
> URL: https://issues.apache.org/jira/browse/HDFS-611
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> I am seeing that when we delete a large directory that has plenty of blocks,
> the heartbeat times from datanodes increase significantly from the normal
> value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in
> the Datanode deletes a bunch of blocks sequentially, this causes the
> heartbeat times to increase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
