[
https://issues.apache.org/jira/browse/HADOOP-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HADOOP-4663:
-------------------------------------
Attachment: deleteTmp2.patch
When a datanode receives a replication request, it starts the block in a
separate directory. These blocks will be cleaned up when the datanode restarts.
The general algorithm is as follows:
1. When a datanode receives a block from a client, it starts the block in
datadir/tmp directory.
2. When a datanode receives a block as part of a replication request, it starts
the block in datadir//tmp_replication directory
3. When a block is finalised, it moves from any of the above two temporary
directories to its final location.
4. When a datanode restarts, it moves all the blocks from datadir/tmp directory
to their final location. These blocsk were part of writes by clients and the
client-namenode-genstamp protocol will ensure that the most-uptodate replicas
of the block will be maintained.
4. When a datanode restarts, the blocks in datadir/tmp_replication directory
are removed. No recovery is needed for these blocks.
The question that remains is what do we need to do for blocks that are moved as
part of rebalancer? I think the blocks that rebalancer moves should go into
tmp_replication as well.
> Datanode should delete files under tmp when upgraded from 0.17
> --------------------------------------------------------------
>
> Key: HADOOP-4663
> URL: https://issues.apache.org/jira/browse/HADOOP-4663
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp.patch, deleteTmp2.patch
>
>
> Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp
> directory since these files are not valid anymore. But in 0.18 it moves these
> files to normal directory incorrectly making them valid blocks. One of the
> following would work :
> - remove the tmp files during upgrade, or
> - if the files under /tmp are in pre-18 format (i.e. no generation), delete
> them.
> Currently effect of this bug is that, these files end up failing block
> verification and eventually get deleted. But cause incorrect over-replication
> at the namenode before that.
> Also it looks like our policy regd treating files under tmp needs to be
> defined better. Right now there are probably one or two more bugs with it.
> Dhruba, please file them if you rememeber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
