[
https://issues.apache.org/jira/browse/HDFS-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507783#comment-13507783
]
Daryn Sharp commented on HDFS-4248:
-----------------------------------
The code for renaming and leasing handling is problematic. There's two
parallel code paths for rename, based on whether you call it with src/dest or
src/dest/opts. They are generally the same copy-n-paste except if the target
is a pre-existing directory: the src/dest method will implicitly nested the
src under the dest, ie. rename /d1 to /d2 produces /d1/d2. The src/dest/opts
method will fail unless the OVERWRITE option is used - which will replace the
dest, ie. rename /d1 to /d2 will overwrite /d2.
The logic for the replacing or nest into the target is so low level that the
callers don't know the dest was changed. FSEditLogLoader and FSNameSystem both
had to pass a file status of the dest to change the lease so it could "assume"
if the dest was to be nested. Unfortunately this causes the src/dest/opt
method to have leases incorrectly rewritten, ie. renaming /d1 to /d2 with
overwrite produced leases of /d2/d1/d1-contents while the fs had
/d2/d1-contents.
As best I can tell, the saving of the namespace is disjointed for finalized
inodes, and underconstruction inodes. First the finalized inodes are
serialized. Then the path strings in the leases are used to locate
underconstruction inodes for serialization. This seems a bit fragile, in that
data loss will occur if the lease path goes out of sync with the namespace
paths. I believe with this design, the state of the leases are so integral to
the integrity of the namespace, that updates to the namespace and leases much
occur "atomically" under the same lock at the same time.
Also, the src/dest/opts method appeared to just "throw away" the src leases
resulting in data loss for open files.
So... What I did:
* changing lease paths only requires src & dest, no more file status of the
dest before the operation
* the namespace and the edit log loader aren't both responsible for the call to
update lease paths
* the leases are updated immediately after the move is successfully
* the src/dest/opts method is not allowed to throw away the source leases
I'm thinking another jira is needed to immediately reclaim leases on src files.
Otherwise files being written by one user can be moved by another. The updated
lease ensures image integrity, but is useless to the writer because it fails NN
operations (path-based) and has no way to know where the file went and it can't
close the file either because the original path no longer exists and leases are
looked up based on path. The client continues to hold the lease until it stops
actively writing to other streams, which for a daemon might be a really long
time.
> Renames may cause dangling leases
> ---------------------------------
>
> Key: HDFS-4248
> URL: https://issues.apache.org/jira/browse/HDFS-4248
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Blocker
> Attachments: HDFS-4248.branch-0.23.patch, HDFS-4248.patch
>
>
> Renames of directories may incorrectly re-write the paths in leases under the
> tree being moved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
