[jira] [Commented] (HADOOP-8842) local file system behavior of mv into an empty directory is inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470083#comment-13470083
]
Suresh Srinivas commented on HADOOP-8842:
-
branch 20.x, since some of the applications could be dependent on the behavior
they might be getting from their implementation of the file system, we decided
not to break the compatibility by retaining the old behavior.
local file system behavior of mv into an empty directory is inconsistent with
HDFS
--
Key: HADOOP-8842
URL: https://issues.apache.org/jira/browse/HADOOP-8842
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.2
Reporter: Julien Le Dem
moving into an empty directory replaces the directory instead.
See output of attached script to reproduce :
repro.sh
{noformat}
rm -rf local_fs_bug
mkdir local_fs_bug
hdfs -rmr local_fs_bug
hdfs -mkdir local_fs_bug
echo HDFS: normal behavior
touch part-
hdfs -mkdir local_fs_bug/a
hdfs -copyFromLocal part- local_fs_bug/a
hdfs -mkdir local_fs_bug/b
hdfs -mkdir local_fs_bug/b/c
echo content of a: 1 part
hdfs -ls local_fs_bug/a
echo content of b/c: empty
hdfs -ls local_fs_bug/b/c
echo mv a b/c
hdfs -mv local_fs_bug/a local_fs_bug/b/c
echo resulting content of b/c
hdfs -ls local_fs_bug/b/c
echo a is moved inside of c
echo
echo local fs: bug
mkdir -p local_fs_bug/a
touch local_fs_bug/a/part-
mkdir -p local_fs_bug/b/c
echo content of a: 1 part
hdfs -fs local -ls local_fs_bug/a
echo content of b/c: empty
hdfs -fs local -ls local_fs_bug/b/c
echo mv a b/c
hdfs -fs local -mv local_fs_bug/a local_fs_bug/b/c
echo resulting content of b/c
hdfs -fs local -ls local_fs_bug/b/c
echo bug: a replaces c
echo
echo but it works if the destination is not empty
mkdir local_fs_bug/a2
touch local_fs_bug/a2/part-
mkdir -p local_fs_bug/b2/c2
touch local_fs_bug/b2/c2/dummy
echo content of a2: 1 part
hdfs -fs local -ls local_fs_bug/a2
echo content of b2/c2: 1 dummy file
hdfs -fs local -ls local_fs_bug/b2/c2
echo mv a2 b2/c2
hdfs -fs local -mv local_fs_bug/a2 local_fs_bug/b2/c2
echo resulting content of b/c
hdfs -fs local -ls local_fs_bug/b2/c2
echo a2 is moved inside of c2
{noformat}
Output:
{noformat}
HDFS: normal behavior
content of a: 1 part
Found 1 items
-rw-r--r-- 3 julien g 0 2012-09-25 17:16
/user/julien/local_fs_bug/a/part-
content of b/c: empty
mv a b/c
resulting content of b/c
Found 1 items
drwxr-xr-x - julien g 0 2012-09-25 17:16
/user/julien/local_fs_bug/b/c/a
a is moved inside of c
local fs: bug
content of a: 1 part
12/09/25 17:16:34 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/a/part-
content of b/c: empty
12/09/25 17:16:34 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
mv a b/c
12/09/25 17:16:35 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
resulting content of b/c
12/09/25 17:16:35 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/b/c/part-
bug: a replaces c
but it works if the destination is not empty
content of a2: 1 part
12/09/25 17:16:36 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/a2/part-
content of b2/c2: 1 dummy file
12/09/25 17:16:37 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/b2/c2/dummy
mv a2 b2/c2
12/09/25 17:16:37 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
resulting content of b/c
12/09/25 17:16:38 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 2 items
drwxr-xr-x - julien g 4096 2012-09-25 17:16
/home/julien/local_fs_bug/b2/c2/a2
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/b2/c2/dummy
a2 is moved inside of c2
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8842) local file system behavior of mv into an empty directory is inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463185#comment-13463185
]
Julien Le Dem commented on HADOOP-8842:
---
this comes from the behavior of java.io.File.renameTo(File)
to see it:
{noformat}
System.out.println(move to empty dir);
File tmp = File.createTempFile(tmp_, _dir);
tmp.delete();
tmp.mkdirs();
File a = new File(tmp, a);
File c = new File(tmp, b/c);
a.mkdirs();
new FileWriter(new File(a, part)).close();
c.mkdirs();
System.out.println(a.renameTo(c)); // true
System.out.println(Arrays.toString(c.listFiles())); // b/c/part
System.out.println(move to non empty dir);
File a2 = new File(tmp, a2);
File c2 = new File(tmp, b2/c2);
a2.mkdirs();
new FileWriter(new File(a2, part)).close();
c2.mkdirs();
new FileWriter(new File(c2, dummy)).close();
System.out.println(a2.renameTo(c2)); // false
{noformat}
Which RawLocalFileSystem is using:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup
{noformat}
249 public boolean rename(Path src, Path dst) throws IOException {
250 if (pathToFile(src).renameTo(pathToFile(dst))) {
251 return true;
252 }
253 return FileUtil.copy(this, src, this, dst, true, getConf());
254 }
{noformat}
I would suggest something like:
{noformat}
public boolean rename(Path src, Path dst) throws IOException {
File srcFile = pathToFile(src);
File dstFile = pathToFile(dst);
if (dstFile.exists() dstFile.isDirectory()) {
dstFile = new File(destFile, srcFile.getName());
}
if (srcFile.renameTo(dstFile)) {
return true;
}
return FileUtil.copy(this, src, this, new Path(dstFile.toURI()), true,
getConf());
}
{noformat}
Or possibly move this logic up in the mv command to pass the final path as dst
instead of just what was passed to the mv command
local file system behavior of mv into an empty directory is inconsistent with
HDFS
--
Key: HADOOP-8842
URL: https://issues.apache.org/jira/browse/HADOOP-8842
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.2
Reporter: Julien Le Dem
moving into an empty directory replaces the directory instead.
See output of attached script to reproduce :
repro.sh
{noformat}
rm -rf local_fs_bug
mkdir local_fs_bug
hdfs -rmr local_fs_bug
hdfs -mkdir local_fs_bug
echo HDFS: normal behavior
touch part-
hdfs -mkdir local_fs_bug/a
hdfs -copyFromLocal part- local_fs_bug/a
hdfs -mkdir local_fs_bug/b
hdfs -mkdir local_fs_bug/b/c
echo content of a: 1 part
hdfs -ls local_fs_bug/a
echo content of b/c: empty
hdfs -ls local_fs_bug/b/c
echo mv a b/c
hdfs -mv local_fs_bug/a local_fs_bug/b/c
echo resulting content of b/c
hdfs -ls local_fs_bug/b/c
echo a is moved inside of c
echo
echo local fs: bug
mkdir -p local_fs_bug/a
touch local_fs_bug/a/part-
mkdir -p local_fs_bug/b/c
echo content of a: 1 part
hdfs -fs local -ls local_fs_bug/a
echo content of b/c: empty
hdfs -fs local -ls local_fs_bug/b/c
echo mv a b/c
hdfs -fs local -mv local_fs_bug/a local_fs_bug/b/c
echo resulting content of b/c
hdfs -fs local -ls local_fs_bug/b/c
echo bug: a replaces c
echo
echo but it works if the destination is not empty
mkdir local_fs_bug/a2
touch local_fs_bug/a2/part-
mkdir -p local_fs_bug/b2/c2
touch local_fs_bug/b2/c2/dummy
echo content of a2: 1 part
hdfs -fs local -ls local_fs_bug/a2
echo content of b2/c2: 1 dummy file
hdfs -fs local -ls local_fs_bug/b2/c2
echo mv a2 b2/c2
hdfs -fs local -mv local_fs_bug/a2 local_fs_bug/b2/c2
echo resulting content of b/c
hdfs -fs local -ls local_fs_bug/b2/c2
echo a2 is moved inside of c2
{noformat}
Output:
{noformat}
HDFS: normal behavior
content of a: 1 part
Found 1 items
-rw-r--r-- 3 julien g 0 2012-09-25 17:16
/user/julien/local_fs_bug/a/part-
content of b/c: empty
mv a b/c
resulting content of b/c
Found 1 items
drwxr-xr-x - julien g 0 2012-09-25 17:16
/user/julien/local_fs_bug/b/c/a
a is moved inside of c
local fs: bug
content of a: 1 part
12/09/25 17:16:34 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/a/part-
content of b/c: empty
12/09/25 17:16:34 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
mv a b/c
12/09/25 17:16:35 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
resulting content of b/c
12/09/25 17:16:35 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g
[jira] [Commented] (HADOOP-8842) local file system behavior of mv into an empty directory is inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463341#comment-13463341
]
Julien Le Dem commented on HADOOP-8842:
---
thanks for the link.
What about the branch 20.x ?
local file system behavior of mv into an empty directory is inconsistent with
HDFS
--
Key: HADOOP-8842
URL: https://issues.apache.org/jira/browse/HADOOP-8842
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.2
Reporter: Julien Le Dem
moving into an empty directory replaces the directory instead.
See output of attached script to reproduce :
repro.sh
{noformat}
rm -rf local_fs_bug
mkdir local_fs_bug
hdfs -rmr local_fs_bug
hdfs -mkdir local_fs_bug
echo HDFS: normal behavior
touch part-
hdfs -mkdir local_fs_bug/a
hdfs -copyFromLocal part- local_fs_bug/a
hdfs -mkdir local_fs_bug/b
hdfs -mkdir local_fs_bug/b/c
echo content of a: 1 part
hdfs -ls local_fs_bug/a
echo content of b/c: empty
hdfs -ls local_fs_bug/b/c
echo mv a b/c
hdfs -mv local_fs_bug/a local_fs_bug/b/c
echo resulting content of b/c
hdfs -ls local_fs_bug/b/c
echo a is moved inside of c
echo
echo local fs: bug
mkdir -p local_fs_bug/a
touch local_fs_bug/a/part-
mkdir -p local_fs_bug/b/c
echo content of a: 1 part
hdfs -fs local -ls local_fs_bug/a
echo content of b/c: empty
hdfs -fs local -ls local_fs_bug/b/c
echo mv a b/c
hdfs -fs local -mv local_fs_bug/a local_fs_bug/b/c
echo resulting content of b/c
hdfs -fs local -ls local_fs_bug/b/c
echo bug: a replaces c
echo
echo but it works if the destination is not empty
mkdir local_fs_bug/a2
touch local_fs_bug/a2/part-
mkdir -p local_fs_bug/b2/c2
touch local_fs_bug/b2/c2/dummy
echo content of a2: 1 part
hdfs -fs local -ls local_fs_bug/a2
echo content of b2/c2: 1 dummy file
hdfs -fs local -ls local_fs_bug/b2/c2
echo mv a2 b2/c2
hdfs -fs local -mv local_fs_bug/a2 local_fs_bug/b2/c2
echo resulting content of b/c
hdfs -fs local -ls local_fs_bug/b2/c2
echo a2 is moved inside of c2
{noformat}
Output:
{noformat}
HDFS: normal behavior
content of a: 1 part
Found 1 items
-rw-r--r-- 3 julien g 0 2012-09-25 17:16
/user/julien/local_fs_bug/a/part-
content of b/c: empty
mv a b/c
resulting content of b/c
Found 1 items
drwxr-xr-x - julien g 0 2012-09-25 17:16
/user/julien/local_fs_bug/b/c/a
a is moved inside of c
local fs: bug
content of a: 1 part
12/09/25 17:16:34 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/a/part-
content of b/c: empty
12/09/25 17:16:34 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
mv a b/c
12/09/25 17:16:35 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
resulting content of b/c
12/09/25 17:16:35 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/b/c/part-
bug: a replaces c
but it works if the destination is not empty
content of a2: 1 part
12/09/25 17:16:36 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/a2/part-
content of b2/c2: 1 dummy file
12/09/25 17:16:37 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 1 items
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/b2/c2/dummy
mv a2 b2/c2
12/09/25 17:16:37 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
resulting content of b/c
12/09/25 17:16:38 WARN fs.FileSystem: local is a deprecated filesystem
name. Use file:/// instead.
Found 2 items
drwxr-xr-x - julien g 4096 2012-09-25 17:16
/home/julien/local_fs_bug/b2/c2/a2
-rw-r--r-- 1 julien g 0 2012-09-25 17:16
/home/julien/local_fs_bug/b2/c2/dummy
a2 is moved inside of c2
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
