[jira] [Comment Edited] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087899#comment-17087899
]
Shashikant Banerjee edited comment on HDFS-14504 at 4/20/20, 4:25 PM:
--
Thanks [~hemanthboyina] for updating the patch.
{code:java}
@Test
public void testRenameAcrossDirWithinSnapshot() throws Exception {
// snapshottable directory
String dirr = "/dir";
Path rootDir = new Path(dirr);
hdfs.mkdirs(rootDir);
hdfs.allowSnapshot(rootDir);
// set quota for source directory under snapshottable root directory
Path dir2 = new Path(rootDir, "dir2");
Path fil1 = new Path(dir2, "file1");
hdfs.mkdirs(dir2);
hdfs.setQuota(dir2, 3, 0);
hdfs.create(fil1);
Path file2 = new Path(dir2, "file2");
hdfs.rename(fil1, file2);
Path fil3 = new Path(dir2, "file3");
hdfs.create(fil3);
// destination directory under snapshottable root directory
Path dir1 = new Path(rootDir, "dir1");
Path dir1fil1 = new Path(dir1, "file1");
hdfs.mkdirs(dir1);
hdfs.create(dir1fil1);
Path dir1fil2 = new Path(dir1, "file2");
hdfs.rename(dir1fil1, dir1fil2);
hdfs.createSnapshot(rootDir, "snap1");
Path filex = new Path(dir2, "filex");
// create a file after exceeding namespace quota
LambdaTestUtils.intercept(NSQuotaExceededException.class,
"The NameSpace quota (directories and files) of "
+ "directory /dir/dir2 is exceeded",
() -> hdfs.create(filex));
// Rename across directories within snapshot with quota set on source
// directory
assertTrue(hdfs.rename(fil3, dir1));
}
{code}
In the test above, if "filex" cound not created in "dir2" because of exceeding
quota limit but rename of "fil3" existing under the same directory "dir2" seems
successful which ideally should fail as it will create InodeReferene in dir2
diff list for the snapshot snap1 hence will exceed the quota limit.
Can you plz check?
was (Author: shashikant):
Thanks [~hemanthboyina] for updating the patch.
{code:java}
@Test
public void testRenameAcrossDirWithinSnapshot() throws Exception {
// snapshottable directory
String dirr = "/dir";
Path rootDir = new Path(dirr);
hdfs.mkdirs(rootDir);
hdfs.allowSnapshot(rootDir);
// set quota for source directory under snapshottable root directory
Path dir2 = new Path(rootDir, "dir2");
Path fil1 = new Path(dir2, "file1");
hdfs.mkdirs(dir2);
hdfs.setQuota(dir2, 3, 0);
hdfs.create(fil1);
Path file2 = new Path(dir2, "file2");
hdfs.rename(fil1, file2);
Path fil3 = new Path(dir2, "file3");
hdfs.create(fil3);
// destination directory under snapshottable root directory
Path dir1 = new Path(rootDir, "dir1");
Path dir1fil1 = new Path(dir1, "file1");
hdfs.mkdirs(dir1);
hdfs.create(dir1fil1);
Path dir1fil2 = new Path(dir1, "file2");
hdfs.rename(dir1fil1, dir1fil2);
hdfs.createSnapshot(rootDir, "snap1");
Path filex = new Path(dir2, "filex");
// create a file after exceeding namespace quota
LambdaTestUtils.intercept(NSQuotaExceededException.class,
"The NameSpace quota (directories and files) of "
+ "directory /dir/dir2 is exceeded",
() -> hdfs.create(filex));
// Rename across directories within snapshot with quota set on source
// directory
assertTrue(hdfs.rename(fil3, dir1));
}
{code}
In the test above, if "filex" cound not created in "dir2" because of exceeding
quota limit but rename of "fil3" existing under the same directory "dir2" seems
successful which ideally should fail as it will create InodeReferene in dir2
diff list for the snapshot snap1 hence exceeding the quota limit.
Can you plz check?
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14504.001.patch, HDFS-14504.002.patch
>
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21
[jira] [Comment Edited] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087899#comment-17087899
]
Shashikant Banerjee edited comment on HDFS-14504 at 4/20/20, 4:17 PM:
--
Thanks [~hemanthboyina] for updating the patch.
{code:java}
@Test
public void testRenameAcrossDirWithinSnapshot() throws Exception {
// snapshottable directory
String dirr = "/dir";
Path rootDir = new Path(dirr);
hdfs.mkdirs(rootDir);
hdfs.allowSnapshot(rootDir);
// set quota for source directory under snapshottable root directory
Path dir2 = new Path(rootDir, "dir2");
Path fil1 = new Path(dir2, "file1");
hdfs.mkdirs(dir2);
hdfs.setQuota(dir2, 3, 0);
hdfs.create(fil1);
Path file2 = new Path(dir2, "file2");
hdfs.rename(fil1, file2);
Path fil3 = new Path(dir2, "file3");
hdfs.create(fil3);
// destination directory under snapshottable root directory
Path dir1 = new Path(rootDir, "dir1");
Path dir1fil1 = new Path(dir1, "file1");
hdfs.mkdirs(dir1);
hdfs.create(dir1fil1);
Path dir1fil2 = new Path(dir1, "file2");
hdfs.rename(dir1fil1, dir1fil2);
hdfs.createSnapshot(rootDir, "snap1");
Path filex = new Path(dir2, "filex");
// create a file after exceeding namespace quota
LambdaTestUtils.intercept(NSQuotaExceededException.class,
"The NameSpace quota (directories and files) of "
+ "directory /dir/dir2 is exceeded",
() -> hdfs.create(filex));
// Rename across directories within snapshot with quota set on source
// directory
assertTrue(hdfs.rename(fil3, dir1));
}
{code}
In the test above, if "filex" cound not created in "dir2" because of exceeding
quota limit but rename of "fil3" existing under the same directory "dir2" seems
successful which ideally should fail as it will create InodeReferene in dir2
diff list for the snapshot snap1 hence exceeding the quota limit.
Can you plz check?
was (Author: shashikant):
Thanks [~hemanthboyina] for updating the patch.
{code:java}
@Test
public void testRenameAcrossDirWithinSnapshot() throws Exception {
// snapshottable directory
String dirr = "/dir";
Path rootDir = new Path(dirr);
hdfs.mkdirs(rootDir);
hdfs.allowSnapshot(rootDir);
// set quota for source directory under snapshottable root directory
Path dir2 = new Path(rootDir, "dir2");
Path fil1 = new Path(dir2, "file1");
hdfs.mkdirs(dir2);
hdfs.setQuota(dir2, 3, 0);
hdfs.create(fil1);
Path file2 = new Path(dir2, "file2");
hdfs.rename(fil1, file2);
Path fil3 = new Path(dir2, "file3");
hdfs.create(fil3);
// destination directory under snapshottable root directory
Path dir1 = new Path(rootDir, "dir1");
Path dir1fil1 = new Path(dir1, "file1");
hdfs.mkdirs(dir1);
hdfs.create(dir1fil1);
Path dir1fil2 = new Path(dir1, "file2");
hdfs.rename(dir1fil1, dir1fil2);
hdfs.createSnapshot(rootDir, "snap1");
Path filex = new Path(dir2, "filex");
// create a file after exceeding namespace quota
LambdaTestUtils.intercept(NSQuotaExceededException.class,
"The NameSpace quota (directories and files) of "
+ "directory /dir/dir2 is exceeded",
() -> hdfs.create(filex));
// Rename across directories within snapshot with quota set on source
// directory
assertTrue(hdfs.rename(fil3, dir1));
}
{code}
In the test above, if "filex" cound not created in "dir2" because of exceeding
quota limit but rename of "fil3" existing under the same directory "dir2" seems
successful which ideally should fail as it will crete InodeReferene in dir2
diff list for the snapshot snap1 hence exceeding the quota limit.
Can you plz check?
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14504.001.patch, HDFS-14504.002.patch
>
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21
[jira] [Comment Edited] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077306#comment-17077306
]
Shashikant Banerjee edited comment on HDFS-14504 at 4/8/20, 5:52 PM:
-
Thanks [~hemanthboyina] for working on this. There are multiple cases to be
considered while doing the rename across directories in a snapshot like:
1) Rename across directories within snapshot with quota set on source directory
2) Rename within the same directory within a snapshottable root with quota set
3) Rename from a directory under snapshottable root to a directory with quota
set to a directory not under under any snapshottable root.
The fix might not address all the cases here.
Also, disallowing the rename in case of a rename happening under a snapshotted
directory with quota set might become incompatible change as well.
was (Author: shashikant):
Thanks [~hemanthboyina] for working on this. There are multiple cases to be
considered while doing the rename across directories in a snapshot like:
1) Rename across directories within snapshot with quota set on source directory
2) Rename within the same directory within a snapshottable root with quota set
3) Rename from a directory under snapshottable root to a directory with quota
set to a directory not under under any snapshottable root.
The fix might not address all the cases here.
Also, disallowing the rename in case of a rename happening under a snapshotted
directory with quota set will become incompatible change as well.
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14504.001.patch
>
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
> 2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap1
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
> 2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
> 2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=4
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
> 2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap2
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
> 2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
>
