[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-14504:
--
Component/s: snapshots
Hadoop Flags: Reviewed
Target Version/s: 3.3.1, 3.4.0
Affects Version/s: 3.3.1
3.4.0
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Shashikant Banerjee
>Assignee: Hemanth Boyina
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-14504.001.patch, HDFS-14504.002.patch
>
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
> 2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap1
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
> 2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
> 2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=4
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
> 2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap2
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
> 2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
> 2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=5
> {code}
> // create operation fails here as it has already exceeded the quota limit
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
> 2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap3
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
> 2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
> // Rename operation succeeds here adding on to the namespace quota
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
> 2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Boyina updated HDFS-14504:
--
Fix Version/s: 3.4.0
3.3.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Hemanth Boyina
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-14504.001.patch, HDFS-14504.002.patch
>
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
> 2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap1
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
> 2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
> 2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=4
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
> 2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap2
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
> 2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
> 2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=5
> {code}
> // create operation fails here as it has already exceeded the quota limit
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
> 2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap3
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
> 2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
> // Rename operation succeeds here adding on to the namespace quota
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
> 2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory
[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina updated HDFS-14504:
-
Attachment: HDFS-14504.002.patch
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14504.001.patch, HDFS-14504.002.patch
>
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
> 2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap1
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
> 2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
> 2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=4
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
> 2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap2
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
> 2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
> 2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=5
> {code}
> // create operation fails here as it has already exceeded the quota limit
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
> 2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap3
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
> 2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
> // Rename operation succeeds here adding on to the namespace quota
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
> 2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=6
> {code}
> // File creation fails here but file count has been increased to 6, bcoz of
>
[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina updated HDFS-14504:
-
Attachment: HDFS-14504.001.patch
Status: Patch Available (was: Open)
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14504.001.patch
>
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
> 2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap1
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
> 2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
> 2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=4
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
> 2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap2
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
> 2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
> 2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=5
> {code}
> // create operation fails here as it has already exceeded the quota limit
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
> 2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap3
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
> 2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
> // Rename operation succeeds here adding on to the namespace quota
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
> 2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=6
> {code}
> // File creation fails here but file count has been
[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina updated HDFS-14504:
-
Attachment: HDFS-14504.001.patch
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Priority: Major
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
> 2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap1
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
> 2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
> 2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=4
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
> 2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap2
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
> 2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
> 2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=5
> {code}
> // create operation fails here as it has already exceeded the quota limit
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
> 2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap3
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
> 2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
> // Rename operation succeeds here adding on to the namespace quota
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
> 2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=6
> {code}
> // File creation fails here but file count has been increased to 6, bcoz of
> the previous rename operation{code}
> The quota being set here is 3. Each successive rename adds an entry
[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hemanthboyina updated HDFS-14504:
-
Attachment: (was: HDFS-14504.001.patch)
> Rename with Snapshots does not honor quota limit
>
>
> Key: HDFS-14504
> URL: https://issues.apache.org/jira/browse/HDFS-14504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: hemanthboyina
>Priority: Major
>
> Steps to Reproduce:
>
> {code:java}
> HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
> 2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
> 2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
> 2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Allowing snapshot on /dir2 succeeded
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
> 2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
> 2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap1
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
> 2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
> 2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=4
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
> 2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap2
> HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
> 2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Found 1 items
> -rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
> 2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
> 2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=5
> {code}
> // create operation fails here as it has already exceeded the quota limit
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
> 2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Created snapshot /dir2/.snapshot/snap3
> HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
> 2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
> // Rename operation succeeds here adding on to the namespace quota
> {code}
> HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
> 2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> touchz: The NameSpace quota (directories and files) of directory /dir2 is
> exceeded: quota=3 file count=6
> {code}
> // File creation fails here but file count has been increased to 6, bcoz of
> the previous rename operation{code}
> The quota being set
[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-14504:
---
Description:
Steps to Reproduce:
{code:java}
HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Allowing snapshot on /dir2 succeeded
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Created snapshot /dir2/.snapshot/snap1
HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Found 1 items
-rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
touchz: The NameSpace quota (directories and files) of directory /dir2 is
exceeded: quota=3 file count=4
HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Created snapshot /dir2/.snapshot/snap2
HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Found 1 items
-rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
touchz: The NameSpace quota (directories and files) of directory /dir2 is
exceeded: quota=3 file count=5
{code}
// create operation fails here as it has already exceeded the quota limit
{code}
HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Created snapshot /dir2/.snapshot/snap3
HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
{code}
// Rename operation succeeds here adding on to the namespace quota
{code}
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
touchz: The NameSpace quota (directories and files) of directory /dir2 is
exceeded: quota=3 file count=6
{code}
// File creation fails here but file count has been increased to 6, bcoz of the
previous rename operation{code}
The quota being set here is 3. Each successive rename adds an entry to the
deleted list of the snapshot diff which gets accounted in the namespace quota,
but the rename operation is allowed even when it exceeds the quota limit with
snapshots. Once, an attempt is made to create a file, it fails.
was:
Steps to Reproduce:
{code:java}
HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
[jira] [Updated] (HDFS-14504) Rename with Snapshots does not honor quota limit
[
https://issues.apache.org/jira/browse/HDFS-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-14504:
---
Description:
Steps to Reproduce:
{code:java}
HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Allowing snapshot on /dir2 succeeded
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Created snapshot /dir2/.snapshot/snap1
HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Found 1 items
-rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
touchz: The NameSpace quota (directories and files) of directory /dir2 is
exceeded: quota=3 file count=4
HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Created snapshot /dir2/.snapshot/snap2
HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Found 1 items
-rw-r--r-- 1 sbanerjee hadoop 0 2019-05-21 15:10 /dir2/file2
HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
touchz: The NameSpace quota (directories and files) of directory /dir2 is
exceeded: quota=3 file count=5
// create operation fails here as it has already exceeded the quota limit
HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Created snapshot /dir2/.snapshot/snap3
HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
// Rename operation succeeds here adding on to the namespace quota
HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
touchz: The NameSpace quota (directories and files) of directory /dir2 is
exceeded: quota=3 file count=6
// File creation fails here but file count has been increased to 6, bcoz of the
previous rename operation{code}
The quota being set here is 3. Each successive rename adds an entry to the
deleted list of the snapshot diff which gets accounted in the namespace quota,
but the rename operation is allowed even when it exceeds the quota limit with
snapshots. Once, an attempt is made to create a file, it fails.
was:
Steps to Reproduce:
{code:java}
HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
HW15685:bin
