Shashikant Banerjee created HDFS-14499:
------------------------------------------
Summary: Misleading REM_QUOTA value with snasphot and trash
feature enabled for a directory
Key: HDFS-14499
URL: https://issues.apache.org/jira/browse/HDFS-14499
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Reporter: Shashikant Banerjee
This is the flow of steps where we see a discrepancy between REM_QUOTA and new
file operation failure. REM_QUOTA shows a value of 1 but file creation
operation does not succeed.
{code:java}
hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1
hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1
hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1
Allowing snaphot on /dir1 succeeded
hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1
Created snapshot /dir1/.snapshot/snap1
hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
PATHNAME
2 0 none inf 1 1 0 /dir1
hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1
19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved:
'hdfs://smajetinn/dir1/file1' to trash at:
hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772
hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE
PATHNAME
2 1 none inf 1 0 0 /dir1
hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
touchz: The NameSpace quota (directories and files) of directory /dir1 is
exceeded: quota=2 file count=3{code}
The issue here, is that the count command takes only files and directories into
account not the inode references. When trash is enabled, the deletion of files
inside a directory actually does a rename operation as a result of which an
inode reference is maintained in the deleted list of the snapshot diff which is
taken into account while computing the namespace quota, but count command
(getContentSummary()) ,just takes into account just the files and directories,
not the referenced entity for calculating the REM_QUOTA. The referenced entity
is taken into account for space quota only.
InodeReference.java:
-------------------
{code:java}
@Override
public final ContentSummaryComputationContext computeContentSummary(
int snapshotId, ContentSummaryComputationContext summary) {
final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId;
// only count storagespace for WithName
final QuotaCounts q = computeQuotaUsage(
summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false, s);
summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace());
summary.getCounts().addTypeSpaces(q.getTypeSpaces());
return summary;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
