[jira] [Updated] (HDFS-14499) Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory
[ https://issues.apache.org/jira/browse/HDFS-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14499: --- Fix Version/s: 3.2.2 3.1.4 > Misleading REM_QUOTA value with snapshot and trash feature enabled for a > directory > -- > > Key: HDFS-14499 > URL: https://issues.apache.org/jira/browse/HDFS-14499 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-14499.000.patch, HDFS-14499.001.patch, > HDFS-14499.002.patch > > > This is the flow of steps where we see a discrepancy between REM_QUOTA and > new file operation failure. REM_QUOTA shows a value of 1 but file creation > operation does not succeed. > {code:java} > hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1 > hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1 > hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1 > Allowing snaphot on /dir1 succeeded > hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1 > hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1 > Created snapshot /dir1/.snapshot/snap1 > hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1 > QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE > PATHNAME > 2 0 none inf 1 1 0 /dir1 > hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1 > 19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://smajetinn/dir1/file1' to trash at: > hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772 > hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1 > QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE > PATHNAME > 2 1 none inf 1 0 0 /dir1 > hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1 > touchz: The NameSpace quota (directories and files) of directory /dir1 is > exceeded: quota=2 file count=3{code} > The issue here, is that the count command takes only files and directories > into account not the inode references. When trash is enabled, the deletion of > files inside a directory actually does a rename operation as a result of > which an inode reference is maintained in the deleted list of the snapshot > diff which is taken into account while computing the namespace quota, but > count command (getContentSummary()) ,just takes into account just the files > and directories, not the referenced entity for calculating the REM_QUOTA. The > referenced entity is taken into account for space quota only. > InodeReference.java: > --- > {code:java} > @Override > public final ContentSummaryComputationContext computeContentSummary( > int snapshotId, ContentSummaryComputationContext summary) { > final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId; > // only count storagespace for WithName > final QuotaCounts q = computeQuotaUsage( > summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false, > s); > summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace()); > summary.getCounts().addTypeSpaces(q.getTypeSpaces()); > return summary; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14499) Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory
[ https://issues.apache.org/jira/browse/HDFS-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-14499: --- Resolution: Fixed Fix Version/s: 3.3.0 Target Version/s: 3.3.0 Status: Resolved (was: Patch Available) Thanks [~szetszwo] for the review. I have committed this change to trunk. > Misleading REM_QUOTA value with snapshot and trash feature enabled for a > directory > -- > > Key: HDFS-14499 > URL: https://issues.apache.org/jira/browse/HDFS-14499 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14499.000.patch, HDFS-14499.001.patch, > HDFS-14499.002.patch > > > This is the flow of steps where we see a discrepancy between REM_QUOTA and > new file operation failure. REM_QUOTA shows a value of 1 but file creation > operation does not succeed. > {code:java} > hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1 > hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1 > hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1 > Allowing snaphot on /dir1 succeeded > hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1 > hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1 > Created snapshot /dir1/.snapshot/snap1 > hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1 > QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE > PATHNAME > 2 0 none inf 1 1 0 /dir1 > hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1 > 19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://smajetinn/dir1/file1' to trash at: > hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772 > hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1 > QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE > PATHNAME > 2 1 none inf 1 0 0 /dir1 > hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1 > touchz: The NameSpace quota (directories and files) of directory /dir1 is > exceeded: quota=2 file count=3{code} > The issue here, is that the count command takes only files and directories > into account not the inode references. When trash is enabled, the deletion of > files inside a directory actually does a rename operation as a result of > which an inode reference is maintained in the deleted list of the snapshot > diff which is taken into account while computing the namespace quota, but > count command (getContentSummary()) ,just takes into account just the files > and directories, not the referenced entity for calculating the REM_QUOTA. The > referenced entity is taken into account for space quota only. > InodeReference.java: > --- > {code:java} > @Override > public final ContentSummaryComputationContext computeContentSummary( > int snapshotId, ContentSummaryComputationContext summary) { > final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId; > // only count storagespace for WithName > final QuotaCounts q = computeQuotaUsage( > summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false, > s); > summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace()); > summary.getCounts().addTypeSpaces(q.getTypeSpaces()); > return summary; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14499) Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory
[ https://issues.apache.org/jira/browse/HDFS-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14499: -- Summary: Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory (was: Misleading REM_QUOTA value with snasphot and trash feature enabled for a directory) > Misleading REM_QUOTA value with snapshot and trash feature enabled for a > directory > -- > > Key: HDFS-14499 > URL: https://issues.apache.org/jira/browse/HDFS-14499 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-14499.000.patch, HDFS-14499.001.patch, > HDFS-14499.002.patch > > > This is the flow of steps where we see a discrepancy between REM_QUOTA and > new file operation failure. REM_QUOTA shows a value of 1 but file creation > operation does not succeed. > {code:java} > hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1 > hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1 > hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1 > Allowing snaphot on /dir1 succeeded > hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1 > hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1 > Created snapshot /dir1/.snapshot/snap1 > hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1 > QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE > PATHNAME > 2 0 none inf 1 1 0 /dir1 > hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1 > 19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://smajetinn/dir1/file1' to trash at: > hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772 > hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1 > QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE > PATHNAME > 2 1 none inf 1 0 0 /dir1 > hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1 > touchz: The NameSpace quota (directories and files) of directory /dir1 is > exceeded: quota=2 file count=3{code} > The issue here, is that the count command takes only files and directories > into account not the inode references. When trash is enabled, the deletion of > files inside a directory actually does a rename operation as a result of > which an inode reference is maintained in the deleted list of the snapshot > diff which is taken into account while computing the namespace quota, but > count command (getContentSummary()) ,just takes into account just the files > and directories, not the referenced entity for calculating the REM_QUOTA. The > referenced entity is taken into account for space quota only. > InodeReference.java: > --- > {code:java} > @Override > public final ContentSummaryComputationContext computeContentSummary( > int snapshotId, ContentSummaryComputationContext summary) { > final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId; > // only count storagespace for WithName > final QuotaCounts q = computeQuotaUsage( > summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false, > s); > summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace()); > summary.getCounts().addTypeSpaces(q.getTypeSpaces()); > return summary; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org