[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15518031#comment-15518031
]
Manoj Govindassamy commented on HDFS-9063:
--
Thanks for the clarification [~xiaochen]. I was mistaking
{{SnpashotDirectoryCount}} to include snapshot count also. After looking at the
code, I see the raw snapshot count is not exposed in {{ContentSummary}} or
{{Count}} command.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Labels: incompatible
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9063.000.patch, test.001.patch, test.002.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
> In the meanwhile, the snapshot diff list size is no longer included in the
> computation result. This can bring minor incompatibility but is consistent
> with the change in HDFS-7728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15517352#comment-15517352
]
Xiao Chen commented on HDFS-9063:
-
Hm, I'm a little confused by this comment:
bq. I.e., since we already have a number in the content summary to indicate the
total number of snapshots, the number of snapshots is no longer added into the
directory number.
[~jingzhao], could you clarify?
I have just chatted offline with Manoj. The snapshot in {{ContentSummary}},
added by HDFS-8986, was to track how many files/directories are created under a
snapshot, so that {{count -x}} could exclude them and calculate a result of
direct usage. It is *not* tracking how many snapshots have been taken on a
directory.
Do we have number of snapshots visible to users? I only found
{{SnapshotManager#snapshotCounter}} at the hdfs level, and
{{DirectorySnapshottableFeature#getNumSnapshots}} at the dir level - neither
seems to be propagating to any shell commands.
My 2 cents are this is will not reasonably impact quotas, so maybe good as-is.
Nice to have this displayed too, if users want to get it absolutely accurate.
Please share your thoughts, thanks!
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Labels: incompatible
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9063.000.patch, test.001.patch, test.002.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
> In the meanwhile, the snapshot diff list size is no longer included in the
> computation result. This can bring minor incompatibility but is consistent
> with the change in HDFS-7728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515048#comment-15515048 ] Manoj Govindassamy commented on HDFS-9063: -- [~jingzhao], Did you get a chance to look at the attached test case ? Is there anything wrong with the expectation in the test ? I can file a jira to track this issue, if you also feel snapDir count is not working as expected. Thanks. > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Labels: incompatible > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-9063.000.patch, test.001.patch, test.002.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. > In the meanwhile, the snapshot diff list size is no longer included in the > computation result. This can bring minor incompatibility but is consistent > with the change in HDFS-7728. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511227#comment-15511227 ] Jing Zhao commented on HDFS-9063: - Agree. Added the incompatible label and also updated the release note. Thanks for looking into this, [~manojg]. > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Labels: incompatible > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-9063.000.patch, test.001.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. > In the meanwhile, the snapshot diff list size is no longer included in the > computation result. This can bring minor incompatibility but is consistent > with the change in HDFS-7728. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511199#comment-15511199
]
Manoj Govindassamy commented on HDFS-9063:
--
Thanks for looking into this. Thats right, {{ContentSummary}} does have good
info about Snapshots and I do feel its better to keep dirCount and snapCount
separate. However, the behavior of the {{Count}} command has changed and it
would be good to mark this jira as incompatible.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9063.000.patch, test.001.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511051#comment-15511051
]
Jing Zhao commented on HDFS-9063:
-
Ah, I see what you mean. Yes, we have a minor incompatibility here, please see
my first comment:
{quote}
In the meanwhile, the snapshot diff list size is no longer included in the
computation result. This can bring minor incompatibility but is consistent with
the change in HDFS-7728.
{quote}
I.e., since we already have a number in the content summary to indicate the
total number of snapshots, the number of snapshots is no longer added into the
directory number. To me the old behavior is more like a bug, but we can still
mark this jira as incompatible maybe.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9063.000.patch, test.001.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510852#comment-15510852 ] Manoj Govindassamy commented on HDFS-9063: -- Attached test fails on the latest trunk 3.0.0.alpha2. I didn't see the test passing after reverting HDFS-8986 fix. But test passes after reverting HDFS-9063 fix. Let me know if my understanding is wrong. > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-9063.000.patch, test.001.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510683#comment-15510683
]
Jing Zhao commented on HDFS-9063:
-
[~manojg], the patch fixed a bug and did not change the semantic of {{du}}
command. Before the patch, files that were created outside of a snapshot could
be included when calculating the count/usage of the snapshot. The patch fixed
this bug and there is such an example int the description.
For your case, maybe HDFS-8986 is the one related? Which Hadoop version are you
using?
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510568#comment-15510568
]
Manoj Govindassamy commented on HDFS-9063:
--
Hi [~jingzhao],
I am seeing a difference in behavior on the {{Count}} command before and after
this fix. {{Count}} command used to account for {{.snapshot}} directory, but
after this fix, _numDirs_ from Count command is less than expected as it is not
including the {{.snapshot}} directory. Is the change in behavior intended ?
Please clarify.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876372#comment-14876372 ] Hudson commented on HDFS-9063: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #392 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/392/]) HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0 > > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876242#comment-14876242 ] Hudson commented on HDFS-9063: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2330 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2330/]) HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0 > > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876208#comment-14876208 ] Hudson commented on HDFS-9063: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1151 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1151/]) HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0 > > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876195#comment-14876195 ] Hudson commented on HDFS-9063: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2357 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2357/]) HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0 > > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876017#comment-14876017 ] Hudson commented on HDFS-9063: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #410 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/410/]) HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0 > > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875937#comment-14875937 ] Hudson commented on HDFS-9063: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #417 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/417/]) HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0 > > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875913#comment-14875913 ] Hudson commented on HDFS-9063: -- FAILURE: Integrated in Hadoop-trunk-Commit #8481 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8481/]) HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.8.0 > > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875883#comment-14875883 ] Jing Zhao commented on HDFS-9063: - Thanks for the review, Yi! I will commit the patch shortly (and also fix the 2 checkstyle warnings). > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14860285#comment-14860285 ] Yi Liu commented on HDFS-9063: -- BTW, you can cleanup the checkstyle while committing if no other people have comments. Thanks. > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804930#comment-14804930
]
Yi Liu commented on HDFS-9063:
--
+1, thanks Jing.
I check the patch, now if {{getContentSummary}} on snapshot path, only content
in snapshot is counted.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804918#comment-14804918
]
Yi Liu commented on HDFS-9063:
--
Thanks Jing for the explanation, I think I missed the fact that
{{getContentSummary}} also counts the files in snapshot.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804482#comment-14804482
]
Hadoop QA commented on HDFS-9063:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 15s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 24s | The applied patch generated 2
new checkstyle issues (total was 177, now 179). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 2s | Tests failed in hadoop-hdfs. |
| | | 209m 56s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.TestNameNodeResourceChecker |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12756374/HDFS-9063.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 58d1a02 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12511/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12511/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12511/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12511/console |
This message was automatically generated.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[ https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803250#comment-14803250 ] Jing Zhao commented on HDFS-9063: - The test failures look suspicious. I just triggered the Jenkins again. > Correctly handle snapshot path for getContentSummary > > > Key: HDFS-9063 > URL: https://issues.apache.org/jira/browse/HDFS-9063 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-9063.000.patch > > > The current getContentSummary implementation does not take into account the > snapshot path, thus if we have the following ops: > 1. create dirs /foo/bar > 2. take snapshot s1 on /foo > 3. create a 1 byte file /foo/bar/baz > then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", > which is incorrect since the 1 byte file is not included in snapshot s1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803241#comment-14803241
]
Jing Zhao commented on HDFS-9063:
-
Thanks Yi! {{getContentSummary(dir's current path)}} should include (all the
current files/directories) + (all the deleted files/directories but still in
snapshots). Thus in the above case, the return value 16 in step 6 is correct:
we have 15 files in the current dir, and the original first file in
dir/.snapshot/s1.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791678#comment-14791678
]
Yi Liu commented on HDFS-9063:
--
Thanks [~jingzhao] for working on this. Jenkins' report has issue,
{{TestGetContentSummaryWithSnapshot}} can pass locally.
I also found similar issue (not the same, but also problem of getContentSummary
if there is snapshot) when writing tests for large directory in HDFS-9053, it
also exists in current trunk. I apply your patch, the issue I saw is still
there.
I think you can fix the issue too and write test of following steps to
reproduce the issue I found (Of course if you don't want to fix it, I can do it
separately :)):
# Suppose we have a directory named 'dir', create 16 files in the dir
# remove the last 1 file -- now total 15 files in dir
# create a snapshot 's1' of dir
# add 1 file in dir -- now total 16 files in dir
# remove the first 1 file in dir -- now total 15 files in the dir
# call getContentSummary(dir), and then {{getFileCount}}. -- the expected
result is 15, but the return is 16.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9063) Correctly handle snapshot path for getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791489#comment-14791489
]
Hadoop QA commented on HDFS-9063:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 26s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 53s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 30s | The applied patch generated 2
new checkstyle issues (total was 177, now 179). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 43s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 30s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 74m 57s | Tests failed in hadoop-hdfs. |
| | | 124m 27s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| Timed out tests | org.apache.hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12756374/HDFS-9063.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 941001b |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12496/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12496/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12496/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12496/console |
This message was automatically generated.
> Correctly handle snapshot path for getContentSummary
>
>
> Key: HDFS-9063
> URL: https://issues.apache.org/jira/browse/HDFS-9063
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9063.000.patch
>
>
> The current getContentSummary implementation does not take into account the
> snapshot path, thus if we have the following ops:
> 1. create dirs /foo/bar
> 2. take snapshot s1 on /foo
> 3. create a 1 byte file /foo/bar/baz
> then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar",
> which is incorrect since the 1 byte file is not included in snapshot s1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
