[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128781#comment-15128781
]
Hudson commented on HDFS-9721:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9225 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9225/])
HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains (lei:
rev 9d494f0c0eaa05417f3a3e88487d878d1731da36)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/IgnoreSnapshotException.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageDelimitedTextWriter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch, HDFS-9721.04.patch, HDFS-9721.05.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128797#comment-15128797
]
Xiao Chen commented on HDFS-9721:
-
Thanks for the helpful reviews and commit [~eddyxu].
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch, HDFS-9721.04.patch, HDFS-9721.05.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128675#comment-15128675
]
Xiao Chen commented on HDFS-9721:
-
Failed tests seem unrelated.
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch, HDFS-9721.04.patch, HDFS-9721.05.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126858#comment-15126858
]
Lei (Eddy) Xu commented on HDFS-9721:
-
Hi, [~xiaochen] Thanks a lot for addressing the above comments.
I feel that {{private String ignoreSnapshotName(long inode) throws
IOException}} can be a {{static}} method of {{MetadataMap}} and returns
{{void}}, what do you think?
Btw, it seems that we do not need to use pre-increment in the code, e.g.,
{{++ignored}} and {{++dirCount}}. Would you mind to change it to comply the
coding style used in the rest of this file?
Would +1 after fixing these.
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch, HDFS-9721.04.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127121#comment-15127121
]
Xiao Chen commented on HDFS-9721:
-
Hm, BTW, should we mark this as incompatible change, due to the added {{d}} /
{{-}} in the permissions string?
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch, HDFS-9721.04.patch, HDFS-9721.05.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127502#comment-15127502
]
Hadoop QA commented on HDFS-9721:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 23s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 19s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 30s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 256m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| |
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125216#comment-15125216
]
Hadoop QA commented on HDFS-9721:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 19s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 8s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 178m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123277#comment-15123277
]
Hadoop QA commented on HDFS-9721:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 43s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 52s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785114/HDFS-9721.02.patch |
| JIRA Issue | HDFS-9721 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124472#comment-15124472
]
Xiao Chen commented on HDFS-9721:
-
Thanks very much for reviewing [~eddyxu]! I'll address the last 2 comments
shortly.
I'm not sure about the log level though. Logging it at INFO or above will flood
the console, and may confuse the user. For example, in a big cluster, currently
the tool prints the following:
{noformat}
16/01/29 11:07:01 INFO offlineImageViewer.PBImageTextWriter: Found 44801245
INodes in the INode section
16/01/29 11:22:34 INFO offlineImageViewer.PBImageTextWriter: Ignored 2235 nodes.
16/01/29 11:22:34 INFO offlineImageViewer.PBImageTextWriter: Outputted 44801245
INodes.
{noformat}
If we log every exception, there will be 2235 log entries, hence flooding out
the summary info.
I understand your concern. How about I add a sentence to the {{Ignored 2235
nodes.}} to say {{please turn on debug log for details?}}, and change this log
level to WARN?
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124443#comment-15124443
]
Lei (Eddy) Xu commented on HDFS-9721:
-
Hi, [~xiaochen]
The patch LGTM in general. Will +1 after addressing the following comments:
{code}
} catch (IOException ioe) {
++ignored;
if (LOG.isDebugEnabled()) {
LOG.debug("Exception caught, ignoring node:{}.", p.getId());
}
{code}
Would you log the {{IOE}} in log as well? Also I feel that it should put an
{{IOE}} to higher log level, e.g., INFO?
{{getSnapshotName()}} seems just ignoring a ref Id. Would you change the
function name accordingly? Additionally, can you add some comments to the
following code?
{code}
if (parent == null) {
return getSnapshotName(inode);
}
{code}
Thanks!
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124577#comment-15124577
]
Lei (Eddy) Xu commented on HDFS-9721:
-
[~xiaochen] would you consider that {{getSnapshotName()}} dose not throw an
{{IOE}}? If that's an expected case.
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124664#comment-15124664
]
Xiao Chen commented on HDFS-9721:
-
Hm, since it's called in a recursive method, I don't think returning can
distinguish it. How does a specific type of Exception (e.g.
{{IgnoreSnapshotException}}) sound to you [~eddyxu]? We can ignore that and log
IOE then.
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch, HDFS-9721.02.patch,
> HDFS-9721.03.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124243#comment-15124243
]
Hadoop QA commented on HDFS-9721:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 50s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 52m 11s
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785221/HDFS-9721.03.patch |
| JIRA Issue | HDFS-9721 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8a3a6ddb5792 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c9a09d6 |
|
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122865#comment-15122865
]
Xiao Chen commented on HDFS-9721:
-
I studied the legacy_oiv Delimited tool, XML format OIV tool, and have decided
to just allow the Delimited OIV tool to finish loading normal metadata, without
worrying about snapshots.
The legacy_oiv tool seems to print out both the normal namespace and the all
found paths for snapshots all together, plus the snapshot name itself... IMO
this is more confusing than not printing at all. If snapshot info is needed,
one can easily get it from XML OIV tool.
Attached patch 1.
- Load the INodeReference section
- Store and lookup needed mappings along the INodeReference line
- Ignore snapshots
- Fixed a minor bug that writer's {{close}} didn't {{flush}} the stream.
Below is an example of the legacy OIV, and the Delimited OIV in patch 1:
Metadata was simply constructed with:
{code}
bin/hdfs dfs -mkdir /src
bin/hdfs dfs -touchz /src/file
bin/hdfs dfsadmin -allowSnapshot /src
bin/hdfs dfs -createSnapshot /src snap
{code}
{noformat}
Legacy OIV output:
drwxr-xr-x - xiao supergroup 0 2016-01-28 10:37 /snap
drwxr-xr-x - xiao supergroup 0 2016-01-28 10:36 /
drwxr-xr-x - xiao supergroup 0 2016-01-28 10:37 /src
-rw-r--r-- 1 xiao supergroup 0 2016-01-28 10:36 /src/file
Patch 1 oiv -p Delimited output:
/ 0 2016-01-28 10:361969-12-31 16:000 0
0 9223372036854775807 -1 rwxr-xr-x xiaosupergroup
/src0 2016-01-28 10:371969-12-31 16:000 0
0 -1 -1 rwxr-xr-x xiaosupergroup
/src/file 1 2016-01-28 10:362016-01-28 10:36
134217728 0 0 0 0 rw-r--r-- xiao
supergroup
{noformat}
A (a little bit) more complex case that involves INodeReference is included in
the unit test.
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122866#comment-15122866
]
Xiao Chen commented on HDFS-9721:
-
I think at least we need to separate the snapshots and regular paths in the
listing. If there's requirement for that, I'd prefer to address it in a
separate JIRA. Since the XML OIV tool clearly lists snapshots along with INodes
/ INodeReference, I doubt we really need the Delimit tool to do the duplicate
work.
> Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
>
>
> Key: HDFS-9721
> URL: https://issues.apache.org/jira/browse/HDFS-9721
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9721.01.patch
>
>
> HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer
> based fsimage.
> However, if the fsimage contains {{INodeReference}}, the tool fails because:
> {code}Preconditions.checkState(e.getRefChildrenCount() == 0);{code}
> This jira is to propose allow the tool to finish, so that user can get full
> metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9721) Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference
[
https://issues.apache.org/jira/browse/HDFS-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123000#comment-15123000
]
Hadoop QA commented on HDFS-9721:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 13s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 30s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 8s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Switch statement found in
org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.visit(RandomAccessFile)
where default case is missing At PBImageTextWriter.java:where default case is
missing At PBImageTextWriter.java:[lines 467-476] |
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
