[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Description: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6 This problem can be easily reproduced when there are at least TWO snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point. was: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6 This problem can be easily reproduced when there are at least TWO snapshot-enabled directories. Apply > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-14148.test.001.patch, fsimage_024, > fsimage_441 > > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) > $ grep -n "" fsimage_0026542.xml > 228:222049220495 > {code} > This is also reproduced on latest trunk > fba222a85603d6321419aa37bcc48d276dd6c4a6 > This problem can be easily reproduced when there are at least TWO > sn
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Description: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6 This problem can be easily reproduced when there are at least TWO snapshot-enabled directories. Apply was: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6 It seems this problem can be easily reproduced when there are at least TWO snapshot-enabled directories. {code:title=trunk} $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149) 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException {code} > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-14148.test.001.patch, fsimage_024, > fsimage_441 > > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImage
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Attachment: HDFS-14148.test.001.patch > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-14148.test.001.patch, fsimage_024, > fsimage_441 > > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) > $ grep -n "" fsimage_0026542.xml > 228:222049220495 > {code} > This is also reproduced on latest trunk > fba222a85603d6321419aa37bcc48d276dd6c4a6 > It seems this problem can be easily reproduced when there are at least TWO > snapshot-enabled directories. > {code:title=trunk} > $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149) > 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: > ExitException > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Attachment: fsimage_024 > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: fsimage_024, fsimage_441 > > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) > $ grep -n "" fsimage_0026542.xml > 228:222049220495 > {code} > This is also reproduced on latest trunk > fba222a85603d6321419aa37bcc48d276dd6c4a6 > It seems this problem can be easily reproduced when there are at least TWO > snapshot-enabled directories. > {code:title=trunk} > $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149) > 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: > ExitException > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Description: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6 It seems this problem can be easily reproduced when there are at least TWO snapshot-enabled directories. was: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: fsimage_441 > > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) > $ grep -n "" fsimage_0026542.xml > 228:222049220495 > {code} > This is also reproduced on latest trunk > fba222a85603d6321419aa37bcc48d276dd6c4a6 > It seems this problem can be easily reproduced when there are at least TWO > snapshot-enabled directories. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Description: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6 It seems this problem can be easily reproduced when there are at least TWO snapshot-enabled directories. {code:title=trunk} $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149) 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException {code} was: The current HDFS OIV tool doesn't seem to support snapshot well when reversing XML back to binary. {code:bash} $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML OfflineImageReconstructor failed: Found unknown XML keys in : dir java.io.IOException: Found unknown XML keys in : dir at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) $ grep -n "" fsimage_0026542.xml 228:222049220495 {code} This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6 It seems this problem can be easily reproduced when there are at least TWO snapshot-enabled directories. > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: fsimage_441 > > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReco
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Attachment: fsimage_441 > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: fsimage_441 > > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) > $ grep -n "" fsimage_0026542.xml > 228:222049220495 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14148: -- Summary: HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir (was: HDFS OIV ReverseXML doesn't support snapshot well) > HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir > > > Key: HDFS-14148 > URL: https://issues.apache.org/jira/browse/HDFS-14148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Priority: Major > > The current HDFS OIV tool doesn't seem to support snapshot well when > reversing XML back to binary. > {code:bash} > $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML > OfflineImageReconstructor failed: Found unknown XML keys in : dir > java.io.IOException: Found unknown XML keys in : dir > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193) > at > org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136) > $ grep -n "" fsimage_0026542.xml > 228:222049220495 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org