[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

This problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
*TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

This problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories. Apply 


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> sn

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

This problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories. Apply 

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.

{code:title=trunk}
$ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImage

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Attachment: HDFS-14148.test.001.patch

> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories.
> {code:title=trunk}
> $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
> 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: 
> ExitException
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Attachment: fsimage_024

> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_024, fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories.
> {code:title=trunk}
> $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
> 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: 
> ExitException
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.

{code:title=trunk}
$ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReco

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-01-29 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Attachment: fsimage_441

> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2018-12-13 Thread Siyao Meng (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Summary: HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in 
: dir  (was: HDFS OIV ReverseXML doesn't support snapshot well)

> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Priority: Major
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org