[
https://issues.apache.org/jira/browse/ORC-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948550#comment-16948550
]
Yukihiro Okada edited comment on ORC-557 at 10/10/19 6:13 PM:
--------------------------------------------------------------
Reproduced this bug with master branch.
{code:shell}
❯ ls -lh /tmp/output.000.orc
-rw-r--r-- 1 yuokada wheel 4.5G 10 10 21:31 /tmp/output.000.orc
❯ java -jar tools/target/orc-tools-1.7.0-SNAPSHOT-uber.jar meta
/tmp/output.000.orc
Processing data file /tmp/output.000.orc [length: 4805398749]
Exception in thread "main" java.io.IOException: Problem reading file footer
/tmp/output.000.orc
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:717)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:500)
at org.apache.orc.OrcFile.createReader(OrcFile.java:365)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:333)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
at org.apache.orc.tools.FileDump.main(FileDump.java:135)
at org.apache.orc.tools.Driver.main(Driver.java:108)
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (43291 >
29438)
at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
at java.base/java.nio.Buffer.position(Buffer.java:293)
at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
at
org.apache.orc.impl.InStream$UncompressedStream.setCurrent(InStream.java:134)
at
org.apache.orc.impl.InStream$UncompressedStream.reset(InStream.java:110)
at
org.apache.orc.impl.InStream$UncompressedStream.<init>(InStream.java:100)
at org.apache.orc.impl.InStream.create(InStream.java:844)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:706)
... 7 more
{code}
But, I got another error message when I build jar file with JDK8 and run it.
{code:shell}
❯ java -jar tools/target/orc-tools-1.7.0-SNAPSHOT-uber.jar meta
/tmp/output.000.orc
[03:10:14]
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Processing data file /tmp/output.000.orc [length: 4805398749]
Exception in thread "main" java.io.IOException: Problem reading file footer
/tmp/output.000.orc
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:716)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:500)
at org.apache.orc.OrcFile.createReader(OrcFile.java:365)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:333)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
at org.apache.orc.tools.FileDump.main(FileDump.java:135)
at org.apache.orc.tools.Driver.main(Driver.java:108)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at
org.apache.orc.impl.InStream$UncompressedStream.setCurrent(InStream.java:134)
at
org.apache.orc.impl.InStream$UncompressedStream.reset(InStream.java:110)
at
org.apache.orc.impl.InStream$UncompressedStream.<init>(InStream.java:100)
at org.apache.orc.impl.InStream.create(InStream.java:844)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:706)
... 7 more
{code}
was (Author: yuokada):
Reproduced this bug with master branch.
{code:shell}
❯ ls -lh /tmp/output.000.orc
-rw-r--r-- 1 yuokada wheel 4.5G 10 10 21:31 /tmp/output.000.orc
❯ java -jar tools/target/orc-tools-1.7.0-SNAPSHOT-uber.jar meta
/tmp/output.000.orc
Processing data file /tmp/output.000.orc [length: 4805398749]
Exception in thread "main" java.io.IOException: Problem reading file footer
/tmp/output.000.orc
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:717)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:500)
at org.apache.orc.OrcFile.createReader(OrcFile.java:365)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:333)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
at org.apache.orc.tools.FileDump.main(FileDump.java:135)
at org.apache.orc.tools.Driver.main(Driver.java:108)
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (43291 >
29438)
at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
at java.base/java.nio.Buffer.position(Buffer.java:293)
at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
at
org.apache.orc.impl.InStream$UncompressedStream.setCurrent(InStream.java:134)
at
org.apache.orc.impl.InStream$UncompressedStream.reset(InStream.java:110)
at
org.apache.orc.impl.InStream$UncompressedStream.<init>(InStream.java:100)
at org.apache.orc.impl.InStream.create(InStream.java:844)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:706)
... 7 more
{code}
> Large ORC file parsing failed
> -----------------------------
>
> Key: ORC-557
> URL: https://issues.apache.org/jira/browse/ORC-557
> Project: ORC
> Issue Type: Bug
> Components: Reader, tools
> Affects Versions: 1.6.0, 1.6.1
> Reporter: 周娜
> Priority: Major
> Attachments: image-2019-10-09-15-36-48-079.png
>
>
> When writing a more than 4G ORC file. The following error will occur when
> dump the file：
> !image-2019-10-09-15-36-48-079.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
