[
https://issues.apache.org/jira/browse/ORC-978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408243#comment-17408243
]
Yiqun Zhang edited comment on ORC-978 at 9/1/21, 3:46 PM:
----------------------------------------------------------
I found the reason. [~dongjoon]
{code:java}
@Override
public void close() throws IOException {
if (!isClosed) {
try {
if (batch.size > 0) {
writer.addRowBatch(batch);
batch.reset();
}
} finally {
writer.close();
this.isClosed = true;
}
}
}
{code}
writer.addRowBatch(batch);
{code:java}
.......
checkMemory();
} catch (Throwable t) {
try {
close();
} catch (Throwable ignore) {
// ignore
}
if (t instanceof IOException) {
throw (IOException) t;
} else {
throw new IOException("Problem adding row to " + path, t);
}
}
{code}
*addRowBatch method throws java.lang.OutOfMemoryError causing writerImpl to
close twice*
The first close already set the rawWriter to null, so the second time throw a
NullPointerException.
!screenshot-1.png!
was (Author: guiyankuang):
I found the reason.
{code:java}
@Override
public void close() throws IOException {
if (!isClosed) {
try {
if (batch.size > 0) {
writer.addRowBatch(batch);
batch.reset();
}
} finally {
writer.close();
this.isClosed = true;
}
}
}
{code}
writer.addRowBatch(batch);
{code:java}
.......
checkMemory();
} catch (Throwable t) {
try {
close();
} catch (Throwable ignore) {
// ignore
}
if (t instanceof IOException) {
throw (IOException) t;
} else {
throw new IOException("Problem adding row to " + path, t);
}
}
{code}
*addRowBatch method throws java.lang.OutOfMemoryError causing writerImpl to
close twice*
The first close already set the rawWriter to null, so the second time throw a
NullPointerException.
!screenshot-1.png!
> Fix NPE in TestFlinkOrcReaderWriter
> -----------------------------------
>
> Key: ORC-978
> URL: https://issues.apache.org/jira/browse/ORC-978
> Project: ORC
> Issue Type: Bug
> Components: Java
> Affects Versions: 1.7.0
> Reporter: Dongjoon Hyun
> Priority: Blocker
> Attachments: screenshot-1.png
>
>
> This is a regression at Apache ORC 1.7.0
> -
> https://github.com/dongjoon-hyun/iceberg/pull/8/checks?check_run_id=3475456844
> {code}
> org.apache.iceberg.flink.data.TestFlinkOrcReaderWriter > testMixedTypes FAILED
> java.lang.NullPointerException
> at
> org.apache.orc.impl.PhysicalFsWriter.writeFileMetadata(PhysicalFsWriter.java:415)
> at org.apache.orc.impl.WriterImpl.writeMetadata(WriterImpl.java:578)
> at org.apache.orc.impl.WriterImpl.writeFooter(WriterImpl.java:643)
> at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:735)
> at
> org.apache.iceberg.orc.OrcFileAppender.close(OrcFileAppender.java:127)
> at
> org.apache.iceberg.flink.data.TestFlinkOrcReaderWriter.writeAndValidate(TestFlinkOrcReaderWriter.java:61)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
