[
https://issues.apache.org/jira/browse/ORC-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645259#comment-16645259
]
ASF GitHub Bot commented on ORC-416:
------------------------------------
omalley closed pull request #320: ORC-416: Avoid opening data reader when there
is no stripe
URL: https://github.com/apache/orc/pull/320
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java
b/java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java
index a8e0be17e3..731b46e26e 100644
--- a/java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java
+++ b/java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java
@@ -249,7 +249,6 @@ protected RecordReaderImpl(ReaderImpl fileReader,
.withZeroCopy(zeroCopy)
.build());
}
- this.dataReader.open();
firstRow = skippedRows;
totalRowCount = rows;
Boolean skipCorrupt = options.getSkipCorruptRecords();
diff --git a/java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
b/java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
index 1cf01dea71..66951ff680 100644
--- a/java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
+++ b/java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
@@ -25,6 +25,7 @@
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
@@ -2096,4 +2097,26 @@ public void testPickRowGroupsError() throws Exception {
assertEquals(1, applier.getExceptionCount()[0]);
assertEquals(0, applier.getExceptionCount()[1]);
}
+
+ @Test
+ public void testSkipDataReaderOpen() throws Exception {
+ IOException ioe = new IOException("Don't open when there is no stripe");
+
+ DataReader mockedDataReader = mock(DataReader.class);
+ doThrow(ioe).when(mockedDataReader).open();
+ when(mockedDataReader.clone()).thenReturn(mockedDataReader);
+ doNothing().when(mockedDataReader).close();
+
+ Configuration conf = new Configuration();
+ Path path = new Path(workDir, "empty.orc");
+ FileSystem.get(conf).delete(path, true);
+ OrcFile.WriterOptions options =
OrcFile.writerOptions(conf).setSchema(TypeDescription.createLong());
+ Writer writer = OrcFile.createWriter(path, options);
+ writer.close();
+
+ Reader reader = OrcFile.createReader(path, OrcFile.readerOptions(conf));
+ Reader.Options readerOptions =
reader.options().dataReader(mockedDataReader);
+ RecordReader recordReader = reader.rows(readerOptions);
+ recordReader.close();
+ }
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Avoid opening data reader when there is no stripe
> -------------------------------------------------
>
> Key: ORC-416
> URL: https://issues.apache.org/jira/browse/ORC-416
> Project: ORC
> Issue Type: Bug
> Components: Java
> Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3
> Reporter: Dongjoon Hyun
> Priority: Major
>
> Currently, `RecordReaderImpl` invokes `dataReader.open` in the middle of the
> constructor. We can postpone this until `advanceToNextRow` reads the stripe
> at the end of the constructor.
> {code:java}
> this.dataReader.open();
> {code}
> This will reduce the chance of potential open-file leakages due to
> IOException and OOM during `RecordReaderImpl` construction. Also, we can
> avoid it completely when there is no stripe in the file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
