jeanlyn created ORC-2226:
----------------------------
Summary: UncompressedStream.seek throws IllegalArgumentException
when SARG selects only an all-null trailing row group
Key: ORC-2226
URL: https://issues.apache.org/jira/browse/ORC-2226
Project: ORC
Issue Type: Bug
Components: Java, Reader
Affects Versions: 2.1.2
Reporter: jeanlyn
h2. Description
Reading an *uncompressed* ORC file with a SearchArgument that selects *only the
last row group of a stripe* fails with:
{noformat}
java.io.IOException: Problem opening stripe 0 footer in orc-seek-eof-repro.orc.
Caused by: java.lang.IllegalArgumentException: Seek in column 2 kind DATA to
8192 is outside of the data
at org.apache.orc.impl.InStream$UncompressedStream.seek(InStream.java:...)
at org.apache.orc.impl.TreeReaderFactory$StringDirectTreeReader.seek(...)
at org.apache.orc.impl.RecordReaderImpl.seekToRowEntry(...)
{noformat}
This happens when a (string) column is entirely NULL in that trailing row
group: its row-group index records the DATA position exactly at the *end of the
DATA stream*. The same file reads fine with a full scan (no SARG) and with the
C++ reader. Reproduced with orc-core *2.1.2*.
h2. Root cause
Two parts:
# {{StripePlanner.processStream}} (2.x;
{{RecordReaderImpl.planReadPartialDataStreams}} in 1.x): when the selected
row-group run ends at the stripe's last row group, the planned read range is
{{[stream.offset + pos, stream.offset + stream.length)}}. For an all-null
trailing row group, {{pos == stream.length}}, so the planned range is *0
bytes*, {{stream.firstChunk}} stays null, and the stream is created with
{{bytes == null}}.
# {{InStream.UncompressedStream.seek(long)}}: the empty-stream shortcut only
handles {{desired == 0 && bytes == null}}. A seek to {{desired == length}} — a
legitimate EOF position that both the Java and C++ writers record for all-null
trailing row groups — falls through the range loop and throws
{{IllegalArgumentException}}.
Related historical issues: HIVE-9471, HIVE-12360 (fixed the Hive-era bundled
reader; this standalone-ORC path is still present in 2.1.2).
h2. Minimal reproduction
Self-contained program (writes a tiny synthetic file, then reads it with a
SARG). Only needs orc-core + its deps on the classpath; tested with the
{{orc-tools}} 2.1.2 uber jar:
{code:java}
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.orc.CompressionKind;
import org.apache.orc.OrcFile;
import org.apache.orc.Reader;
import org.apache.orc.RecordReader;
import org.apache.orc.TypeDescription;
import org.apache.orc.Writer;
import org.apache.orc.storage.ql.exec.vector.BytesColumnVector;
import org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch;
import org.apache.orc.storage.ql.io.sarg.PredicateLeaf;
import org.apache.orc.storage.ql.io.sarg.SearchArgument;
import org.apache.orc.storage.ql.io.sarg.SearchArgumentFactory;
import java.nio.charset.StandardCharsets;
/**
* Minimal repro: reading an *uncompressed* ORC file with a SARG that selects
* ONLY the last row group of a stripe, where a sparse string column is
* entirely NULL in that row group, fails with:
* IllegalArgumentException: Seek in column 2 kind DATA to 8192 is outside of
the data
*
* Layout crafted by this program (single stripe, row index stride 1024):
* row group 0: 1024 rows, k='a', v=distinct 8-byte strings (DATA stream of v
= 8192 bytes)
* row group 1: 512 rows, k='c', v=ALL NULL (row-index DATA position =
8192 = stream end)
* SARG k='c' selects only row group 1 -> the read plan assigns 0 bytes to the
* DATA stream of v -> UncompressedStream.seek(8192) falls off the end and
throws.
*/
public class OrcSeekEofRepro {
public static void main(String[] args) throws Exception {
Path path = new Path(args.length > 0 ? args[0] : "orc-seek-eof-repro.orc");
Configuration conf = new Configuration();
TypeDescription schema =
TypeDescription.fromString("struct<k:string,v:string>");
try (Writer w = OrcFile.createWriter(path, OrcFile.writerOptions(conf)
.setSchema(schema)
.compress(CompressionKind.NONE) // must be uncompressed
.rowIndexStride(1024))) {
VectorizedRowBatch batch = schema.createRowBatchV2();
BytesColumnVector k = (BytesColumnVector) batch.cols[0];
BytesColumnVector v = (BytesColumnVector) batch.cols[1];
batch.reset();
batch.size = 1024;
for (int i = 0; i < batch.size; i++) {
k.setVal(i, "a".getBytes(StandardCharsets.UTF_8));
// distinct 8-byte values so the writer picks DIRECT encoding (no
dictionary)
v.setVal(i, String.format("%07dx", i).getBytes(StandardCharsets.UTF_8));
}
w.addRowBatch(batch);
batch.reset();
batch.size = 512;
v.noNulls = false;
v.isRepeating = true;
v.isNull[0] = true;
for (int i = 0; i < batch.size; i++) {
k.setVal(i, "c".getBytes(StandardCharsets.UTF_8));
}
w.addRowBatch(batch);
}
System.out.println("written: " + path);
Reader reader = OrcFile.createReader(path, OrcFile.readerOptions(conf));
SearchArgument sarg = SearchArgumentFactory.newBuilder()
.equals("k", PredicateLeaf.Type.STRING, args.length > 1 ? args[1] :
"c").build();
try (RecordReader rows = reader.rows(reader.options()
.searchArgument(sarg, new String[]{"k"}))) {
VectorizedRowBatch batch = reader.getSchema().createRowBatchV2();
long total = 0;
while (rows.nextBatch(batch)) {
total += batch.size;
}
System.out.println("READ OK, candidate rows = " + total + " (bug NOT
present)");
} catch (Exception e) {
System.out.println("REPRODUCED: " + e);
if (e.getCause() != null) {
System.out.println("caused by: " + e.getCause());
}
System.exit(1);
}
}
}
{code}
Run:
{noformat}
java -cp orc-tools-2.1.2-uber.jar OrcSeekEofRepro.java
{noformat}
Output on ORC 2.1.2:
{noformat}
REPRODUCED: java.io.IOException: Problem opening stripe 0 footer in
orc-seek-eof-repro.orc.
caused by: java.lang.IllegalArgumentException: Seek in column 2 kind DATA to
8192 is outside of the data
{noformat}
Control case ({{k = 'a'}}, selects only row group 0) reads fine. File metadata
confirms the layout ({{8192}} == the DATA stream length; the row-group index of
the all-null trailing row group points exactly at the end of the stream):
{noformat}
Compression: NONE
Stream: column 2 section DATA start: 153 length 8192
Encoding column 2: DIRECT_V2
Entry 0: count: 1024 hasNull: false ... positions: 0,0,0,0,0,0
Entry 1: count: 0 hasNull: true positions: 0,128,0,8192,8,0
{noformat}
h2. Proposed fix
In {{InStream.UncompressedStream.seek(long)}} (and analogously
{{CompressedStream.seek(long)}}), accept {{desired == length}} as a valid EOF
position instead of throwing:
{code:java}
public void seek(long desired) throws IOException {
if (desired == 0 && bytes == null) {
return;
}
// NEW: seeking to end-of-stream is legitimate — the row-group index of an
// all-null trailing row group points here, and the read planner may have
// assigned zero bytes to this stream (bytes == null). Park at EOF;
// read() already returns -1 when position == length.
if (desired == length) {
position = desired;
currentRange = null;
decrypted = null;
return;
}
...
}
{code}
{{read()}} already returns {{-1}} at {{position == length}} and {{available()}}
returns {{0}}, so parking at EOF is safe. With this change the repro prints
{{READ OK, candidate rows = 512}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
