ericyuan915 opened a new issue, #19208:
URL: https://github.com/apache/hudi/issues/19208
### Bug Description
**What happened:**
Reading a nested ROW column from a base Parquet file whose row count exceeds
the vectorized read batch size (RecordIterators.DEFAULT_BATCH_SIZE = 2048)
throws ArrayIndexOutOfBoundsException from NestedColumnReader#readRow. This is
a plain base-file read. It reproduces for any base Parquet file with a nested
ROW column that spans more than one read batch.
**What you expected:**
The nested ROW column reads back correctly across the batch boundary — rows
past 2048 returned, all-null child rows collapsed to NULL as intended — with no
exception, identical to reading the same data when it fits in a single batch.
**Steps to reproduce:**
1. Write more than 2048 rows (e.g. 2048 + 100) containing a nested ROW
column (e.g. row<f0 int, f1 string>) into a single base Parquet file, so one
read split crosses the 2048-row batch boundary.
2. Read the table selecting the nested ROW column.
3. ParquetColumnarRowSplitReader#nextBatch crashes with
ArrayIndexOutOfBoundsException the moment it processes the batch that crosses
row 2048.
(A row<row<decimal>> shape reproduces the same crash via a slightly
different path — see the DECIMAL note in root cause.)
**Root cause:**
The FLINK-35702 Dremel nested reader carries a one-record lookahead. On a
full, non-final batch, NestedPrimitiveColumnReader#readAndNewVector reads one
value past the batch in its do { … } while (readValue() && repetitionLevel !=
0) loop, and #getLevelDelegation returns the accumulated definition/repetition
level arrays including that trailing level (then re-seeds the lists with it for
the next batch). NestedPositionUtil#calculateRowOffsets returns positionsCount
= fieldDefinitionLevels.length, so positionsCount = batchSize + 1 = 2049, while
the materialized child column vectors are sized to their value count = 2048.
This lookahead is harmless in Flink itself, and harmless on Hudi's
primitive/collection paths, because ParquetColumnarRowSplitReader#nextBatch
caps the emitted batch at num = min(batchSize, remaining) — the phantom 2049th
position is never surfaced downstream.
The crash is in a Hudi-specific divergence from Flink:
NestedColumnReader#readRow has an extra "collapse a row whose children are all
null into a null row" loop, so that a SQL row(null, null) round-trips to NULL
(preserving the deleted legacy RowColumnReader semantics; the in-code comment
notes this diverges from Flink 2.1, which would surface Row(null, null)). This
loop does
`int rowCount = rowPosition.getPositionsCount(); // 2049
for (int j = 0; j < rowCount; j++) {
if (heapRowVector.isNullAt(j)) continue;
for (WritableColumnVector child : finalChildrenVectors) {
if (!child.isNullAt(j)) { ... } // child len == 2048 →
j==2048 overruns
}
} `
At j = 2048 it calls child.isNullAt(2048) on a length-2048
AbstractHeapVector → Index 2048 out of bounds for length 2048. Flink's own
NestedColumnReader has no such loop, so upstream Flink is unaffected — this is
an upstream-Hudi defect introduced with the nested-reader port.
DECIMAL subtlety: DECIMAL leaves are materialized as a ParquetDecimalVector
wrapping a Heap*Vector, which is not itself an AbstractHeapVector. A nested row
whose short children are all decimals (e.g. row<row<decimal>>) hits the same
overrun and must be unwrapped when computing the safe bound.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT). Present since the FLINK-35702
Dremel nested-reader port — #18636 (2026-05-07), #18700 (2026-05-08), #18701
(2026-05-09) — and extended to hudi-flink2.0 by the backport #18959
(2026-06-11). Not present in 1.2.0 or earlier.
**Query engine:** (Spark/Flink/Trino etc) Flink. Affects every hudi-flink
module that carries the Dremel NestedColumnReader: hudi-flink1.18.x,
hudi-flink1.19.x, hudi-flink1.20.x, hudi-flink2.0.x, hudi-flink2.1.x.
(hudi-flink1.17.x does not have this reader and is unaffected.)
**Relevant configs:** None — default vectorized COW read path. No config
gates it; the batch size is the hardcoded RecordIterators.DEFAULT_BATCH_SIZE =
2048. Triggered whenever a single read split over a base file with a nested ROW
column crosses 2048 rows.
### Logs and Stack Trace
java.lang.ArrayIndexOutOfBoundsException: Index 2048 out of bounds for
length 2048
at
org.apache.flink.table.data.columnar.vector.heap.AbstractHeapVector.isNullAt(AbstractHeapVector.java:94)
at
org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader.readRow(NestedColumnReader.java:162)
...
at
org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader.nextBatch(ParquetColumnarRowSplitReader.java:352)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
