LuciferYang opened a new issue, #9633: URL: https://github.com/apache/paimon/issues/9633
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Spark and Flink, on a table with `row-tracking.enabled`, reading Parquet through the vectorized reader. ### Minimal reproduce step Read a row-tracking table whose data file has more than a few batches. `ColumnarRowIterator.assignRowTracking` replaces the `_ROW_ID` and `_SEQUENCE_NUMBER` vectors with a wrapper over the vector that is currently there: ```java final ColumnVector rowIdVector = vectors[index]; vectors[index] = new LongColumnVector() { @Override public long getLong(int i) { if (rowIdVector.isNullAt(i)) { return firstRowId + returnedPosition(); } else { return ((LongColumnVector) rowIdVector).getLong(i); } } ... ``` `VectorizedParquetRecordReader` builds one `ColumnarBatch` per file and returns the same iterator for every batch, and `DataFileRecordReader.readBatchInternal` calls `assignRowTracking` once per batch, so batch N gets a wrapper around batch N-1's wrapper. Reading one row then walks N levels of delegation, and a file with enough batches ends in a `StackOverflowError`. Wrapping 100000 times in a unit test and reading one row reproduces exactly that. Separately, `firstRowId` is nullable: `RawFileSplitRead` notes that a file may carry no stored row id. When it is null and a row's `_rowid` is null too, `firstRowId + returnedPosition()` unboxes null and throws a `NullPointerException`. The row-based branch of the same method already guards this with `firstRowId != null`. ### What doesn't meet your expectations? Per-batch bookkeeping should not accumulate per-batch state. The wrapper is idempotent in intent: assigning row tracking twice should produce the same vector, not two layers. Today the cost of reading `_ROW_ID` grows with the number of batches already read from the file, which is quadratic over a file and eventually fatal. ### Anything else? The nesting needs `partitionInfo` and `indexMapping` to both be null, since otherwise `mapping()` copies the vector array each batch and the wrappers do not accumulate. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
