LuciferYang opened a new issue, #9560: URL: https://github.com/apache/paimon/issues/9560
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `2788fe596` (2.1-SNAPSHOT). ### Compute Engine Flink and Spark, on any table with `'file.format' = 'row'` that is read with a column projection. ### Minimal reproduce step Open two readers from one reader factory and read a row from each: ```java FormatReaderFactory factory = format.createReaderFactory(fullType, projectedType, filters); try (FileRecordReader<InternalRow> a = factory.createReader(ctxA); FileRecordReader<InternalRow> b = factory.createReader(ctxB)) { InternalRow rowA = a.readBatch().next(); // row of file A InternalRow rowB = b.readBatch().next(); // row of file B rowA.getInt(0); // file B's value: rowA and rowB are one object } ``` `RowFileFormat.createReaderFactory` builds a single `NestedProjectedRow` and stores it in the factory: ```java NestedProjectedRow projection = NestedProjectedRow.create(dataSchemaRowType, projectedRowType); return new RowFormatReaderFactory(dataSchemaRowType, projection); ``` `RowFormatReaderFactory.createReader` then passes that same instance to every reader, and `RowFileRecordIterator` applies it with `projection.replaceRow(row)`, which returns `this`. So all readers of one factory hand out the same object, and reading from one overwrites the row a consumer of another is still holding. ### What doesn't meet your expectations? `RecordReader.readBatch` documents that "the returned iterator object and any contained objects may be held onto by the source for some time, so it should not be immediately reused by the reader". Per reader that holds. Across readers of one factory it does not, and nothing reports a problem. Where it matters is merge-on-read, which holds one row per sorted run: `KeyValueFileReaderFactory` caches one `FormatReaderMapping` per `(schemaId, formatIdentifier)`, `MergeTreeReaders` opens one reader per sorted run through it, and `SortMergeReaderWithMinHeap` (or the loser tree) keeps each run's current `KeyValue` while advancing the others. With `file.format = 'row'` and a projection, those keys all point at the last row read, so the merge compares the wrong keys. I traced that path from the code and did not reproduce it end to end. ### Anything else? The row format has done this since it was added in #7934. Other formats keep this state per reader, so the same aliasing does not arise there: the parquet reader builds its `ColumnarRow` inside the reader. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
