SteNicholas opened a new issue, #560: URL: https://github.com/apache/paimon-cpp/issues/560
### Search before asking - [x] I searched the existing issues and open pull requests and found no issue tracking incremental decoding of page-filtered row groups. Related work: #308 sizes builders for selected rows; #228 provides format-level benchmarks. ### Motivation [PageFilteredRowGroupReader::ReadFilteredRowGroup](https://github.com/apache/paimon-cpp/blob/main/src/paimon/format/parquet/page_filtered_row_group_reader.cpp#L416) currently reads every projected field for the complete filtered row group, collects the resulting ChunkedArrays, constructs an Arrow Table, and only then exposes batches through TableRecordBatchReader. The [existing TODO](https://github.com/apache/paimon-cpp/blob/main/src/paimon/format/parquet/page_filtered_row_group_reader.cpp#L470) describes this difference from the fully matched path, which decodes incrementally. For a large row group or wide projection, the first output batch therefore waits for all selected rows and columns to be decoded. Peak decoded-output memory grows with the complete selected row group, even when the caller consumes one small batch at a time or stops early. Sizing builders by selected rows (#308) avoids excessive reservations, but does not remove the retained output itself. ### Solution Make the page-filtered path a stateful RecordBatchReader: - Keep per-field decoder state, selection-range cursors, and skip/read progress across ReadNext calls. - Decode one bounded output window at a time, respecting the requested batch size and keeping columns aligned by selected logical rows. - Assemble each batch directly instead of materializing a whole-row-group Table first. - Preserve existing page-range pruning, pre-buffering, bitmap selection, dictionary output, and Arrow layout/lifetime contracts. - Handle nested repetition/definition levels and columns whose physical page boundaries differ. Scope this change to incremental decoding in the Parquet format layer. Streaming the generic late-materialization probe/payload reader is a separate improvement. No file-format or table-protocol change is needed. ### Anything else? Acceptance and validation: - [ ] On a row group spanning several output batches, producing the first batch does not decode every selected row of every projected field. - [ ] Peak decoded-output memory is bounded by the active output window plus required page/decoder state, excluding batches deliberately retained by the caller. - [ ] Results and original file-row IDs match the existing path for contiguous and scattered selections, partial pages, nulls, nested columns, dictionaries, multiple row groups, and missing page indexes. - [ ] Cover seek, read-range changes, EOF, and early close without leaking buffers or invalidating returned batches. - [ ] Extend the existing page-filtered reader tests and Parquet benchmark to report first-batch latency, peak tracked memory, throughput, and storage bytes. Verify that unchanged page pruning does not increase physical reads unexpectedly. This proposal is based on source inspection; performance improvements should be measured rather than assumed. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
