SteNicholas opened a new issue, #560:
URL: https://github.com/apache/paimon-cpp/issues/560

   ### Search before asking
   
   - [x] I searched the existing issues and open pull requests and found no 
issue tracking incremental decoding of page-filtered row groups. Related work: 
#308 sizes builders for selected rows; #228 provides format-level benchmarks.
   
   ### Motivation
   
   
[PageFilteredRowGroupReader::ReadFilteredRowGroup](https://github.com/apache/paimon-cpp/blob/main/src/paimon/format/parquet/page_filtered_row_group_reader.cpp#L416)
 currently reads every projected field for the complete filtered row group, 
collects the resulting ChunkedArrays, constructs an Arrow Table, and only then 
exposes batches through TableRecordBatchReader. The [existing 
TODO](https://github.com/apache/paimon-cpp/blob/main/src/paimon/format/parquet/page_filtered_row_group_reader.cpp#L470)
 describes this difference from the fully matched path, which decodes 
incrementally.
   
   For a large row group or wide projection, the first output batch therefore 
waits for all selected rows and columns to be decoded. Peak decoded-output 
memory grows with the complete selected row group, even when the caller 
consumes one small batch at a time or stops early. Sizing builders by selected 
rows (#308) avoids excessive reservations, but does not remove the retained 
output itself.
   
   ### Solution
   
   Make the page-filtered path a stateful RecordBatchReader:
   
   - Keep per-field decoder state, selection-range cursors, and skip/read 
progress across ReadNext calls.
   - Decode one bounded output window at a time, respecting the requested batch 
size and keeping columns aligned by selected logical rows.
   - Assemble each batch directly instead of materializing a whole-row-group 
Table first.
   - Preserve existing page-range pruning, pre-buffering, bitmap selection, 
dictionary output, and Arrow layout/lifetime contracts.
   - Handle nested repetition/definition levels and columns whose physical page 
boundaries differ.
   
   Scope this change to incremental decoding in the Parquet format layer. 
Streaming the generic late-materialization probe/payload reader is a separate 
improvement. No file-format or table-protocol change is needed.
   
   ### Anything else?
   
   Acceptance and validation:
   
   - [ ] On a row group spanning several output batches, producing the first 
batch does not decode every selected row of every projected field.
   - [ ] Peak decoded-output memory is bounded by the active output window plus 
required page/decoder state, excluding batches deliberately retained by the 
caller.
   - [ ] Results and original file-row IDs match the existing path for 
contiguous and scattered selections, partial pages, nulls, nested columns, 
dictionaries, multiple row groups, and missing page indexes.
   - [ ] Cover seek, read-range changes, EOF, and early close without leaking 
buffers or invalidating returned batches.
   - [ ] Extend the existing page-filtered reader tests and Parquet benchmark 
to report first-batch latency, peak tracked memory, throughput, and storage 
bytes. Verify that unchanged page pruning does not increase physical reads 
unexpectedly.
   
   This proposal is based on source inspection; performance improvements should 
be measured rather than assumed.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to