SteNicholas opened a new pull request, #563:
URL: https://github.com/apache/paimon-cpp/pull/563

   <!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): 
<description> -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #560
   
   <!-- What is the purpose of the change -->
   Page-filtered Parquet reads decoded every selected row of every projected 
field before returning
   any batch. The reader retained decoded columns for the entire filtered row 
group, delaying the
   first batch and making decoded-output memory grow with the selected row 
count.
   
   The change follows the stateful batching approach used by Velox's Parquet 
row reader, retaining
   field decoders and advancing them by the requested batch size:
   
   - **`PageFilteredRowGroupReader`**: keeps per-field decoders, per-leaf 
selection cursors and
     skip/read progress across `ReadNext` calls. Each call decodes a bounded 
window of selected
     logical rows, including nested leaves whose page boundaries and compressed 
row coordinates
     differ. Existing page pruning, pre-buffering and missing-index fallback 
remain in use.
   - **Batch assembly**: creates `RecordBatch` output directly when every 
column has one chunk.
     Columns split into multiple chunks are aligned within the current decoding 
window, with
     array offsets normalized. The reader exposes the projected schema before 
reading data and
     preserves dictionary output, nested rows and ownership of returned batches.
   - **Reader lifecycle**: caches decoding errors, releases decoder state at 
EOF or close and
     keeps previously returned batches valid. 
`FileReaderWrapper::PrepareForReading` discards
     the active filtered reader when read ranges or projections change; file 
row IDs continue
     to refer to the original selected rows across batches and row groups.
   - **Parquet benchmarks**: report mean first-batch latency and peak tracked 
Arrow memory
     alongside throughput and storage bytes. A partial-row-group benchmark 
compares 64, 1024
     and 4096 rows per batch with page-index filtering enabled and disabled.
   
   The memory bound applies to decoded output for the active window. 
Pre-buffered input, required
   page/decoder state and batches retained by the caller remain separate memory 
costs. Generic
   late-materialization probe/payload streaming is outside this change.
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   UT:
   
   - `src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp`:
     - `MultiColumnMultiBatchStreaming`: consolidates overlapping batch/offset 
tests and checks
       integer/string column alignment, exact batch sizes, contents and 
normalized offsets.
     - `NestedStructSubColumnProjection` and
       `NestedColumnsWithNullsMisalignedPagesMultiRowGroup`: projected nested 
schemas, nulls,
       misaligned struct/list/map pages, mixed full/partial row groups, 
original file row IDs and
       returned-array validity after reader destruction, with and without page 
indexes.
     - `StreamingPreservesDictionaryValuesAcrossSelections` and
       `DictionaryEmptySelectionDoesNotReadPages`: dictionary schemas and 
values across batches,
       invalid batch sizes, repeated EOF and empty selections without 
dictionary/data-page reads.
     - `StreamingBoundsDecodedMemoryAndPreservesReturnedBatches`: bounded 
tracked memory during
       first-batch and complete reads, early/repeated close, decoder cleanup 
and returned-batch
       ownership after reader destruction.
     - `StreamingDefersLaterPageDecodeErrors`: a corrupt later page does not 
prevent earlier
       batches from being returned; subsequent reads preserve the failure and 
prior batches.
     - `DirectOffsetIndexJumpDoesNotReadUnselectedPageHeaders`,
       `DirectOffsetIndexJumpSupportsDictionaryFallbackToPlain` and
       `DirectOffsetIndexJumpDataPageV2AdjacentPages`: small-batch streaming 
preserves page pruning,
       dictionary-to-plain transitions and DataPageV2 page-boundary handling.
     - `MissingOffsetIndexFallsBackToSequentialRead` and
       `InvalidOffsetIndexFallsBackWithoutChangingSelectedRows`: fallback 
preserves selected rows
       across batches, including nested fields with one valid and one missing 
or malformed index.
   - `src/paimon/format/parquet/file_reader_wrapper_test.cpp`:
     - `SeekBackToConsumedPageFilteredRowGroup`: seeking back and restarting a 
partially consumed
       filtered row group reconstructs the reader and returns the same rows.
     - `PageFilteredRespectsBatchSize`: exact output sizes and original row 
positions across
       several batch sizes, including an empty column projection.
     - `ApplyReadRangesWiderSecondCall` and `PrepareForReading`: changing 
ranges or projections
       during streaming resets the active reader, preserves retained batches 
and works with
       immediate and lazy preparation.
   
   IT:
   
   No standalone integration tests are added. The format-layer tests exercise 
the end-to-end
   `ParquetFileBatchReader` path, including bitmap selection and file row ID 
translation.
   
   ### API and Format
   
   <!-- Does this change affect API in include dir or storage format or 
protocol -->
   No public API under `include/`, stored file format or table protocol 
changes. The existing
   Parquet batch-size setting now bounds incremental decoding in the 
page-filtered path.
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   `docs/source/examples/benchmark.rst` describes the first-batch latency and 
tracked-memory
   counters, their measurement boundaries and the filtered batch-size 
comparison. It also explains
   how to use storage-byte counters to distinguish decoding changes from page 
pruning.
   
   ### Generative AI tooling
   
   Generated-by: Codex (GPT-6)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to