SteNicholas opened a new pull request, #563:
URL: https://github.com/apache/paimon-cpp/pull/563
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #560
<!-- What is the purpose of the change -->
Page-filtered Parquet reads decoded every selected row of every projected
field before returning
any batch. The reader retained decoded columns for the entire filtered row
group, delaying the
first batch and making decoded-output memory grow with the selected row
count.
The change follows the stateful batching approach used by Velox's Parquet
row reader, retaining
field decoders and advancing them by the requested batch size:
- **`PageFilteredRowGroupReader`**: keeps per-field decoders, per-leaf
selection cursors and
skip/read progress across `ReadNext` calls. Each call decodes a bounded
window of selected
logical rows, including nested leaves whose page boundaries and compressed
row coordinates
differ. Existing page pruning, pre-buffering and missing-index fallback
remain in use.
- **Batch assembly**: creates `RecordBatch` output directly when every
column has one chunk.
Columns split into multiple chunks are aligned within the current decoding
window, with
array offsets normalized. The reader exposes the projected schema before
reading data and
preserves dictionary output, nested rows and ownership of returned batches.
- **Reader lifecycle**: caches decoding errors, releases decoder state at
EOF or close and
keeps previously returned batches valid.
`FileReaderWrapper::PrepareForReading` discards
the active filtered reader when read ranges or projections change; file
row IDs continue
to refer to the original selected rows across batches and row groups.
- **Parquet benchmarks**: report mean first-batch latency and peak tracked
Arrow memory
alongside throughput and storage bytes. A partial-row-group benchmark
compares 64, 1024
and 4096 rows per batch with page-index filtering enabled and disabled.
The memory bound applies to decoded output for the active window.
Pre-buffered input, required
page/decoder state and batches retained by the caller remain separate memory
costs. Generic
late-materialization probe/payload streaming is outside this change.
### Tests
<!-- List UT and IT cases to verify this change -->
UT:
- `src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp`:
- `MultiColumnMultiBatchStreaming`: consolidates overlapping batch/offset
tests and checks
integer/string column alignment, exact batch sizes, contents and
normalized offsets.
- `NestedStructSubColumnProjection` and
`NestedColumnsWithNullsMisalignedPagesMultiRowGroup`: projected nested
schemas, nulls,
misaligned struct/list/map pages, mixed full/partial row groups,
original file row IDs and
returned-array validity after reader destruction, with and without page
indexes.
- `StreamingPreservesDictionaryValuesAcrossSelections` and
`DictionaryEmptySelectionDoesNotReadPages`: dictionary schemas and
values across batches,
invalid batch sizes, repeated EOF and empty selections without
dictionary/data-page reads.
- `StreamingBoundsDecodedMemoryAndPreservesReturnedBatches`: bounded
tracked memory during
first-batch and complete reads, early/repeated close, decoder cleanup
and returned-batch
ownership after reader destruction.
- `StreamingDefersLaterPageDecodeErrors`: a corrupt later page does not
prevent earlier
batches from being returned; subsequent reads preserve the failure and
prior batches.
- `DirectOffsetIndexJumpDoesNotReadUnselectedPageHeaders`,
`DirectOffsetIndexJumpSupportsDictionaryFallbackToPlain` and
`DirectOffsetIndexJumpDataPageV2AdjacentPages`: small-batch streaming
preserves page pruning,
dictionary-to-plain transitions and DataPageV2 page-boundary handling.
- `MissingOffsetIndexFallsBackToSequentialRead` and
`InvalidOffsetIndexFallsBackWithoutChangingSelectedRows`: fallback
preserves selected rows
across batches, including nested fields with one valid and one missing
or malformed index.
- `src/paimon/format/parquet/file_reader_wrapper_test.cpp`:
- `SeekBackToConsumedPageFilteredRowGroup`: seeking back and restarting a
partially consumed
filtered row group reconstructs the reader and returns the same rows.
- `PageFilteredRespectsBatchSize`: exact output sizes and original row
positions across
several batch sizes, including an empty column projection.
- `ApplyReadRangesWiderSecondCall` and `PrepareForReading`: changing
ranges or projections
during streaming resets the active reader, preserves retained batches
and works with
immediate and lazy preparation.
IT:
No standalone integration tests are added. The format-layer tests exercise
the end-to-end
`ParquetFileBatchReader` path, including bitmap selection and file row ID
translation.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No public API under `include/`, stored file format or table protocol
changes. The existing
Parquet batch-size setting now bounds incremental decoding in the
page-filtered path.
### Documentation
<!-- Does this change introduce a new feature -->
`docs/source/examples/benchmark.rst` describes the first-batch latency and
tracked-memory
counters, their measurement boundaries and the filtered batch-size
comparison. It also explains
how to use storage-byte counters to distinguish decoding changes from page
pruning.
### Generative AI tooling
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]