lucasfang opened a new issue, #308: URL: https://github.com/apache/paimon-cpp/issues/308
## Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ## Motivation A page-filtered read resets every leaf through `PageFilteredRowGroupReader::ExecuteSkipReadPattern`, which calls `parquet::arrow::ColumnReader::ResetLeaf(col_idx, total)` where `total` is the leaf's whole-row-group size in post-page-filter compressed space, and that single number drives `RecordReader::Reserve(total)` for the leaf. When a selective predicate appends only a handful of rows out of a large row group, this mis-sizes the Arrow builders in two directions at once. For a variable-width (BYTE_ARRAY) leaf, `Reserve(total)` allocates the `::arrow::BinaryBuilder` offsets buffer for the entire row group — tens of MiB of offsets that are never written, because only the selected rows are appended. At the same time the builder's value-data buffer is not reserved at all: `Reserve()` deliberately leaves it alone for BYTE_ARRAY readers (`uses_values_` is false for them), so it grows by doubling once per decoded batch, and every doubling copies everything the row group has accumulat ed so far, which is quadratic on a multi-batch selective read. ## Solution Split the single `reserve` into three numbers that describe what actually happens, threaded from the format layer down into the patched Arrow, so each leaf sizes its builders once for the selection instead of for the whole row group. - `reserve_records` is the old `total`, the leaf's compressed-space size, and still bounds the rep/def levels that `SkipRecords` walks, so it stays at the full row-group size. - `reserve_values` is `RowRanges::RowCount()` of the effective selection — the rows actually appended — and sizes the offsets buffer for the selection rather than the group. - `reserve_value_bytes` drives a new `parquet::RecordReader::ReserveValueBytes(num_values, num_bytes)` hook that reserves the BinaryBuilder data buffer once; only the BYTE_ARRAY readers override it, and fixed-width readers keep the default no-op because `Reserve()` already sizes their values buffer. - The byte estimate is `min(avg_width * reserve_values, total_uncompressed_size)` with `avg_width = column_chunk->total_uncompressed_size() / column_chunk->num_values()`. `total_uncompressed_size` also covers page headers, levels, and BYTE_ARRAY's 4-byte length prefixes, so `avg_width` over-estimates — the safe direction, since reserving too much costs one allocation while reserving too little costs a doubling copy of everything read so far — and the chunk size caps it, which no selection can exceed. It is guarded by `num_values() > 0 && reserve_values > 0`. - `ReserveValueBytes` clamps the data reservation to `kBinaryMemoryLimit - value_data_length()`, mirroring `ArrowBinaryHelper::Prepare`, because `ArrowBinaryHelper` rolls a chunk at `kBinaryMemoryLimit` and `ReserveData()` would fail its overflow check on a request past it. ## Anything else? No paimon public API (`include/paimon/`), storage format, or protocol change. The only API touched is paimon's own vendored Arrow patch (`cmake_modules/arrow.diff`): the internal `parquet::arrow::ColumnReader::ResetLeaf` signature and the new `parquet::RecordReader::ReserveValueBytes` virtual, neither of which is upstream Arrow's public API. The reservation is a capacity hint, so output is unchanged and the existing `page_filtered_row_group_reader_test.cpp` cases already cover the touched paths — notably `StringColumnMultiBatchStreaming`, a multi-batch BYTE_ARRAY selective read with per-value assertions, plus the nested struct/list/map cases that thread `ResetLeaf` through `StructReader` and `ListReader`. ## Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
