lucasfang opened a new pull request, #336: URL: https://github.com/apache/paimon-cpp/pull/336
<!-- PR title: perf(read): skip the payload Concatenate when late materialization matches a single run --> ### Purpose Linked issue: close #335 The payload pass of late materialization compacts every batch down to the rows that passed the probe pass: `LateMaterializingFileBatchReader::ReadPayloadBatch` turns the matched bitmap into one zero-copy slice per contiguous run (`ReaderUtils::GenerateFilteredArrayVector`), then merges the runs with `arrow::Concatenate`. When the matched rows form a single run there is nothing to merge, so `Concatenate` only copies the payload bytes a second time to move the slice back to offset zero — and that rebase is redundant, because `AssembleFullBatch` already passes every column it takes through `ArrowUtils::NormalizeArrayOffsets`. A single run is the common shape here: the payload pass re-reads with the matched rows as its selection, so a precisely filtering reader returns fully matched batches, and a range predicate over clustered data keeps whole batches at a time. This PR hands a lone slice straight to `AssembleFullBatch` and keeps `Concatenate` for two runs or more. Neither half of the single-run case is worse than before: a run starting at the batch head reaches the output with no copy at all (`NeedsNormalization` is false, so normalization is a no-op), and a run starting mid-batch is still copied exactly once, by the per-column normalization instead of by `Concatenate`. The output is unchanged — the assembled batch is still built from normalized, equal-length columns, so the exported `ArrowArray` keeps the offset-zero, children-match-parent shape consumers rely on. Known trade-off: where the run stays zero-copy it keeps the inner batch's buffers alive until the consumer releases the assembled batch. That retains more than the matched rows only when the run starts at the head without reaching the end of the batch, and stays bounded by the read batch size times the prefetch queue depth. ### Tests New `LateMaterializingFileBatchReaderTest.DictionaryPayloadColumnKeepsEncoding`: a dictionary-encoded payload column (`dictionary<int32, utf8>`, built with `StringDictionary32Builder` so the indices match the declared type) read under `k >= 5` with a fixed batch size, which reaches both shapes of a lone run — one at batch offset 0 that passes through untouched, one at batch offset 2 that the per-column normalization copies through `CopyToZeroOffset`, the one type whose normalization is a full copy rather than a buffer rebase. The test asserts the output stays dictionary-encoded and decodes to the expected values. Row-level correctness of the single-run and multi-run paths is already covered by the existing `ContiguousSubsetAcrossBatches`, `ScatteredAlternatingMatch`, `NestedPayloadColumn`, and `WorksAsInnerOfPrefetchReader` cases. Verified in a Debug build (`-j 96`): - `paimon-common-test --gtest_filter='LateMaterializingFileBatchReaderTest.*' --gtest_repeat=20`: 17 tests passed on every repetition (the mock reader randomizes batch sizes elsewhere in the suite, so repetition covers varying run shapes). - `paimon-common-test`: 1620 of 1622 tests from 185 suites passed, 2 skipped (pre-existing skips in `prefetch_file_batch_reader_impl_test.cpp`). - `paimon-core-test`: 2073 tests from 233 suites passed. - `pre-commit run --files <the two changed files>` and `git diff --check`: clean. ### API and Format No. No header under `include/paimon/` is touched, and neither the storage format nor the protocol changes. The reader's externally observable output is byte-for-byte the same batch. ### Documentation No. This is an internal read-path optimization with no user-visible API, configuration, or behavior change. ### Generative AI tooling Generated-by: Qoder -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
