zhf999 opened a new pull request, #188:
URL: https://github.com/apache/paimon-cpp/pull/188
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
`BatchReader` requires every returned array, including its nested children,
to have a zero offset.
`ArrowUtils::NormalizeRecordBatchOffsets` enforced that by calling
`arrow::Concatenate({column})`,
which has no single-array fast path and therefore deep copies every buffer
of the column.
This is on the hot path of page-filtered Parquet reads.
`PageFilteredRowGroupReader` materializes a
filtered row group as an `arrow::Table` and streams it through
`arrow::TableBatchReader`, which
returns `chunk->Slice(offset, chunksize)` for every batch after the first.
As a result every batch
except the first was deep copied, column by column, adding a full extra copy
of all scanned data.
Since nested columns now also go through this path, `Concatenate`
recursively copied list offsets,
child values and per-level validity bitmaps as well.
Normalizing a slice does not require copying anything: a contiguous slice
can be rebased to a zero
offset by slicing the underlying buffers.
- Fixed width: slice the value buffer.
- Boolean: rebase the value bitmap, which only needs a bit shift when the
offset is not byte aligned.
- String / binary / large binary: rewrite the offsets so the first one is
zero, and slice the value
buffer to the referenced range.
- List / large list / map: rewrite the offsets, then recursively rebase the
contiguous child range
the offsets span.
- Struct: recursively rebase every child, sliced in step with the parent,
because a struct slice
keeps full length children.
- Validity bitmaps: sliced when byte aligned, and dropped entirely when the
array has no nulls.
Layouts that cannot be rebased this way, currently dictionary and any type
not listed above, keep
the previous behavior and fall back to `arrow::Concatenate`.
### Tests
Added to `ArrowUtilsTest`:
- `TestNormalizeRecordBatchOffsetsCoversSupportedTypes`
16 types crossed with 8 slices each. Every combination asserts
`ValidateFull()`, equality with
the original batch, and recursively that all offsets are zero.
- `TestNormalizeRecordBatchOffsetsSharesValueBuffers`
Over the same 16 types, asserts that the buffer holding the values of each
layout, addressed by a
path of child indices plus a buffer index, is contained in the address
range of the corresponding
source buffer. This pins the optimization: an implementation that copies
would allocate outside the source range and fail here.
- `TestNormalizeRecordBatchOffsetsFallsBackForDictionary`
Checks that the copying fallback still satisfies the zero offset contract,
and asserts that its
index buffer is *not* contained in the source range. That negative case is
what makes the
containment assertions above meaningful rather than trivially true.
### Perf Test
#### Datasets
Multiple performance testing were conducted on 1 dataset.
|Dataset| File size | num_rows | num_columns | num_rowgroups | page length
(in rows)|
| --- | --- | --- | --- | --- | --- |
| Dataset#1 | 6.7GB | 2M | 1,176 | 39 | 1024 |
#### Test Methodology
To validate bitmap behavior, we organize tests into six groups based on
bitmap distribution patterns. Each group contains five distinct datasets, and
the final result reported is the average of these five runs.
#### Test Cases
1. Matching records are contiguous and only one row is queried.
2. Matching records are contiguous and 10,000 rows are queried.
3. Matching records are scattered in 5 segments, each segment containing 1
row.
4. Matching records are scattered in 5 segments, each segment containing
100 rows.
5. Matching records are scattered in 100 segments, each segment containing
1 row.
6. Skip 100 rows, read 100 rows, skip 100 rows, read 100 rows util EOF.
7. Skip 10000 rows, read 10000 rows, skip 10000 rows, read 10000 rows util
EOF.
8. Scattered bitmap, skip 1 row, read 1 row, skip 1 row, read 1 row until
EOF (i.e. `1010101010...`).
9. Scattered bitmap, skip 32 rows, read 1 row, skip 32 rows, skip 1 row
until EOF.
10. Scattered bitmap, skip 99 rows, read 1 row, skip 99 rows, skip 1 row
until EOF.
11. Scattered bitmap, skip 9999 rows , read 1 row, skip 9999 rows, skip 1
row until EOF.
#### Test Results
On dataset1
| Branch | Case 1 | Case 2 | Case3 | Case 4 | Case 5 | Case 6 | Case 7 |
Case 8 | Case 9 | Case 10 | Case 11| Case 12|
| --- | --- | --- | --- | --- | --- | --- | --- | ---| --- | --- | --- | ---
|
| Before | 0.38s | 0.84s | 1.47s |1.49s |16.07s | 115.14s | 82.03s | 140.68s
| 87.00s | 74.39s | 21.84s |
| After | 0.34s | 0.77s | 1.31s | 1.33s | 15.27s | 100.06s | 71.48s |
122.77s | 81.91s | 67.82s | 20.34s |
### API and Format
No change.
### Documentation
No change.
### Generative AI tooling
Generated-by: GLM 5.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
