zghong opened a new pull request, #67507:
URL: https://github.com/apache/doris/pull/67507
### What problem does this PR solve?
Issue Number: close #67505
Problem Summary:
#### What does this PR optimize?
For `DUP_KEYS` memtables, each input row was previously represented by a
`shared_ptr<RowInBlock>`. However, duplicate-key tables do not require the
per-row aggregation state stored in `RowInBlock`; they only need the source row
position for sorting.
This PR introduces a compact representation for `DUP_KEYS`:
- Store row positions in a contiguous `uint32_t` vector instead of
allocating a `shared_ptr<RowInBlock>` for every row.
- Keep `_row_in_blocks` only for `UNIQUE_KEYS` and `AGG_KEYS`, which still
need per-row aggregation state.
- Generalize the sorting logic so that it supports both compact row
positions and `RowInBlock` objects.
- Store allocated LSNs in a sidecar vector and apply the same permutation as
the sorted rows, preserving row-to-LSN correspondence.
- Preserve the existing ordering semantics for duplicate keys, where newer
rows are ordered before older rows with the same key.
- Skip sorting entirely for zero-key duplicate tables and retain their
physical insertion order.
#### Benchmark
The benchmark measures the row-index construction path directly affected by
this change. It was run with a Release build and five repetitions:
```shell
./be/output/lib/benchmark_test \
--benchmark_filter='BM_DuplicateKeyRowIndex.*' \
--benchmark_repetitions=5 \
--benchmark_report_aggregates_only=true
```
CPU mean results:
| Rows | Previous `shared_ptr<RowInBlock>` | Compact `uint32_t index` |
Speedup | Time reduction |
| --------- | --------------------------------- | ------------------------ |
------- | -------------- |
| 4,096 | 132 us | 0.370 us |
356.76x | 99.72% |
| 65,536 | 3,186 us | 9.50 us |
335.37x | 99.70% |
| 1,048,576 | 49,139 us | 305 us |
161.11x | 99.38% |
Estimated row-metadata footprint:
| Representation | Estimated bytes per row
|
| ------------------------------------------------ | -----------------------
|
| Previous `shared_ptr<RowInBlock>` representation | 56 bytes
|
| Compact row position without LSN | 4 bytes
|
| Compact row position with LSN | 12 bytes
|
This reduces the estimated row-metadata footprint by:
- 92.86% without LSNs (56 -> 4 bytes/row), approximately 14x more compact.
- 78.57% with LSNs (56 -> 12 bytes/row), approximately 4.67x more compact.
The previous estimate does not include the allocation/control-block overhead
of make_shared, so the actual memory reduction may be larger.
Note that this is a focused microbenchmark for row-index construction and
does not represent the end-to-end performance improvement of a complete load
job.
### Release note
Reduce duplicate-key load memory use and row-index construction overhead.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
