HappenLee opened a new pull request, #68350:
URL: https://github.com/apache/doris/pull/68350
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
The generic numeric DISTINCT combinator (for example, `multi_distinct_sum`
over integer columns) copies the entire source hash set before merging it. When
merging serialized partial aggregates, the default implementation first builds
a temporary hash set and then copies that set again. This adds allocation,
hashing, and traversal work in the aggregation merge stage.
Insert existing source keys directly without modifying the source. Override
`deserialize_and_merge` so numeric keys are read directly into the destination
set. Reserve the known number of incoming keys only when the destination is
empty: reserving the sum of source and destination sizes can unnecessarily grow
heavily overlapping sets. Generic/StringRef states keep their existing
deserialize-then-merge path, including copying key bytes into the destination
arena. Serialization and NULL semantics are unchanged.
An earlier standalone merge-kernel microbenchmark with 131,072 distinct
int64 keys (Clang 21.1.8, AVX2, nine-run medians) measured serialized merging
into an empty destination at 9.725 ms before versus 0.944 ms with direct
deserialization, and full overlap at 6.233 ms versus 0.250 ms. This uses phmap
with a counting standard allocator, not a full Doris SQL query; these numbers
are not end-to-end latency claims. The dedicated `multi_distinct_count`
implementation already has its own merge optimization and is outside this
change.
### Release note
Reduce temporary hash-set allocations and CPU work when merging generic
numeric DISTINCT aggregate states.
### Check List (For Author)
- Test:
- [x] Unit Test: 16 ASAN tests passed for all five integer widths,
source preservation, repeated/empty merges, unused numeric scratch sets, both
Nullable implementations, grouped/selected batch dispatch, and generic string
ownership.
- [x] Regression test: one-/two-phase aggregation, grouped/ungrouped
queries, nullable/non-null inputs, empty/all-null input, and AggState merging.
- Behavior changed:
- [x] No. SQL results and serialized state formats are unchanged.
- Does this need documentation?
- [x] No.
Validation:
- `./run-be-ut.sh -j 96 --run
--filter='DistinctNumericMergeTest/*.*:NullImplementations/DistinctMergeDispatchTest.*'`:
16 tests passed.
- Header hygiene and clang-format 16 checks passed.
- `build-support/run-clang-tidy.sh`: no findings on changed code (header
analyzed with a compile command derived from the new test TU).
- `./run-regression-test.sh --run -d query_p0/aggregate -s
test_numeric_distinct_merge -genOut` generated the expected output; rerunning
without `-genOut` passed all 10 result checks and the merge-plan assertion.
- `./build.sh --be --fe -j 96` completed with ASAN BE and FE Checkstyle
enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
