VietCT04 opened a new pull request, #349:
URL: https://github.com/apache/paimon-cpp/pull/349
### Purpose
Linked issue: N/A
Optimize distinct lookup in `FieldCollectAgg` and key lookup in
`FieldMergeMapAgg`.
The current implementation performs linear `FieldAggregateUtils::Equals`
scans while processing accumulator and input values. For large arrays/maps,
this leads to quadratic behavior.
This change adds a semantic hash-backed lookup path for supported types while
preserving the existing C++ aggregation semantics:
- process both accumulator and input, including duplicate accumulators;
- preserve first-seen order in `FieldCollectAgg`;
- preserve key order and input-overwrite semantics in `FieldMergeMapAgg`;
- hash STRING/BINARY by content;
- preserve floating-point semantics where all NaNs compare equal and `-0.0`
remains distinct from `+0.0`;
- hash TIMESTAMP consistently with its equality semantics;
- keep DECIMAL and constructed types on the existing linear fallback;
- retain the linear path for small cardinalities to avoid hash-table
overhead.
`FieldNestedUpdateAgg` is intentionally left unchanged.
A fixed threshold of 192 elements/entries is used for the hash path. This
keeps small inputs close to the current implementation while providing a
clear
benefit once cardinality grows.
Focused benchmarks, using the actual production aggregators and pinned CPU
runs, show the crossover and large-cardinality scaling:
| Workload | N | Baseline | Hash path | Speedup |
| --- | ---: | ---: | ---: | ---: |
| Collect INT64 | 192 | 127.5 us | 10.1 us | 12.6x |
| Collect STRING | 192 | 101.3 us | 11.9 us | 8.5x |
| MergeMap INT64 | 192 | 73.5 us | 15.8 us | 4.7x |
| MergeMap STRING | 192 | 96.6 us | 18.6 us | 5.2x |
| Collect INT64 | 8192 | 109169.3 us | 375.3 us | 290.9x |
| Collect STRING | 8192 | 180821.9 us | 646.0 us | 279.9x |
| MergeMap INT64 | 8192 | 115265.6 us | 664.5 us | 173.5x |
| MergeMap STRING | 8192 | 181807.6 us | 903.6 us | 201.2x |
At `N=96`, performance remains approximately neutral:
| Workload | Baseline | Hash-enabled implementation |
| --- | ---: | ---: |
| Collect INT64 | 18.0 us | 18.2 us |
| Collect STRING | 26.5 us | 23.4 us |
| MergeMap INT64 | 20.6 us | 20.3 us |
| MergeMap STRING | 31.2 us | 32.1 us |
Repeated one-element merges also improve substantially once the accumulator
becomes large, despite rebuilding the temporary hash index on each `Agg`
call:
| Workload | K | Baseline | Hash-enabled implementation | Speedup |
| --- | ---: | ---: | ---: | ---: |
| Collect INT64 | 512 | 81248.9 us | 13183.1 us | 6.2x |
| Collect INT64 | 2048 | 5121360.1 us | 149463.3 us | 34.3x |
| MergeMap INT64 | 512 | 111933.1 us | 18205.9 us | 6.1x |
| MergeMap INT64 | 2048 | 6049633.9 us | 297083.7 us | 20.4x |
At `K=128`, Collect remains approximately neutral (`1499.4 us -> 1557.4 us`).
### Tests
Added and ran focused unit tests covering:
- duplicate accumulator values;
- duplicate input values;
- duplicate accumulator and input map keys;
- accumulator/input overlap;
- first-seen ordering;
- map key ordering and overwrite behavior;
- null handling;
- STRING/BINARY content equality across different backing storage;
- FLOAT/DOUBLE signed zero, infinities, and multiple NaN payloads;
- TIMESTAMP equality/hash behavior;
- DECIMAL fallback;
- constructed-type fallback.
The hash-path tests use cardinalities above the threshold so they exercise
the
optimized implementation directly. DECIMAL and constructed-type tests also
use large cardinalities to verify that they remain on the linear fallback
because of type support rather than the size threshold.
Results:
- `FieldCollectAggTest.*`: 13/13 passed
- `FieldMergeMapAggTest.*`: 5/5 passed
- relevant aggregate/factory tests: 10/10 passed
- `git diff --check` passed
A full integration/dependency rebuild was not run because it would rebuild
the
bundled dependencies. No end-to-end compaction performance claim is made.
### API and Format
No public API, storage format, or protocol changes.
### Documentation
No documentation changes required. This is an internal performance
optimization with unchanged aggregation semantics.
### Generative AI tooling
Generated-by: OpenAI Codex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
