VietCT04 opened a new issue, #363: URL: https://github.com/apache/paimon-cpp/issues/363
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation ### Problem `FieldCollectAgg` and `FieldMergeMapAgg` currently use linear scans with `FieldAggregateUtils::Equals` for distinct-value and key lookup. For large arrays/maps, repeatedly scanning previously collected values causes quadratic behavior. For example: - `FieldCollectAgg` scans previously collected elements to detect duplicates. - `FieldMergeMapAgg` scans existing keys through `FindKey` before overwriting or inserting an entry. The implementation already contains a TODO noting that these lookups should use type-aware hashing for hashable types, similar to the Java implementation. ### Solution ### Proposed improvement Add a hash-backed lookup path for types with well-defined semantic hashing, while preserving the existing aggregation behavior. The optimized path should: - process both accumulator and input values; - preserve first-seen order for `FieldCollectAgg`; - preserve key order and input-overwrite semantics for `FieldMergeMapAgg`; - hash STRING/BINARY by content; - preserve FLOAT/DOUBLE semantics: - all NaNs compare equal; - `-0.0` and `+0.0` remain distinct; - support TIMESTAMP consistently with its equality semantics; - retain the existing linear fallback for DECIMAL and constructed types; - retain linear lookup for small cardinalities where hash-table construction overhead is higher. `FieldNestedUpdateAgg` can be considered separately as follow-up work. ### Performance Focused benchmarks using the production aggregators show that the quadratic lookup becomes significant once cardinality grows. At 192 elements/entries: | Workload | Current | Hash lookup | Speedup | | --- | ---: | ---: | ---: | | Collect INT64 | 127.5 us | 10.1 us | 12.6x | | Collect STRING | 101.3 us | 11.9 us | 8.5x | | MergeMap INT64 | 73.5 us | 15.8 us | 4.7x | | MergeMap STRING | 96.6 us | 18.6 us | 5.2x | At 8192 elements/entries: | Workload | Current | Hash lookup | Speedup | | --- | ---: | ---: | ---: | | Collect INT64 | 109169.3 us | 375.3 us | 290.9x | | Collect STRING | 180821.9 us | 646.0 us | 279.9x | | MergeMap INT64 | 115265.6 us | 664.5 us | 173.5x | | MergeMap STRING | 181807.6 us | 903.6 us | 201.2x | Small-cardinality performance remains approximately unchanged by retaining the existing linear path below the hash threshold. ### Related PR #349 ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
