PyRSA opened a new pull request, #11033: URL: https://github.com/apache/paimon/pull/11033
### Purpose Add `count-limit` support to Python `FieldCollectAgg` and optimize distinct deduplication performance. ### Changes - Support positive and negative `fields.<field>.count-limit` values to limit the number of collected elements. - Preserve element ordering and existing aggregation semantics for distinct and non-distinct modes. - Use a Set for distinct deduplication of hashable atomic element types, reducing average lookup complexity from O(n²) to O(n). - Keep List-based deduplication for complex types such as ARRAY, MAP, and ROW to preserve equality semantics. - Avoid creating additional Set instances when `distinct` is disabled. ### Tests - `python -m pytest pypaimon/tests/test_aggregation_e2e.py -q` - `python -m pytest pypaimon/tests/test_aggregation_merge_function.py -q` - `python -m pytest pypaimon/tests/test_field_aggregators.py -q` - `git diff --check` ### References - Related Java optimization: #10607 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
