PyRSA opened a new pull request, #11033:
URL: https://github.com/apache/paimon/pull/11033

   ### Purpose
   
   Add `count-limit` support to Python `FieldCollectAgg` and optimize distinct 
deduplication performance.
   
   ### Changes
   
   - Support positive and negative `fields.<field>.count-limit` values to limit 
the number of collected elements.
   - Preserve element ordering and existing aggregation semantics for distinct 
and non-distinct modes.
   - Use a Set for distinct deduplication of hashable atomic element types, 
reducing average lookup complexity from O(n²) to O(n).
   - Keep List-based deduplication for complex types such as ARRAY, MAP, and 
ROW to preserve equality semantics.
   - Avoid creating additional Set instances when `distinct` is disabled.
   
   ### Tests
   
   - `python -m pytest pypaimon/tests/test_aggregation_e2e.py -q`
   - `python -m pytest pypaimon/tests/test_aggregation_merge_function.py -q`
   - `python -m pytest pypaimon/tests/test_field_aggregators.py -q`
   - `git diff --check`
   
   ### References
   
   - Related Java optimization: #10607


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to