LuciferYang opened a new pull request, #1352: URL: https://github.com/apache/paimon-rust/pull/1352
### Purpose Linked issue: close #1351 `collect` ignored `fields.<col>.count-limit`, merged `aggReversed` in a different order from Java master, and checked `distinct` membership with a linear scan. This ports the merge logic of Java `FieldCollectAgg` as of apache/paimon#10607. ### Brief change log - The merge is a port of the private `agg(accumulator, accumulatorHasDuplicate, input, inputHasDuplicate)` of `FieldCollectAgg`. `agg` merges `(acc, false, input, true)` and `aggReversed` merges `(input, true, acc, false)`. A negative limit collects both operands backwards, input first, and reverses the result. For element types where Java uses the equaliser, `checkDuplicateUntil` decides how far back duplicates are checked. - `fields.<col>.count-limit` is read as a 32-bit integer and defaults to `i32::MAX`. A DELETE value is still stored as it is; the next merge applies the limit and the deduplication, like Java. - The ignore-retract wrapper passes `agg_reversed` to the wrapped function, as `FieldIgnoreRetractAgg` does since apache/paimon#10540. `agg_reversed_via_agg` had no other user and is removed. - Whether an element type uses the equaliser follows Java `needsEqualiser`, based on the Paimon type: ARRAY, VECTOR, MULTISET, MAP, ROW, BINARY and VARBINARY. Before, the choice followed the Arrow type, which put VARIANT and BLOB on the equaliser path and VECTOR on the set path. - With `distinct=true`, element types that Java collects into a `LinkedHashSet` are compared by their arrow-row encoded key in a hash set. Writing 20,000 single-element rows for one key with `distinct=true` (release build, in-memory FileIO): 8.7 s before, 7 ms after. I also replayed about 210k random `agg` / `aggReversed` / DELETE / retract steps whose results came from Java master's `FieldCollectAgg`, covering INT, ROW<INT> and BYTES elements, `distinct` on and off, limits from -5 to 5 plus the default, with and without ignore-retract. Rust matched every step. The same replay finds differences on main. ### Tests - `count_limit_matches_java_for_scalar_and_row_elements`: the cases of Java `testFieldCollectAggWithCountLimit`, for one row, two rows forward, and two rows reversed. - `agg_reversed_puts_older_input_first` and `ignore_retract_reversed_puts_older_input_first`: the older input comes first, with and without the ignore-retract wrapper. - `ignore_retract_reversed_deduplicates_older_binary_input`: with the wrapper delegating, an older binary input's own duplicates are removed (the old test expected them to be kept). - `agg_reversed_keeps_duplicates_of_a_binary_delete_value`: with the equaliser, a DELETE value is checked only against elements collected before it, so its own duplicates stay. - `delete_value_is_capped_by_the_next_merge`: a DELETE value keeps all its elements, and the next merge keeps the first or last N. - `delete_value_duplicates_follow_the_java_element_family`: after a DELETE value with a repeated element, VECTOR keeps both copies and VARIANT and BLOB keep one. - `count_limit_option_is_read_and_validated`: the option is read from the table options, and non-integer or out-of-range values are rejected. - `distinct_keys_follow_arrow_equality`: nulls, retract followed by agg, NaN and -0.0 behave as with Arrow `==`. ### API and Format No API or storage format change. Tables that set `count-limit` on a `collect` column now get capped arrays when Rust merges them. ### Documentation None. Generated-by: Claude Opus -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
