LuciferYang opened a new pull request, #1354: URL: https://github.com/apache/paimon-rust/pull/1354
### Purpose Linked issue: close #1353 With `fields.<col>.nested-key`, `nested_update` and `nested_partial_update` found the stored row for each incoming nested row by comparing its key columns with every stored row. This indexes the stored rows by the BinaryRow of their key columns, which is what Java `FieldNestedUpdateAgg` keys its map by. ### Brief change log - `NestedAgg` keeps a map from the serialized key to the row's position in `rows`, built on the first lookup. Positions of pushed rows are added to it, and it is dropped whenever `rows` is replaced or rows are removed (reset, a DELETE value, raw normalization, retract). - Key types that `extract_datum_from_array` cannot serialize (ARRAY, MAP, MULTISET, ROW, VECTOR, BLOB) keep the scan. So does a key value that fails to serialize, such as an invalid VARIANT payload; the scan is used until the next `reset()`. Writing one row whose ARRAY holds 10,000 distinct nested keys (release build, in-memory FileIO): 4.8 s before, 6 to 8 ms after. 50,000 primary keys with 3 nested rows each: 163 to 179 ms before, 153 to 166 ms after. ### Tests - `many_keys_keep_their_highest_sequence_version`: three versions of 2,000 keys arrive out of sequence order and each key keeps its highest-sequence version. - `keys_are_found_after_retract_and_delete`: lookups after a retract has shifted positions, and a DELETE value with a repeated key deduplicated by the next merge. - `reset_drops_positions_of_the_previous_primary_key`: the next primary key does not use positions from the previous one. - `unserializable_variant_keys_fall_back_to_scanning`: a VARIANT key with an invalid payload still merges by scanning, and `reset()` turns the index back on. ### API and Format None. ### Documentation None. Generated-by: Claude Opus -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
