LuciferYang opened a new pull request, #1354:
URL: https://github.com/apache/paimon-rust/pull/1354

   ### Purpose
   
   Linked issue: close #1353
   
   With `fields.<col>.nested-key`, `nested_update` and `nested_partial_update` 
found the stored row for each incoming nested row by comparing its key columns 
with every stored row. This indexes the stored rows by the BinaryRow of their 
key columns, which is what Java `FieldNestedUpdateAgg` keys its map by.
   
   ### Brief change log
   
   - `NestedAgg` keeps a map from the serialized key to the row's position in 
`rows`, built on the first lookup. Positions of pushed rows are added to it, 
and it is dropped whenever `rows` is replaced or rows are removed (reset, a 
DELETE value, raw normalization, retract).
   - Key types that `extract_datum_from_array` cannot serialize (ARRAY, MAP, 
MULTISET, ROW, VECTOR, BLOB) keep the scan. So does a key value that fails to 
serialize, such as an invalid VARIANT payload; the scan is used until the next 
`reset()`.
   
   Writing one row whose ARRAY holds 10,000 distinct nested keys (release 
build, in-memory FileIO): 4.8 s before, 6 to 8 ms after. 50,000 primary keys 
with 3 nested rows each: 163 to 179 ms before, 153 to 166 ms after.
   
   ### Tests
   
   - `many_keys_keep_their_highest_sequence_version`: three versions of 2,000 
keys arrive out of sequence order and each key keeps its highest-sequence 
version.
   - `keys_are_found_after_retract_and_delete`: lookups after a retract has 
shifted positions, and a DELETE value with a repeated key deduplicated by the 
next merge.
   - `reset_drops_positions_of_the_previous_primary_key`: the next primary key 
does not use positions from the previous one.
   - `unserializable_variant_keys_fall_back_to_scanning`: a VARIANT key with an 
invalid payload still merges by scanning, and `reset()` turns the index back on.
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   
   Generated-by: Claude Opus
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to