PDGGK opened a new pull request, #1319:
URL: https://github.com/apache/paimon-rust/pull/1319

   ### Purpose
   
   Linked issue: close #1318
   
   On a fixed-bucket append table (`'bucket' = 'N'`, `'bucket-key' = '...'`), 
`UPDATE` or a matched `MERGE INTO` update that changes a bucket-key column 
leaves the row in the bucket of its old value. Scans prune buckets by the 
bucket-key hash, so any query filtering on the new value skips the row, while a 
full scan still returns it.
   
   Before this change, on main:
   
   ```
   >>> UPDATE db.orders SET customer = 'erin' WHERE id = 1
   >>> SELECT * FROM db.orders ORDER BY id
   +----+----------+--------+
   | id | customer | amount |
   +----+----------+--------+
   | 1  | erin     | 10     |
   | 2  | bob      | 20     |
   | 3  | carol    | 30     |
   | 4  | dave     | 40     |
   +----+----------+--------+
   >>> SELECT * FROM db.orders WHERE customer = 'erin'
   ++
   ++
   ```
   
   After:
   
   ```
   >>> SELECT * FROM db.orders WHERE customer = 'erin'
   +----+----------+--------+
   | id | customer | amount |
   +----+----------+--------+
   | 1  | erin     | 10     |
   +----+----------+--------+
   ```
   
   The table is `CREATE TABLE db.orders (id INT, customer STRING, amount INT) 
WITH ('bucket' = '4', 'bucket-key' = 'customer')` with rows `(1, 'alice', 10), 
(2, 'bob', 20), (3, 'carol', 30), (4, 'dave', 40)`. Before the fix, 
`orders$files` shows the rewritten file still in `bucket-2` (where `alice` 
hashes), while a new `INSERT` of `'erin'` goes to `bucket-1`.
   
   ### Brief change log
   
   - `CopyOnWriteMergeWriter::prepare_commit`: when the updated columns include 
a `bucket-key` column and the table has a fixed bucket count above 1, compute 
the bucket of each rewritten row with the table's bucket function and bucket 
count, as `TableWrite` does for inserts, and write each bucket's rows into that 
bucket. The deleted file stays in the commit message of its original bucket; 
rows that moved are added in the message of their new bucket.
   - Updates that do not touch a bucket key, and tables without a fixed bucket 
count, keep the existing path unchanged.
   
   This matches Java: Spark's `UPDATE` on an append table writes the updated 
rows through `PaimonSparkWriter`, which assigns `HASH_FIXED` buckets from the 
bucket keys of each row.
   
   Assumptions worth checking:
   
   - The new buckets use the table's current `bucket` option. `TableWrite` uses 
it too, and the commit records it as the files' total bucket count. A partition 
still holding files with an older bucket count (rescaled without overwrite) 
gets the rewritten rows hashed with the current count.
   - When a bucket key is updated, every surviving row of a touched file is 
hashed again. Rows whose key did not change hash to the bucket they were 
already in.
   
   ### Tests
   
   - `update_tests::test_update_bucket_key_moves_row_to_new_bucket`: UPDATE of 
the bucket key, then a filter on the new value returns the row.
   - `append_merge_into::test_update_bucket_key_moves_row_to_new_bucket`: the 
same through `MERGE INTO`, with one updated row moving out of a file that also 
keeps an unchanged row.
   - Without the change in `cow_writer.rs`, both tests fail with the filtered 
query returning `[]` instead of the updated rows.
   - `RUST_MIN_STACK=16777216 cargo test -p paimon-datafusion --test 
update_tests --test append_merge_into --test delete_tests --test 
merge_into_tests` (21, 35, 25 and 29 tests) pass.
   - `cargo fmt --all -- --check` and `cargo clippy --locked -p paimon -p 
paimon-datafusion --all-targets -- -D warnings` pass.
   - I also checked by hand a partitioned bucketed table, an update to `NULL`, 
`'bucket-function.type' = 'mod'`, and a MERGE mixing matched DELETE, matched 
UPDATE and NOT MATCHED INSERT.
   
   The full `paimon` unit-test suite and the vortex feature are left to CI.
   
   ### API and Format
   
   No API or storage format change.
   
   ### Documentation
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to