PDGGK opened a new issue, #1318:
URL: https://github.com/apache/paimon-rust/issues/1318

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar.
   
   ### Please describe the bug 🐞
   
   On a fixed-bucket append table, `UPDATE` or `MERGE INTO ... WHEN MATCHED 
THEN UPDATE` that changes a `bucket-key` column leaves the row in the bucket of 
its old value. A query that filters on the new value reads only the new value's 
bucket, so it does not return the row. A full scan still shows it.
   
   Reproduce with `SQLContext` on a filesystem catalog (main at 5bb42a1):
   
   ```sql
   CREATE TABLE db.orders (id INT, customer STRING, amount INT)
     WITH ('bucket' = '4', 'bucket-key' = 'customer');
   INSERT INTO db.orders VALUES (1, 'alice', 10), (2, 'bob', 20), (3, 'carol', 
30), (4, 'dave', 40);
   UPDATE db.orders SET customer = 'erin' WHERE id = 1;
   
   SELECT * FROM db.orders ORDER BY id;              -- (1, erin, 10) is there
   SELECT * FROM db.orders WHERE customer = 'erin';  -- returns no rows
   ```
   
   ```
   >>> SELECT * FROM db.orders ORDER BY id
   +----+----------+--------+
   | id | customer | amount |
   +----+----------+--------+
   | 1  | erin     | 10     |
   | 2  | bob      | 20     |
   | 3  | carol    | 30     |
   | 4  | dave     | 40     |
   +----+----------+--------+
   >>> SELECT * FROM db.orders WHERE customer = 'erin'
   ++
   ++
   ```
   
   `orders$files` shows why: the rewritten file is still in `bucket-2`, where 
`alice` hashes. A later `INSERT INTO db.orders VALUES (5, 'erin', 50)` writes 
to `bucket-1`, and the same query then returns only id 5:
   
   ```
   >>> SELECT * FROM db.orders WHERE customer = 'erin'
   +----+----------+--------+
   | id | customer | amount |
   +----+----------+--------+
   | 5  | erin     | 50     |
   +----+----------+--------+
   ```
   
   `MERGE INTO` has the same problem:
   
   ```sql
   MERGE INTO db.orders AS t USING db.fixes AS s ON t.id = s.id
   WHEN MATCHED THEN UPDATE SET customer = s.customer;   -- fixes = (1, 
'erin'), (3, 'frank')
   
   SELECT * FROM db.orders WHERE customer IN ('erin', 'frank');  -- returns no 
rows
   ```
   
   Java readers prune fixed-bucket append scans by bucket key too 
(`AppendOnlyFileStore` uses `BucketSelectConverter`), so Spark and Flink 
queries on the table miss these rows as well.
   
   Both statements go through `CopyOnWriteMergeWriter::prepare_commit`, which 
writes every rewritten file back into the original file's bucket. Partition 
columns are rejected as update targets, but bucket-key columns are not, and 
their rows are never re-bucketed.
   
   ### Solution
   
   When the updated columns include a bucket-key column, compute the bucket of 
each rewritten row with the table's bucket function and bucket count, the same 
way `TableWrite` does for inserts, and write each bucket's rows into that 
bucket. Java's Spark writer does the same for the rows produced by `UPDATE` on 
append tables (`PaimonSparkWriter`, `HASH_FIXED`).
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to