PDGGK opened a new issue, #1318:
URL: https://github.com/apache/paimon-rust/issues/1318
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Please describe the bug 🐞
On a fixed-bucket append table, `UPDATE` or `MERGE INTO ... WHEN MATCHED
THEN UPDATE` that changes a `bucket-key` column leaves the row in the bucket of
its old value. A query that filters on the new value reads only the new value's
bucket, so it does not return the row. A full scan still shows it.
Reproduce with `SQLContext` on a filesystem catalog (main at 5bb42a1):
```sql
CREATE TABLE db.orders (id INT, customer STRING, amount INT)
WITH ('bucket' = '4', 'bucket-key' = 'customer');
INSERT INTO db.orders VALUES (1, 'alice', 10), (2, 'bob', 20), (3, 'carol',
30), (4, 'dave', 40);
UPDATE db.orders SET customer = 'erin' WHERE id = 1;
SELECT * FROM db.orders ORDER BY id; -- (1, erin, 10) is there
SELECT * FROM db.orders WHERE customer = 'erin'; -- returns no rows
```
```
>>> SELECT * FROM db.orders ORDER BY id
+----+----------+--------+
| id | customer | amount |
+----+----------+--------+
| 1 | erin | 10 |
| 2 | bob | 20 |
| 3 | carol | 30 |
| 4 | dave | 40 |
+----+----------+--------+
>>> SELECT * FROM db.orders WHERE customer = 'erin'
++
++
```
`orders$files` shows why: the rewritten file is still in `bucket-2`, where
`alice` hashes. A later `INSERT INTO db.orders VALUES (5, 'erin', 50)` writes
to `bucket-1`, and the same query then returns only id 5:
```
>>> SELECT * FROM db.orders WHERE customer = 'erin'
+----+----------+--------+
| id | customer | amount |
+----+----------+--------+
| 5 | erin | 50 |
+----+----------+--------+
```
`MERGE INTO` has the same problem:
```sql
MERGE INTO db.orders AS t USING db.fixes AS s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET customer = s.customer; -- fixes = (1,
'erin'), (3, 'frank')
SELECT * FROM db.orders WHERE customer IN ('erin', 'frank'); -- returns no
rows
```
Java readers prune fixed-bucket append scans by bucket key too
(`AppendOnlyFileStore` uses `BucketSelectConverter`), so Spark and Flink
queries on the table miss these rows as well.
Both statements go through `CopyOnWriteMergeWriter::prepare_commit`, which
writes every rewritten file back into the original file's bucket. Partition
columns are rejected as update targets, but bucket-key columns are not, and
their rows are never re-bucketed.
### Solution
When the updated columns include a bucket-key column, compute the bucket of
each rewritten row with the table's bucket function and bucket count, the same
way `TableWrite` does for inserts, and write each bucket's rows into that
bucket. Java's Spark writer does the same for the rows produced by `UPDATE` on
append tables (`PaimonSparkWriter`, `HASH_FIXED`).
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]