sundapeng opened a new pull request, #866: URL: https://github.com/apache/paimon-rust/pull/866
### Purpose Linked issue: close #865 `UPDATE`, `DELETE` and `MERGE INTO` on a Format Table wrote Paimon data files into the table and left them there, so the table read more rows than it held. On a 3-row table, `UPDATE` and `MERGE INTO` returned 33 rows afterwards and `DELETE` returned 8, with and without `metastore.partitioned-table`. The rewritten files are visible to every engine that reads the table. The three statements take the copy-on-write path. `CopyOnWriteMergeWriter` rewrites the rows it keeps into `<partition>/bucket-0/`, then the commit tries to publish them. A Format Table has no snapshot, so the commit fails with `Delete conflict: file ... does not exist in the current snapshot`, but the files stay, and a Format Table scan lists a partition recursively where `bucket-0` is not a hidden name. `INSERT` is already refused by `FormatWriteBuilder::new_write`; the copy-on-write path never asks the write builder for a writer, so it went around that check. This refuses the rewrite in `CopyOnWriteMergeWriter::new`, which is the one place the three statements pass through and which runs before any file is written. Java refuses the same statements: `PaimonFormatTable` does not implement Spark's row-level operations. ### Brief change log - `CopyOnWriteMergeWriter::new` returns `Unsupported` for a Format Table, next to its existing checks for primary keys and data evolution. The message names the table and the three statements. - `docs/src/sql.md` says under Format Table Partitions that `INSERT`, `UPDATE`, `DELETE` and `MERGE INTO` are all refused. ### Tests `crates/integrations/datafusion/tests/rest_format_table_dml.rs`, both cases running all three statements: - a Format Table whose partitions a REST catalog manages; - a Format Table that discovers its partitions from directories, in a filesystem catalog. Each case asserts the error, that the set of files under the table is byte-for-byte the one it started with, and that a following `SELECT` still returns the original rows. With the new check removed, both fail: the statements report success and the files appear. Commands run on the top commit: ``` cargo fmt --all -- --check cargo clippy --locked -p paimon -p paimon-datafusion --all-targets --features paimon/fulltext,paimon/vortex -- -D warnings cargo test -p paimon-datafusion --test rest_format_table_dml # 2 passed cargo test -p paimon-datafusion --test delete_tests # 25 passed cargo test -p paimon-datafusion --test update_tests # 20 passed cargo test -p paimon-datafusion --test merge_into_tests # 29 passed cargo test -p paimon-datafusion --test append_merge_into # 34 passed cargo test -p paimon --lib cow_writer # 9 passed ``` ### API and Format `CopyOnWriteMergeWriter::new` now returns an error for a Format Table, where it used to build a writer. No storage format change. ### Documentation `docs/src/sql.md`, one paragraph under "Format Table Partitions". Related: #863 fixes TRUNCATE TABLE on a Format Table, which had the same shape (the statement reported success while doing nothing). The two do not overlap in code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
