sundapeng opened a new issue, #865: URL: https://github.com/apache/paimon-rust/issues/865
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Please describe the bug 🐞 `UPDATE`, `DELETE` and `MERGE INTO` on a Format Table write Paimon data files into the table, and the table then reads more rows than it holds. The three statements go through the copy-on-write path. `CopyOnWriteMergeWriter` rewrites the rows it keeps into `<partition>/bucket-0/` under the table directory, then the commit tries to publish them. A Format Table has no snapshot, so the commit fails, but the rewritten files stay where they are. A Format Table scan lists a partition directory recursively and `bucket-0` is not a hidden name, so those files are read as table data. What we measured on a 3-row Format Table (one row matched by the statement): | statement | result | rows after | |---|---|---| | `UPDATE t SET ... WHERE ...` | error `CoW UPDATE failed after 5 retries due to concurrent compaction ... Delete conflict: file ... does not exist in the current snapshot` | 33 | | `MERGE INTO t ... WHEN MATCHED THEN UPDATE SET ...` | same error for MERGE INTO | 33 | | `DELETE FROM t WHERE ...` | same error for DELETE | 8 | It happens with and without `metastore.partitioned-table`, and the files are visible to every engine that reads the table, not only to this client. `INSERT` into a Format Table is already refused by `FormatWriteBuilder::new_write`, but the copy-on-write path never asks the write builder for a writer, so it goes around that check. ### Solution Refuse the rewrite for a Format Table in `CopyOnWriteMergeWriter::new`, next to the existing checks for primary keys and data evolution. It is the single place the three statements pass through, and it runs before any file is written, so nothing is left behind. Java refuses the same statements: `PaimonFormatTable` does not implement Spark's row-level operations. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
