SteNicholas opened a new pull request, #1392: URL: https://github.com/apache/paimon-rust/pull/1392
### Purpose Linked issue: close #1379 DataFusion SQL rejected DELETE on every primary-key table, although the primary-key writer already accepts DELETE records. This adds SQL DELETE for primary-key tables using `merge-engine=deduplicate`, following Spark's `DeleteFromPaimonTableCommand#performPrimaryKeyDelete`. ### Brief change log - Route DELETE on primary-key tables to `execute_primary_key_delete`, which scans the logical rows matching the `WHERE` clause, marks them with a DELETE `_VALUE_KIND` column and writes them through `TableWrite`, so the tombstones keep the primary-key, partition and sequence-field values. Commit them through the existing commit path and return the matched-row count; a delete without matches publishes no snapshot. - Plan the scan against one captured snapshot: the `FROM` source must be the target table itself, not a view or a time-traveled copy, and self-references in the predicate, including subqueries, read the same snapshot. Concurrent upserts follow the normal primary-key merge rules, including `sequence.field` ordering. - Write all tombstones before committing, so commit retries reuse the prepared files instead of rescanning. Close the writer when the scan or write fails to discard unprepared output; when the commit fails, keep the prepared files and report the table and commit user, since the snapshot may have been published. - Reject other merge engines, `ignore-delete` including its fallback keys, `rowkind.field`, `query-auth` and DELETE clauses beyond a single target with `WHERE` before scanning or writing. ### Tests - Unit tests in `crates/integrations/datafusion/src/delete/tests.rs`: snapshot pinning of the target and predicate self-references, replaced targets, commit retries without rescanning, uncertain commits, tombstone write failure cleanup and rejection of unsupported options before scanning. - Integration tests in `crates/integrations/datafusion/tests/pk_delete_tests.rs`: predicates, NULLs, no-match and never-written tables, subqueries, partitions, composite keys, dynamic buckets, quoted names, sequence fields and sort orders, concurrent upserts, scan failure cleanup, existing target guards and rejected merge engines, options and syntax. - Replace `test_delete_rejects_primary_key_table` and `test_pk_aggregation_rejects_delete`. ### API and Format No API or storage format change. ### Documentation Update `docs/src/sql.md` with the supported primary-key tables, snapshot, concurrency and retry semantics, and the unsupported options. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
