SteNicholas opened a new pull request, #1392:
URL: https://github.com/apache/paimon-rust/pull/1392

   ### Purpose
   
   Linked issue: close #1379
   
   DataFusion SQL rejected DELETE on every primary-key table, although the 
primary-key writer already accepts DELETE records. This adds SQL DELETE for 
primary-key tables using `merge-engine=deduplicate`, following Spark's 
`DeleteFromPaimonTableCommand#performPrimaryKeyDelete`.
   
   ### Brief change log
   
   - Route DELETE on primary-key tables to `execute_primary_key_delete`, which 
scans the logical rows matching the `WHERE` clause, marks them with a DELETE 
`_VALUE_KIND` column and writes them through `TableWrite`, so the tombstones 
keep the primary-key, partition and sequence-field values. Commit them through 
the existing commit path and return the matched-row count; a delete without 
matches publishes no snapshot.
   - Plan the scan against one captured snapshot: the `FROM` source must be the 
target table itself, not a view or a time-traveled copy, and self-references in 
the predicate, including subqueries, read the same snapshot. Concurrent upserts 
follow the normal primary-key merge rules, including `sequence.field` ordering.
   - Write all tombstones before committing, so commit retries reuse the 
prepared files instead of rescanning. Close the writer when the scan or write 
fails to discard unprepared output; when the commit fails, keep the prepared 
files and report the table and commit user, since the snapshot may have been 
published.
   - Reject other merge engines, `ignore-delete` including its fallback keys, 
`rowkind.field`, `query-auth` and DELETE clauses beyond a single target with 
`WHERE` before scanning or writing.
   
   ### Tests
   
   - Unit tests in `crates/integrations/datafusion/src/delete/tests.rs`: 
snapshot pinning of the target and predicate self-references, replaced targets, 
commit retries without rescanning, uncertain commits, tombstone write failure 
cleanup and rejection of unsupported options before scanning.
   - Integration tests in 
`crates/integrations/datafusion/tests/pk_delete_tests.rs`: predicates, NULLs, 
no-match and never-written tables, subqueries, partitions, composite keys, 
dynamic buckets, quoted names, sequence fields and sort orders, concurrent 
upserts, scan failure cleanup, existing target guards and rejected merge 
engines, options and syntax.
   - Replace `test_delete_rejects_primary_key_table` and 
`test_pk_aggregation_rejects_delete`.
   
   ### API and Format
   
   No API or storage format change.
   
   ### Documentation
   
   Update `docs/src/sql.md` with the supported primary-key tables, snapshot, 
concurrency and retry semantics, and the unsupported options.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to