shyjsarah opened a new issue, #572:
URL: https://github.com/apache/paimon-rust/issues/572
### Search before asking
- [x] I searched the existing issues and pull requests and found nothing
similar.
### Please describe the bug
Paimon Rust currently rejects `ignore-delete` options on primary-key tables
using
`merge-engine=partial-update`, even though Java Paimon supports this mode.
For example, creating or opening a table with either option fails validation:
```text
merge-engine=partial-update
ignore-delete=true
```
or:
```text
merge-engine=partial-update
partial-update.ignore-delete=true
```
The read-side and flush-time partial-update merge functions also reject
`DELETE` and `UPDATE_BEFORE` rows unconditionally. This prevents Rust readers
from opening Java-created tables and historical files that use the supported
ignore-delete semantics.
Expected behavior:
- Both `ignore-delete` and `partial-update.ignore-delete` are accepted.
- With an effective value of `true`, `DELETE` and `UPDATE_BEFORE` are
ignored.
- The default and an explicit `false` still reject retract rows.
- Ignored retract rows do not consume writer buffer, sequence numbers, or
enter
input changelog files.
- Once enabled, the effective value cannot be changed from `true` to `false`,
matching Java Paimon compatibility safeguards.
- Other advanced partial-update options remain unsupported.
### Proposed solution
- Narrowly allowlist the two supported option keys in partial-update
validation.
- Apply ignore-delete semantics in both read-side merge and write-side
filtering.
- Filter direct `_VALUE_KIND` batches before writer buffering.
- Add schema, writer, reader, core integration, and DataFusion integration
tests.
- Document the supported behavior and remaining limitations.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
