Pandas886 opened a new pull request, #510:
URL: https://github.com/apache/paimon-rust/pull/510
### Purpose
Linked issue: close #505
**Depends on:** Delta + Changelog PRs (stacked). Against `main` you'll see
the whole stack; the new work is Diff planning + before/after comparison.
For tables with `changelog-producer=none`, Diff is how you still get an
audit-style change stream without materializing changelog during write.
### Brief change log
- Diff planning: before/after snapshot state → `DiffPair` per `(partition,
bucket)`
- Reject bucket rescale between snapshots (explicit `Unsupported`, no silent
wrong results)
- Read path: PK-sorted merge of before/after images, emit `+I` / `-U` / `+U`
/ `-D`, skip equal rows
- Phase-1 limits (fail loud):
- deduplicate only
- no deletion-vector support yet
- nested / decimal types rejected for now
- `diff.parallelism` option (default 4) for pair execution
- After-image mode on plain incremental read (non-audit) returns the
post-change rows only
I used input-delete rows for delete cases in tests so we don't need
compact/lookup generation in this PR.
### Tests
```bash
cargo test -p paimon --test incremental_batch_scan_test
cargo test -p paimon --test audit_log_table_test
cargo fmt --all -- --check
cargo clippy -p paimon --all-targets --features fulltext,vortex -- -D
warnings
```
### API and Format
- Diff mode becomes usable on the existing incremental / AuditLog APIs
- No storage format change
### Documentation
Still holding docs for the DataFusion PR so users see one end-to-end story.
### Known gaps (intentionally)
- Real DV-backed tables: unit-level fail-loud only, no full e2e with DV
files yet
- Non-deduplicate engines: hard reject
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
