JingsongLi opened a new pull request, #10097:
URL: https://github.com/apache/paimon/pull/10097
## Summary
Follow-up to #10081. Enable batch overwrite through the existing opt-in
`commit.native.enabled` route, using apache/paimon-rust#916 (based on
apache/paimon-rust#915).
- Create a native batch committer for overwrite and preserve the Python
writer's `commit_user` through the internal batch identity bridge. Keep the
Java-style public builder API unchanged.
- Preserve static and dynamic partition overwrite semantics, null/default
partitions, fixed-bucket imported messages, the batch identifier, and one-shot
lifecycle. Empty overwrite bypasses append's ignore-empty shortcut.
- Match Java's static/table overwrite behavior in the Python fallback too:
publish an OVERWRITE snapshot even when the table is empty or no files match
the spec. Dynamic empty overwrite remains a no-op.
- Probe the new batch capability before table reconstruction. Older/missing
runtimes, stream overwrite, callbacks, snapshot properties, unsupported
environments, and row-id recovery continue through Python.
- Keep native mutation outside preparation fallback. Publication errors
propagate without Python retry or automatic abort. Truncate APIs retain their
existing route.
## Usage
```python
native_table = table.copy({
"commit.native.enabled": "true",
"dynamic-partition-overwrite": "false",
})
builder = native_table.new_batch_write_builder().overwrite({"pt":
"2026-09-22"})
writer, commit = builder.new_write(), builder.new_commit()
try:
writer.write_arrow(data)
commit.commit(writer.prepare_commit())
finally:
writer.close()
commit.close()
```
The existing dynamic overwrite default is unchanged; set
`dynamic-partition-overwrite=false` for static partition matching as in this
example.
## Verification
- Built a Rust wheel from #915 (`d0948d23`) plus the identity bridge in #916
(`f58478f4`). Rust binding commit tests: **43 passed**.
- Against this wheel, `native_commit_test.py`, `table_commit_test.py`,
`file_store_commit_test.py`, `commit_message_serializer_test.py`,
`overwrite_commit_conflict_test.py`, and `write/commit_callback_test.py`: **126
passed, 2 subtests passed**, with no skipped native cases.
- Real cross-language roundtrips cover static/dynamic/null partition
overwrite, append/primary-key/data-evolution tables, empty input, empty tables
and unmatched static specs, snapshot identity and identifier, abort, and
response loss after publication.
- Conversion-failure fallback replaces only the target partition and never
invokes native commit/abort. Callback, snapshot-property and stream overwrite
cases retain their Python behavior.
- Re-ran the native commit suite against the older #912 wheel: **47 passed,
18 skipped** (new native overwrite cases require the bridge); existing append
and fallback tests pass.
- Flake8 on all changed Python files and `git diff --check` passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
