leaves12138 opened a new pull request, #9956:
URL: https://github.com/apache/paimon/pull/9956
### Purpose
Extract the remaining PyPaimon writer integration from #9743. This builds on
the merged v1 codec/block-pruned reader in #9908 and the manifest-reference
retry fix in #9921; it does not change the sidecar format or Java
implementation.
- Generate sidecars for ordinary and rolling manifest writes, including
manifests produced by minor compaction.
- Publish `_EXTRA_FILES` only after both immutable outputs close
successfully. `manifest.sidecar.enabled` controls reads and writes and
continues to inherit `manifest-sort.enabled` when unset.
- Clean up newly created manifest/sidecar pairs on write or preparation
failure, including previously completed rolling outputs. Preserve existing
manifests and propagate cancellation. Commit outcomes that may be uncertain
keep the existing no-cleanup behavior.
- Keep coarse RowID bounds unknown for invalid ranges, consistent with
sidecar coverage.
- Document writer behavior and exercise the real writer in the existing
pruning tests.
### Tests
- Python 3.11: **276 passed** across manifest, commit/retry, explain, cache,
file-type, and bucket-pruning suites.
- Python 3.6: **116 passed** across sidecar, block-index, and file-type
suites.
- Flake8 passed with `paimon-python/dev/cfg.ini`; `git diff --check` passed.
- Added coverage for explicit disablement, write/close failures,
interruption, rolling cleanup, commit cleanup ownership, and invalid RowID
bounds.
- Additional isolated Python-write/Java-read check: 12 manifests, 1,806
entries, null/deflate/Zstandard compression, ordinary and rolling outputs, an
empty manifest, and 108 block-pruned read comparisons all matched the full Java
reads.
```sh
PYTHONPATH=paimon-python python -m pytest \
paimon-python/pypaimon/tests/manifest \
paimon-python/pypaimon/tests/file_store_commit_test.py \
paimon-python/pypaimon/tests/read_builder_explain_test.py \
paimon-python/pypaimon/tests/caching_file_io_test.py \
paimon-python/pypaimon/tests/file_type_test.py \
paimon-python/pypaimon/tests/pushdown_bucket_test.py -q
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
