JingsongLi opened a new pull request, #8240: URL: https://github.com/apache/paimon/pull/8240
## Summary Add Python-side minor manifest compaction after commits, matching the minor path of Java ManifestFileMerger while intentionally avoiding full compaction and manifest sort rewrite. This prevents committed Python tables from accumulating many small base manifest files. ## Changes - Add a Python ManifestFileMerger that performs minor-only manifest compaction with Java-compatible ADD/DELETE cancellation semantics. - Wire FileStoreCommit to compact existing data manifests before writing the new base manifest list; newly committed files still go to the delta manifest list. - Add manifest target size and merge minimum count options to Python CoreOptions. - Move manifest meta creation into ManifestFileManager so normal writes and compaction share the same metadata calculation. - Add tests for commit-time minor compaction and ADD/DELETE cancellation during minor compaction. ## Testing - python -m compileall -q paimon-python/pypaimon/manifest/manifest_file_merger.py paimon-python/pypaimon/manifest/manifest_file_manager.py paimon-python/pypaimon/write/file_store_commit.py paimon-python/pypaimon/common/options/core_options.py paimon-python/pypaimon/tests/write/table_write_test.py paimon-python/pypaimon/tests/manifest/manifest_entry_identifier_test.py - cd paimon-python && python -m unittest pypaimon.tests.manifest.manifest_entry_identifier_test pypaimon.tests.file_store_commit_test pypaimon.tests.manifest.manifest_manager_test.ManifestFileManagerTest pypaimon.tests.write.table_write_test - git diff --check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
