Stephen0421 commented on PR #10114: URL: https://github.com/apache/paimon/pull/10114#issuecomment-5892128132
> Requirement fit: SUPPORTED for the lifecycle part of this staged #9099 work. There is a concrete existing path: a data-evolution write creates a `.row` extra file, and aborting prepared commit messages previously removed the data file but left that sidecar. I reproduced a real table write with both files present, then `TableCommit.abort(messages)` removing both. The managed-BLOB reference collector is intentionally not yet wired into PK writes, so this PR does not by itself deliver `.blobref` generation; please keep the PR description and the follow-up dependency explicit. > > Implementation: FINDINGS. **[P1] Honor consumer-owned BLOB packs on the native abort route** (`paimon-python/pypaimon/write/file_store_write.py:339`). The new `preserve_blob_files_on_abort` flag is read only by Python `_abort_commit_messages`. With `commit.native.enabled=true`, `TableCommit.abort` routes supported messages to native; `native_messages_supported` currently returns true even when this flag is set, and the Java abort deletes all new files, including `.blob` packs. A write using `with_blob_consumer` followed by native abort can therefore delete a pack whose descriptor the consumer retains. I verified the native eligibility check accepts such a message. Please route these aborts through the Python implementation or carry equivalent ownership semantics through the native protocol, and add a regression that exercises the actual `TableCommit.abort` dispatch. This blocks production merge for the opt-in native path. > > Format/release boundary: Python's sidecar magic, version, modified UTF-8 and payload CRC match the existing Java reader in source and focused tests. The collector currently has no production `collect_table` call or `DataFileMeta.extra_files` registration; the later PK-write PR must provide that path and a Python-write/Java-read or GC reachability test before managed BLOBs can be enabled. No new on-disk format is produced by current PK writes in this PR. > > Verification on head `e721d6fb87`: the three new Python modules passed 15/15; 13 targeted existing abort/multi-prepare tests passed; `git diff --check` passed. A real local data-evolution table write confirmed `.parquet` and `.parquet.row` exist before `TableCommit.abort` and neither exists after. The Java `ManagedBlobReferenceFileTest` passed 4/4 in a separate focused run. Native Rust runtime and an actual Python-to-Java sidecar round trip were not available locally; the native abort issue is established by the dispatch and Java deletion paths. Release gate: BLOCK until the native abort ownership path is fixed and covered. Addressed the native abort ownership gap. Messages with `preserve_blob_files_on_abort` no longer take the native path, because v14 cannot carry the flag and native abort would delete packs a `BlobConsumer` still owns. `TableCommit.abort` uses the Python implementation for those messages, and `test_preserve_blob_abort_skips_native_and_keeps_pack` covers that dispatch: the unmarked message calls `native.abort`, the marked one does not, and `pack.managed.blob` remains. The collector is still not called from primary-key writes, and this PR does not emit `.blobref` files from those writes. The description already says the follow-up PK-write change has to register the sidecar on `DataFileMeta.extra_files` and add a Python-write/Java-read or GC reachability test before managed BLOBs can be enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
