leaves12138 commented on code in PR #10305:
URL: https://github.com/apache/paimon/pull/10305#discussion_r4129164428
##########
paimon-python/pypaimon/write/native_write.py:
##########
@@ -81,7 +81,10 @@ def create_native_write(table, commit_user,
static_partition=None, stream=False)
or not _native_map_layouts_supported(table, schema)
# Rust cannot encode these partition keys yet.
or not _native_partition_types_supported(schema,
table.partition_keys)
- or any(is_blob_file_field(field) for field in
table.table_schema.fields)):
+ # Native dedicated files currently support top-level scalar Blob
fields.
+ or table.options.video_frame_fields()
+ or any(is_blob_file_field(field) and not is_blob_type(field.type)
Review Comment:
[P2] Preserve explicit abort cleanup after preparing a Blob write.
Opening the native route here changes an existing lifecycle guarantee:
`writer.write_arrow(batch); messages = writer.prepare_commit(); writer.abort()`
now leaves the uncommitted `.parquet` and `.blob` files on storage.
`NativeTableWrite.abort()` delegates only to `close()`, while the Rust writer
drains its writers during `prepare_commit()` and intentionally transfers
ownership of those files to the caller. Its subsequent close therefore has
nothing to delete. The Python Blob writer previously removed these files on
explicit abort. Failed/cancelled writes after preparation can consequently leak
large Blob files.
This is reproduced by the existing
`pypaimon/tests/data_evolution_formats_test.py::DataEvolutionFormatsTest::test_blob_abort_deletes_uncommitted_files`,
with native write enabled and the freshly rebuilt #989 binding: it passes on
the Python base and fails on this head because the prepared Parquet file still
exists after abort. The new cleanup cases only abort before preparation, so
they miss this boundary.
Please preserve the adapter's cleanup contract for prepared-but-uncommitted
messages (including associated sidecars/external paths), or keep this path on
the Python writer until that is supported. Normal close must still preserve
files handed off for commit; changing Rust close to delete all prepared files
would not be a safe fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
