JingsongLi commented on code in PR #10655:
URL: https://github.com/apache/paimon/pull/10655#discussion_r4374710343
##########
paimon-python/pypaimon/execution/python/write/file_store_write.py:
##########
@@ -127,27 +131,32 @@ def write_row(
require_columns(values_by_name, column_names, 'write_row')
writer = self.data_writers.get(key)
data = None
- if writer is None and self._has_blob_columns():
- # Dedicated writers consume live payloads themselves. Inline fields
- # instead encode to Arrow before the writer captures its layout.
- descriptor_fields = self.options.blob_descriptor_fields()
- view_fields = self.options.blob_view_fields()
+ if self._has_blob_columns():
+ descriptor_fields =
CoreOptions.blob_descriptor_fields(self.options)
+ view_fields = CoreOptions.blob_view_fields(self.options)
inline_columns = [name for name in column_names if name in
descriptor_fields or name in view_fields]
if inline_columns:
values_by_name = dict(values_by_name)
for name in inline_columns:
values_by_name[name] = inline_blob_value(
values_by_name[name], name in descriptor_fields, name
in view_fields)
Review Comment:
[P2] Normalize inline BLOB rows before postpone bucket planning
This normalization is too late for
`table.new_postpone_fixed_bucket_write_builder().new_write()`. On a fresh PK
table with `bucket=-2`, `postpone.batch-write-fixed-bucket=true`,
`blob-descriptor-field=picture`, and `write.native.enabled=false`, writing a
`GenericRow` containing `Blob.from_descriptor_bytes(...)` to a new partition
raises `ValueError: Blob values cannot be converted to Arrow without
materializing the stream`. `PythonPostponeFixedBucketWrite.write_row` first
calls `row_values_to_arrow_table` at `postpone_batch_table_write.py:132` to
estimate the unknown partition's size, so it never reaches this inline encoding.
I reproduced this with Python 3.13 / PyArrow 22.0.0. Serialized descriptor
bytes through Arrow commit and read correctly on the same table; after that
partition exists, the same Blob-object row succeeds. A fresh ordinary
fixed-bucket PK table also succeeds. Please share the inline-reference
normalization with the postpone sizing/buffering path before its Arrow
conversion, and cover the first write into a new partition for descriptor/view
fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]