JingsongLi commented on PR #8181: URL: https://github.com/apache/paimon/pull/8181#issuecomment-4966131544
1. Inline on schema_manager.py [Major] Reject ARRAY<BLOB> partition keys during schema validation _validate_blob_fields validates primary keys but does not receive or check partition_keys. As a result, a table partitioned by an ARRAY<BLOB> column can be created successfully, but its first write fails while grouping partitions with TypeError: unhashable type: 'list'. Please pass the partition keys into this validation and reject BLOB/ARRAY<BLOB> partition columns during both create_table and commit, consistent with the Java validation. 2. Overall review comment because daft_datasource.py is unchanged [Major] ARRAY<BLOB> tables are incorrectly routed to Daft’s native Parquet reader Daft only recognizes top-level large_binary fields as blob columns. Since ARRAY<BLOB> is represented as list<large_binary>, _has_blob_columns remains false. An append-only Parquet table is therefore routed to the native reader, which attempts to read the table’s .blob file as Parquet and fails. Please detect dedicated blob storage from the Paimon schema and route ARRAY<BLOB> through the Pypaimon reader. If ARRAY<BLOB> conversion is not supported in Daft yet, it should be rejected explicitly instead of scheduling .blob files as Parquet. 3. Inline on format_blob_reader.py [Major] ARRAY<BLOB> bypasses blob_parallelism Each non-null array element is read using a separate synchronous seek/read operation. Unlike scalar BLOB values, array elements never enter the existing blobs_to_resolve / read_blobs_concurrent path, so blob_parallelism has no effect. On object stores, this makes request count and latency scale with the total number of array elements. Please batch the element ranges through the existing concurrent/coalesced resolver and rebuild the arrays afterward. For the serial path, the contiguous element payload could also be read once and sliced in memory. 4. Inline on row_utils.py [Minor] Avoid recursively scanning every nested non-BLOB value _contains_blob_value now recursively traverses every list, tuple, and dictionary, including ordinary ARRAY/MAP fields. In DedicatedFormatWriter.write_row, normal nested values are checked here and then checked again through row_values_to_arrow_table, causing two full Python traversals before Arrow conversion. Please make this validation schema-aware so known non-BLOB fields can bypass the recursive scan, or at least remove the duplicate check in the dedicated row-writing path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
