leaves12138 commented on code in PR #7602:
URL: https://github.com/apache/paimon/pull/7602#discussion_r3042727545
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java:
##########
@@ -495,6 +497,11 @@ private Schema toInitialSchema(
field.dataType() instanceof
org.apache.spark.sql.types.BinaryType,
"The type of blob field must be binary");
type = new BlobType();
+ } else if (blobRefFields.contains(name)) {
Review Comment:
Same concern on the Spark side: adding the catalog/type mapping here is not
enough by itself. `SparkInternalRow.blobFields(...)` still only collects
`DataTypeRoot.BLOB`, so reads return serialized reference bytes, and
`SparkInternalRowWrapper#getBlob` still only recognizes `BlobDescriptor`, so V2
writes wrap `BLOB_REF` bytes as `BlobData` and then fail in
`BinaryWriter#serializeBlobReference`. Could we update those runtime wrappers
too?
##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/reader/ParquetVectorUpdaterFactory.java:
##########
@@ -230,6 +231,11 @@ public UpdaterFactory visit(BlobType blobType) {
};
}
+ @Override
+ public UpdaterFactory visit(BlobRefType blobRefType) {
Review Comment:
Likewise for parquet, the reader updater is mirrored here, but the
write/schema side still only switches on `BLOB` (`ParquetSchemaConverter`,
`ParquetRowDataWriter`, and `ParquetReaderUtil`). With the current diff a
parquet table containing `BLOB_REF` is still unsupported. Should those code
paths be updated together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
