leaves12138 commented on code in PR #9371:
URL: https://github.com/apache/paimon/pull/9371#discussion_r3841483534
##########
paimon-arrow/src/main/java/org/apache/paimon/arrow/writer/ArrowFieldWriters.java:
##########
@@ -591,11 +597,14 @@ protected void doWrite(
InternalRow rowData = rowColumnVector.getRow(row);
if (variantSchema != null && rowData.getFieldCount() !=
fieldCount) {
Review Comment:
This discriminator is ambiguous for a valid two-field top-level shredding
schema. For example, variantShreddingSchema(DataTypes.VARIANT()) produces
[metadata, value], while the logical columnar Variant row is [value, metadata];
both have a field count of 2. This branch therefore treats the logical row as
an already-shredded row and writes the two buffers in the wrong order. I
reproduced the ArrowFormatWriter.write(ColumnVector[], ...) -> ArrowBatchReader
round trip, which fails with MALFORMED_VARIANT. Could we distinguish the input
representation explicitly (or structurally) instead of by field count, and add
a regression test for this two-field schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
