github-actions[bot] commented on code in PR #66326:
URL: https://github.com/apache/doris/pull/66326#discussion_r3689625641
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -97,6 +97,8 @@ public static ConnectorType toConnectorType(DataType
dataType, Options options)
return toTimestampType(dataType);
case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
return toTimestampTzType(dataType, options);
+ case VARIANT:
Review Comment:
[P1] Keep Paimon Variant columns off the native reader. This mapping makes
the column queryable, but append-only Parquet splits are native-eligible by
default because `shouldUseNativeReader` only checks force flags and file
suffixes. The Parquet Variant is a value/metadata group, and the native schema
builder has no `TYPE_VARIANT` case, so it reaches the scalar fallback and fails
with `Parquet column payload is not a scalar`. The regression hides this path
by setting `force_jni_scanner=true` for every query. Please route splits
containing Variant columns (including nested ones) to JNI until the native
reader supports this encoding, and cover the default setting.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonColumnValue.java:
##########
@@ -198,6 +199,20 @@ public byte[] getBytes() {
return record.getBinary(idx);
}
+ @Override
+ public byte[] getVariantMetadata() {
+ return getVariant().metadata();
+ }
+
+ @Override
+ public byte[] getVariantValue() {
+ return getVariant().value();
Review Comment:
[P2] Decode each Variant only once. `VectorColumn` calls
`getVariantMetadata()` and `getVariantValue()` back-to-back, but both methods
re-enter `record.getVariant(idx)`. On Paimon's Parquet columnar path,
`getVariant` materializes both the value and metadata arrays on every call, so
every cell pays for both buffers twice before Doris copies them off heap.
Please cache one Variant for the current wrapper state (clearing it in
`setOffsetRow`, `setIdx`, and the nested-wrapper `reset` method), or expose a
combined accessor so both arrays come from one decode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
