sunchao commented on code in PR #4950:
URL: https://github.com/apache/datafusion-comet/pull/4950#discussion_r3889723040
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -102,6 +102,19 @@ object CometNativeScan extends
CometOperatorSerde[CometScanExec] with CometTypeS
withFallbackReason(scanExec, "Full native scan disabled because
ignoreMissingFiles enabled")
}
+ // The native scan serializes the full data and partition schema (not just
the required
+ // columns), so every field type must be serializable. Types with no proto
representation
+ // (e.g. GEOMETRY / GEOGRAPHY) would otherwise crash schema serialization;
fall back instead.
+ (scanExec.relation.dataSchema.fields ++
scanExec.relation.partitionSchema.fields).foreach {
Review Comment:
[P2] Preserve native scans for pruned Variant collections
This check walks the raw relation schema before `convert` builds
`nativeDataSchema`. For an unprojected `ARRAY<VARIANT>` or `MAP<STRING,
VARIANT>` root, `serializeDataType` returns `None`, so the scan falls back even
though the later conversion would remove that root. I reproduced this under
Spark 4.2 with the existing `expressions/misc/variant.sql`: its native control
at line 103 failed with `Native scan does not support data type array<variant>`
and `map<string,variant>`, leaving the surrounding Sort and Exchange on Spark.
This regresses #5377. Please validate the post-pruning `nativeDataSchema` while
still checking the full partition schema, or otherwise exempt exactly the
Variant roots that the conversion path prunes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
