voonhous commented on code in PR #19558:
URL: https://github.com/apache/hudi/pull/19558#discussion_r3755652356
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -159,7 +159,37 @@ class
SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
structType
}
- val (readSchema, readFilters) =
getSchemaAndFiltersForRead(parquetReadStructType, hasRowIndexField)
+ // Internal reads have no catalyst plan, so nothing rewrites VariantType
fields the way
+ // PushVariantIntoScan does for user queries. Requesting native
VariantType against a
+ // SHREDDED parquet base file clips the file group to {metadata, value}
and reads
+ // value=null; write-side callers (compaction, clustering, merge) would
then persist the
+ // nulls, silently losing the variant data (#19556). Query paths that
build this context
+ // without sparkRequiredSchema (MOR incremental relation, streaming, CDC)
hit the same
Review Comment:
Added the incremental round trip to the compaction test. One precision that
came out of writing it: on current table versions a batch incremental query
scans through the file-group-reader file format with a catalyst schema (the
overlay leg), so that is what the round trip pins. The no-catalyst-schema legs
-- the streaming source and CDC -- are tracked in #19578, and the impact
section now says exactly that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
