wombatu-kun commented on code in PR #19558:
URL: https://github.com/apache/hudi/pull/19558#discussion_r3755749617
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -159,7 +159,37 @@ class
SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
structType
}
- val (readSchema, readFilters) =
getSchemaAndFiltersForRead(parquetReadStructType, hasRowIndexField)
+ // Internal reads have no catalyst plan, so nothing rewrites VariantType
fields the way
+ // PushVariantIntoScan does for user queries. Requesting native
VariantType against a
+ // SHREDDED parquet base file clips the file group to {metadata, value}
and reads
+ // value=null; write-side callers (compaction, clustering, merge) would
then persist the
+ // nulls, silently losing the variant data (#19556). Query paths that
build this context
+ // without sparkRequiredSchema (MOR incremental relation, streaming, CDC)
hit the same
Review Comment:
`HoodieStreamSourceV2` reaches `MergeOnReadIncrementalRelationV2` only when
`hoodie.file.group.reader.enabled` is false (it defaults to true), and both
pre-v8 and v8+ batch incremental go through
`HoodieFileGroupReaderBasedFileFormat` with a catalyst schema, so
`CDCFileGroupIterator` is the only default-config query path that takes this
rewrite. Worth narrowing this comment and the impact section to CDC, or naming
the config gate - not a blocker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
