voonhous commented on code in PR #19558:
URL: https://github.com/apache/hudi/pull/19558#discussion_r3755265626
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -159,7 +159,31 @@ class
SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
structType
}
- val (readSchema, readFilters) =
getSchemaAndFiltersForRead(parquetReadStructType, hasRowIndexField)
+ // Internal reads have no catalyst plan, so nothing rewrites VariantType
fields the way
+ // PushVariantIntoScan does for user queries. Requesting native
VariantType against a
+ // SHREDDED parquet base file clips the file group to {metadata, value}
and reads
+ // value=null; write-side callers (compaction, clustering, merge) would
then persist the
+ // nulls, silently losing the variant data (#19556). Request the
full-variant projection
+ // shape instead and restore native VariantType after the scan.
User-facing reads pass
+ // sparkRequiredSchema and are overlaid above; Spark < 4.1 has no shredded
read support
+ // and the adapter returns None (shredded files cannot be written there
either).
+ val isParquetBaseFile = !isInlineLog && !FSUtils.isLogFile(filePath) &&
+ HoodieFileFormat.fromFileExtension(filePath.getFileExtension) ==
HoodieFileFormat.PARQUET
+ val (readStructTypeForScan, variantOrdinals) =
+ if (sparkRequiredSchema.isEmpty && isParquetBaseFile) {
Review Comment:
Intended -- those paths request plain `VariantType` against shredded files
and hit the same null reads, so they need the same rewrite. Updated the comment
and the impact section to say so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
