voonhous commented on code in PR #19783:
URL: https://github.com/apache/hudi/pull/19783#discussion_r3893260104
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -93,28 +93,41 @@ class
SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
private lazy val allFilters = filters ++ requiredFilters
// For each field of `target`, replace its dataType with the matching
field's projected
- // variant struct from `source` (when present). Non-matching fields pass
through. Why a parallel
- // `sparkRequiredSchema` overlay exists at all is documented on that
constructor parameter.
+ // variant struct from `source` (when present), recursing into struct
members so a variant
+ // reached through a struct path is overlaid too. Fields are matched by name
(findFieldByName);
+ // non-matching fields pass through. The recursion mirrors
PushVariantIntoScan's
+ // VariantInRelation.rewriteType, which rewrites variants at the root of the
relation output
+ // and below STRUCT paths only, so an array element or a map value is never
overlaid here
+ // either (#19775). Why a parallel `sparkRequiredSchema` overlay exists at
all is documented on
+ // that constructor parameter.
private def overlayVariantProjections(target: StructType, source:
StructType): StructType = {
StructType(target.fields.map { f =>
- SparkFileFormatInternalRowReaderContext.findFieldByName(source,
f.name).map(_.dataType) match {
- case Some(projStruct: StructType) if
sparkAdapter.isVariantProjectionStruct(projStruct) =>
+ (f.dataType,
SparkFileFormatInternalRowReaderContext.findFieldByName(source,
f.name).map(_.dataType)) match {
+ case (_, Some(projStruct: StructType)) if
sparkAdapter.isVariantProjectionStruct(projStruct) =>
f.copy(dataType = projStruct)
+ case (targetStruct: StructType, Some(sourceStruct: StructType)) =>
Review Comment:
In scope: `addMissingFields` now has the arm for the pair `isDataTypeEqual`
already declares equal. A projection struct requested over a file `VariantType`
stays the reader type instead of being folded back to `VariantType` -- which
parquet then decoded as a raw variant that the type-change `Cast` cast to the
projected struct, i.e. nulls for that member. Pinned in
`TestBaseSpark4AdapterVariantMethods.testImplicitSchemaChangeKeepsNestedVariantProjection`
(file `s.n` int, table `s.n` long, `s.inner` projected: the reconciled reader
type keeps the projection struct) and functionally by the new "Implicit
widening of a sibling keeps the nested variant projection" leg in
`TestVariantShreddingMixedLayouts`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
