voonhous opened a new pull request, #19558:
URL: https://github.com/apache/hudi/pull/19558
### Describe the issue this Pull Request addresses
Fixes the Spark-reader half of #19556: internal write-side reads (clustering,
compaction base-file reads, upsert merges under the Spark record merger)
silently read
shredded VARIANT columns as null and rewrote the nulls into new base files,
permanently
losing the data. Clustering was the reported trigger; the same read stack
serves the other
rewrite paths.
Root cause: a shredded parquet file stores the data in `typed_value` and
leaves `value`
physically null. User queries handle this because Spark 4.1's
`PushVariantIntoScan`
rewrites VariantType scan columns into a `VariantMetadata`-tagged projection
struct, which
the parquet reader assembles by field name. Internal reads bypass catalyst,
so they
requested native VariantType, the reader clipped the file group down to
`{metadata, value}`, and every fully-shredded value came back null with no
error.
Note #19556 has a second, independent leg: the AVRO internal read path (e.g.
the CoW
small-file merge under the avro record merger) drops shredded values because
`HoodieVariantReconstruction` does not engage on real files. That is NOT
addressed here;
it needs the reconstruction fixes from the shredding-inference work. Details
on the issue.
### Summary and Changelog
Internal reads now request shredded variants the same way
PushVariantIntoScan would, so
rewrites carry full variant values forward.
- `SparkAdapter.buildFullVariantReadSchema`: new hook that rewrites
top-level VariantType
fields into the full-variant projection struct (single child "0" at path
`$`), mirroring
`RequestedVariantField.fullVariant`. Default None; implemented for Spark
4.1 and 4.2.
Spark 3.x/4.0 have no shredded read support and keep the old behavior
(they cannot
write shredded files either).
- `SparkFileFormatInternalRowReaderContext`: when reading a parquet base
file without a
catalyst-provided `sparkRequiredSchema`, apply the rewrite for the scan
and restore
native VariantType afterwards by projecting child 0 of each rewritten
field. Composes
with the existing VECTOR binary-to-array post-processing; inert for
Lance/ORC/log files
and when the adapter returns None.
- Tests (TestVariantDataType): shredded COW clustering round-trip (the
#19556 repro, now
green), an unshredded twin isolating the shredded layout as the trigger,
and
bulk_insert row-writer round-trips for both layouts. The clustering tests
pin
`hoodie.parquet.small.file.limit=0` so the second commit does not bin-pack
through the
CoW small-file merge, which on the avro merger path still has the unfixed
second leg
above and would conflate the two.
### Impact
No public API or user-facing behavior change; only internal write-side reads
change, and
only for variant columns on Spark 4.1+. User query plans are untouched.
### Risk Level
low. The rewrite only activates when (a) no catalyst-provided required
schema is present,
(b) the file is a parquet base file, and (c) the schema has top-level
VariantType fields
on Spark 4.1+. Verified with the full TestVariantDataType,
TestVectorDataSource, and blob
parquet service round-trip suites under the spark4.1 profile, and the same
suites under
spark3.5 (where the new hook returns None and behavior is unchanged).
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
