voonhous commented on issue #19556:
URL: https://github.com/apache/hudi/issues/19556#issuecomment-5219372386
Investigation update: the loss has TWO independent legs, both on internal
write-side read
paths that bypass catalyst. The original repro conflated them.
**Leg 1 -- Spark internal parquet reader** (clustering rewrites, compaction
base-file
reads, upsert merges under the Spark record merger): internal reads request
native
VariantType, so the reader clips a shredded file group to `{metadata,
value}` and reads
value=null; the rewrite persists the nulls. Fixed by #19558, which requests
the
PushVariantIntoScan full-variant projection shape for internal parquet
base-file reads and
restores native VariantType after the scan.
**Leg 2 -- Avro internal read path** (e.g. the CoW small-file merge under
the avro record
merger): `HoodieVariantReconstruction` does not engage on real files (the
footer
MessageType conversion loses the variant logical type), so the avro reader
silently reads
`{metadata, value}` and drops `typed_value`. In the original repro this is
what actually
nulled the first commit's rows: the second INSERT bin-packed into the first
file group and
the small-file merge corrupted it BEFORE clustering ran. Still open; needs
the
reconstruction fixes from the shredding-inference line (feat-18937).
Repro guidance after #19558: setting `hoodie.parquet.small.file.limit=0`
isolates leg 1
(now green); the default small-file behavior with two inserts still
reproduces leg 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
