voonhous opened a new issue, #19688:
URL: https://github.com/apache/hudi/issues/19688
### Context
Hudi can write and read shredded variants, but every shredded test today
forces ONE `typed_value` layout for the whole table via the test-only config
`hoodie.parquet.variant.force.shredding.schema.for.test`. Once per-file
shredding-schema inference (#18961, part of #18937) lands, mixed layouts become
the normal state of a table:
- files with different `typed_value` schemas in one table (each file infers
its own),
- shredded base files under unshredded logs and the reverse (log files and
compaction/clustering output re-derive the layout from the current write
config, never from the input files),
- rows inside one shredded file whose values did not match the shredding
schema and fell back to the residual `value` column.
None of this was pinned end to end, and auditing the paths that will meet
these mixes turned up real defects.
### Defects found
1. **Avro write path crashes on non-object rows under an object shredding
schema.** `HoodieSchema.createVariantShredded` declared `typed_value` as
required, but the shredding spec makes it optional: a scalar, array, or JSON
null under an object schema leaves `typed_value` null and carries everything in
the residual. parquet-avro throws `Null-value for required field: typed_value`.
The Spark row writer already writes it optional and the shredding provider
already handles the null; only the schema builder disagreed. (This is also the
crash the #18961 Avro inference path would hit on any post-sample row that does
not match the inferred schema.)
2. **Hive returned silent nulls for shredded files.** The Hive reader hands
base files to a plain parquet-avro read at the requested `{metadata, value}`
projection, so typed rows lose their payload (it lives in `typed_value`, which
the projection drops) with no error.
3. **Bootstrap merge dropped `typed_value`.** The one-argument
`HoodieBootstrapFileReader.getRecordIterator` (used by `HoodieMergeHelper`'s
bootstrap branch) read the data file at its own footer schema, where a shredded
variant is a plain record with the logical type lost, so reconstruction never
engaged and the rewrite to the writer schema silently dropped the shredded
payload.
4. **Clustering sort on a variant column failed deep in the Spark job** (an
`AnalysisException` from the row partitioner or a `ClassCastException` from the
RDD one) instead of a validation error naming the column. Same for BLOB/VECTOR.
5. **Schema-on-read + shredded file returned silent nulls** (with
`spark.sql.variant.pushVariantIntoScan=false`): the internal schema models a
variant as a two-field record, so the merged request clips `typed_value` away.
Under the default push-on rewrite the read already fails, but with
engine-internal errors. Related #18285 finding: the schema-on-read DDL also
degrades the CATALOG schema's variant column to a plain
`struct<metadata,value>`, so even plain reads after such a DDL fail in Spark
before any Hudi hook.
6. **Spark 4.0 could silently drop `typed_value` from the requested schema**
in the `reorderVariantFields` workaround (#18334). Empirically Spark 4.0's own
converter rejects the 3-field group first on the common path, but the reorder
needed a guard as defense in depth.
Also: the `org.apache.spark.sql.adapter` test package (the two adapter
variant suites) was matched by no CI lane's wildcard filter and never ran.
### What the fix PR does
- A mixed-layout test matrix (`TestVariantShreddingMixedLayouts`, 15 tests +
a shared `VariantShreddingTestSupport` trait extracted from
`TestVariantDataType`) pinning: same-file typed-vs-residual row placement, four
layouts in one table read through snapshot / time travel / incremental /
read-optimized, small-file bin-pack layout re-derivation, a three-layout MOR
log split compacted three times under three different layouts, table version 9
legacy log blocks, ordering modes with deletes, heterogeneous clustering on the
row-writer and RDD paths (COW and MOR-with-logs), `variant_get` across files
where a path is typed / residual / conflicted / absent, CDC across layout
changes, nested variants, add-column evolution through compaction and
clustering, and rollback/savepoint across layouts.
- Fixes for all six defects above, each pinned by a test that was red before
the fix. Engines that cannot reconstruct shredded variants (Hive, Spark 4.0,
schema-on-read) now fail fast with an error naming the column and the
remediation, instead of silent nulls.
- A Spark 4.1-written mixed-layout fixture
(`variant_backward_compat/variant_shredded_mixed_cow.zip`) for cross-engine
pins, plus a Flink test that reading it fails fast.
### Out of scope / follow-ups
- Inference-specific legs (sample-vs-tail behavior, decline rules) activate
once #18961 merges; the default flip is tracked by #18937.
- Real shredded-variant support under schema-on-read, including the
catalog-schema degradation above: #18285.
- The legacy (non-file-group-reader) Hive path has no requested-schema
anchor and stays unguarded.
- Off-Spark reconstruction (Hive/Trino/Flink reading shredded files as
values rather than failing fast).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
