voonhous opened a new issue, #20349:
URL: https://github.com/apache/hudi/issues/20349
### TLDR
A COW upsert with the default AVRO record type fails with
`AvroTypeException: Invalid default for field v: null not a {record ...}` when
the base file stores a shredded variant column as REQUIRED and the table schema
has it nullable. Every later upsert into that file group fails the same way, so
the file group can no longer be updated. Default configs are enough; the user
never declares NOT NULL.
Two ordinary ways to get a REQUIRED shredded variant into a base file:
- **DataFrame write.** In Spark 4.1, `parse_json` over a non-null expression
is a non-nullable column. Hudi keeps that nullability, so the first insert
writes `v` as REQUIRED. The next upsert whose `v` is nullable (a single NULL
row is enough, or just a nullable schema) fails.
- **Bootstrap.** Vanilla Spark 4.1 writes a non-null shredded variant as
REQUIRED. After a METADATA_ONLY bootstrap the table schema has `v` nullable,
and any COW upsert of a bootstrapped row fails, even when the incoming `v` is
non-null.
Not affected: Spark SQL DDL/DML tables (Hudi stores the column as nullable
even for `v variant not null`), the SPARK record type, and MOR compaction (the
compactor uses the Spark reader).
### To Reproduce
COW, all defaults, Spark 4.1:
```scala
val opts = Map(
"hoodie.table.name" -> "t",
"hoodie.datasource.write.recordkey.field" -> "id",
"hoodie.datasource.write.precombine.field" -> "ts")
spark.sql("""select cast(id as int) id, parse_json(concat('{"a":', id, '}'))
v, 1000L ts from range(10)""")
.write.format("hudi").options(opts).option("hoodie.datasource.write.operation",
"insert")
.mode("overwrite").save(path)
// fails
spark.sql("""select cast(id as int) id,
case when id = 6 then null else parse_json(concat('{"a":', id,
'}')) end v,
1001L ts from range(5, 7)""")
.write.format("hudi").options(opts).option("hoodie.datasource.write.operation",
"upsert")
.mode("append").save(path)
```
```
org.apache.hudi.exception.HoodieUpsertException: Failed to compact file
group ...
Caused by: org.apache.avro.AvroTypeException: Invalid default for field v:
null not a
{"type":"record","name":"v","fields":[{"name":"metadata","type":"bytes"},{"name":"value","type":["null","bytes"]},...typed_value...]}
at org.apache.avro.Schema.validateDefault(Schema.java:1720)
at
org.apache.hudi.common.avro.HoodieAvroUtils.createNewSchemaField(HoodieAvroUtils.java:395)
at
org.apache.hudi.common.schema.HoodieSchemaField.withSchema(HoodieSchemaField.java:282)
at
org.apache.hudi.common.avro.VariantSchemaUtils.swapShreddedVariantFields(VariantSchemaUtils.java:716)
at
org.apache.hudi.common.avro.VariantSchemaUtils.toShreddedReadSchema(VariantSchemaUtils.java:676)
at
org.apache.hudi.io.storage.hadoop.HoodieVariantReconstruction.create(HoodieVariantReconstruction.java:115)
at
org.apache.hudi.io.storage.hadoop.HoodieAvroParquetReader.getIndexedRecordIteratorInternal(HoodieAvroParquetReader.java:193)
at
org.apache.hudi.common.avro.HoodieAvroReaderContext.getFileRecordIterator(HoodieAvroReaderContext.java:264)
at
org.apache.hudi.common.table.read.HoodieFileGroupReader.makeBaseFileIterator(...)
at
org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge(FileGroupReaderBasedMergeHandle.java:296)
```
The bootstrap route goes through
`HoodieFileGroupReader.makeBootstrapBaseFileIteratorHelper` into the same chain.
### Cause
`toShreddedReadSchema` swaps each requested variant column for its on-disk
shredded form. At a shredded position `swapShreddedVariantsAt` takes the file's
schema wholesale, nullability included (`VariantSchemaUtils.java:733`), and
`swapShreddedVariantFields` then rebuilds the requested field around it with
`withSchema` (`:716`), which keeps the requested field's `null` default. A
non-null record schema with a `null` default is invalid Avro. Reading a
REQUIRED column through a nullable reader schema is otherwise a safe widening,
and Hudi does it for every other column type.
### Expected behavior
The upsert succeeds: the read schema keeps the requested side's nullability
at shredded positions (or drops the default when the swapped-in schema is
non-null), and the REQUIRED column is read as nullable.
### Environment
master 616f9ae86765, Spark 4.1, COW, default AVRO record type
(`HoodieAvroRecordMerger` -> `HoodieAvroParquetReader` for the old base file).
Found while porting shredded-variant conformance tests from parquet-testing
(corpus cases 131 and 138 hit the same path).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
