voonhous opened a new issue, #19442: URL: https://github.com/apache/hudi/issues/19442
### Describe the problem `Spark4VariantShreddingProvider` handles Avro FIXED `typed_value` leaves incorrectly in both directions; the behavior is latent since #18065 (`23fe7bfcd7e6`). - **Write**: `avroTypeToScalarType` maps FIXED to `BinaryType`, and `convertScalarToAvro` then returns a `ByteBuffer`. That record fails `ConvertingGenericData.INSTANCE.validate(...)` (the data model `HoodieAvroWriteSupport` hands to parquet-avro), and writing throws `ClassCastException: HeapByteBuffer cannot be cast to GenericFixed`. - **Read**: a FIXED column decodes to `GenericData$Fixed`, which the `(ByteBuffer)` cast in `AvroVariantRow.getBinary` rejects with a `ClassCastException`. - **No size check**: `avroTypeToScalarType` ignores `getFixedSize()`, so a payload of the wrong length shreds without complaint. - Same hole class for the spec-canonical uuid-on-`fixed(16)` encoding: shred writes a `String` (`UnresolvedUnionException` at write), and rebuild on `GenericData$Fixed` fails with `IllegalArgumentException: UUID string too large`. ### Reachability Not reachable from the shredding DDL (`HoodieAvroWriteSupport.parseSimpleType` only accepts int/long/string/double/float/boolean/binary/decimal(p,s)), but reachable when the table Avro schema declares a FIXED `typed_value` directly (the shredding-enabled, no-forced-schema branch of `generateEffectiveSchema` uses the schema as-is), and on the read path for files written by other engines. ### Suggested fix Either: 1. Return null for FIXED in `avroTypeToScalarType` (decline to shred; simplest and safe), or 2. Emit `GenericData.Fixed` in `convertScalarToAvro`, accept `GenericFixed` in `getBinary`/`getUuid`, and enforce `getFixedSize()`. The unit test pinning the broken behavior was removed in PR #19403 review; whichever fix lands should add a test asserting the chosen behavior (decline-to-residual or a real `GenericData.Fixed` round trip) plus `ConvertingGenericData.INSTANCE.validate` on the result. ### Related cleanups - The `Byte`/`Short`/`Integer` widening arms in `convertScalarToAvro` and `AvroVariantRow.getByte`/`getShort` are unreachable (Avro has no byte/short; `avroTypeToScalarType` only emits `IntegralSize.INT`/`LONG`) - delete them. - No test writes a shredded record through `HoodieAvroWriteSupport` to parquet and reads it back through `HoodieAvroParquetReader`; add functional coverage for the DDL-expressible `binary` and `decimal(10,2)` leaves so serde-boundary regressions cannot ship silently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
