voonhous commented on issue #19442:
URL: https://github.com/apache/hudi/issues/19442#issuecomment-6165989504
**Leaf type map for `Spark4VariantShreddingProvider`** (master `61dde86`,
from code reading; not run)
- **Write:** `HoodieAvroWriteSupport` shreds a variant into the
`typed_value` leaf declared by the Avro schema (`avroTypeToScalarType`, then
`convertScalarToAvro`).
- **Read:** `HoodieVariantReconstruction` rebuilds the variant from a base
file. The leaf's Avro type comes from the footer converter, then the same
`avroTypeToScalarType` and the `AvroVariantRow` getters feed Spark's
`ShreddingUtils.rebuild`.
### Write: Avro `typed_value` type, then ScalarType, then the value handed
to the Avro writer
```mermaid
flowchart LR
WOK["boolean, int, long, float, double, string, bytes<br/>decimal on
bytes or fixed, date<br/>timestamp-micros, local-timestamp-micros<br/>uuid on
string"] --> WOKS["matching ScalarType"] --> WOKR["OK"]
WF["fixed, no logical type"] --> WFS["BinaryType"] --> WFV["byte[]
wrapped as ByteBuffer"] --> WFR["ClassCastException:<br/>ByteBuffer is not
GenericFixed<br/>size never checked"]
WU["uuid on fixed(16)"] --> WUS["UuidType"] --> WUV["String from
UUID.toString()"] --> WUR["UnresolvedUnionException"]
WT["time-millis on int<br/>time-micros on long"] -->
WTS["IntegralType<br/>no time branch"] --> WTV["Integer or Long"] -->
WTR["writes, but variant integers<br/>land in a TIME column"]
WD["timestamp-millis, local-timestamp-millis<br/>enum, map"] -->
WDS["null: not shredded"] --> WDR["value stays in the residual<br/>by design"]
classDef ok fill:#bbf7d0,stroke:#15803d,color:#111827
classDef bad fill:#fecaca,stroke:#b91c1c,color:#111827
classDef wrong fill:#fde68a,stroke:#b45309,color:#111827
classDef skip fill:#e5e7eb,stroke:#4b5563,color:#111827
class WOK,WOKS,WOKR ok
class WF,WFS,WFV,WFR,WU,WUS,WUV,WUR bad
class WT,WTS,WTV,WTR wrong
class WD,WDS,WDR skip
```
### Read: parquet `typed_value` type, then Avro type from the footer
converter, then ScalarType, then the rebuilt value
```mermaid
flowchart LR
ROK["boolean, int32, int64, float, double<br/>string, binary,
DATE<br/>DECIMAL on BINARY or FLBA, FLBA needs PR 20324<br/>TIMESTAMP(MICROS),
UUID on FLBA(16)"] --> ROKS["matching ScalarType"] --> ROKR["OK"]
RF["FLBA, no annotation"] --> RFA["fixed"] --> RFS["BinaryType"] -->
RFR["getBinary casts to ByteBuffer:<br/>ClassCastException on GenericFixed"]
RD["DECIMAL on INT32 or INT64"] --> RDA["int or long<br/>converter drops
DECIMAL"] --> RDS["IntegralType"] --> RDR["silent wrong value:<br/>unscaled
integer, 1.50 becomes 150"]
RT["TIME(MILLIS) or TIME(MICROS)"] --> RTA["time-millis or time-micros"]
--> RTS["IntegralType"] --> RTR["time rebuilt as an integer"]
RN["TIMESTAMP(NANOS)"] --> RNA["long<br/>converter drops it"] -->
RNS["IntegralType"] --> RNR["timestamp rebuilt as an integer"]
RM["TIMESTAMP(MILLIS)"] --> RMA["timestamp-millis"] --> RMS["null: no
ScalarType"] --> RMR["NPE in Spark rebuild<br/>when typed_value is non-null"]
classDef ok fill:#bbf7d0,stroke:#15803d,color:#111827
classDef bad fill:#fecaca,stroke:#b91c1c,color:#111827
classDef wrong fill:#fde68a,stroke:#b45309,color:#111827
class ROK,ROKS,ROKR ok
class RF,RFA,RFS,RFR,RD,RDA,RDS,RDR,RM,RMA,RMS,RMR bad
class RT,RTA,RTS,RTR,RN,RNA,RNS,RNR wrong
```
Green: correct. Red: fails or returns a wrong value. Amber: no error, but
the leaf's type is lost. Grey: not shredded, by design.
| Leaf | Direction | Outcome | Reachable from |
|---|---|---|---|
| `fixed`, no logical type | write | ClassCastException, no size check |
Hand-declared Avro schema only |
| uuid on `fixed(16)` | write | UnresolvedUnionException | Hand-declared
Avro schema only |
| FLBA, no annotation | read | ClassCastException | Files written by other
engines |
| DECIMAL on INT32 / INT64 | read | Silent wrong value (unscaled integer) |
Files written by other engines; Hudi writes decimals as FLBA |
| TIME (millis / micros) | write and read | Stored and rebuilt as integers |
Hand-declared Avro schema; other engines' files |
| TIMESTAMP(NANOS) | read | Rebuilt as an integer | Files written by other
engines |
| TIMESTAMP(MILLIS) | read | NPE in `ShreddingUtils.rebuild` | Files written
by other engines |
Notes:
- The first three rows are this issue's report; the rest turned up while
mapping. Only the DECIMAL on INT32 / INT64 row returns a wrong value without an
error.
- The read-side uuid failure in the original report (`GenericFixed` in
`getUuid`) is not reachable through files: the footer converter maps FLBA(16)
UUID to an Avro string, and parquet-avro's `FieldUUIDConverter` reads it as one.
- Declining plain `fixed` and uuid on `fixed` at write time covers the first
two rows only. For DECIMAL on INT and NANOS, the converter has already dropped
the annotation, so the provider cannot see it. TIME keeps its Avro logical
type, but Spark's `VariantSchema` has no time scalar to map it to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
