voonhous opened a new issue, #20322: URL: https://github.com/apache/hudi/issues/20322
### Why Shredded variant writes are in master with write shredding and schema inference on by default. Before this reaches production tables we want one artifact that answers, for every value type a variant can hold: how the value is tagged on write, where it lands on disk under each layout, and what comes back on read. Today's coverage (`TestVariantDataType`, `TestVariantShreddingMixedLayouts`) drives a handful of int, string and decimal keys through the layouts; no test covers the full type set, and nothing pins the numeric re-typing that shredding is allowed to perform. ### How a variant value gets its type Variant is not duck-typed on read. Every value carries a type tag fixed by the producer and Hudi never re-infers it. For `parse_json`: - JSON integer: narrowest of int8/int16/int32/int64 that fits; beyond 64 bits, decimal if 38 digits or fewer, else double. - JSON number with a decimal point and no exponent: DECIMAL (decimal4/8/16 by precision). `1.5` is decimal(2,1), never float. - JSON number with an exponent: DOUBLE. - Strings stay strings, even when they look like dates or numbers. date, timestamp, timestamp_ntz, float and binary are reachable only through `cast(<typed column> as variant)`; uuid only through the raw builder. Integer width is not observable at the SQL layer (`schema_of_variant` reports BIGINT for all four) and is re-narrowed on shredded reconstruction. ### Re-typing that shredding may perform Both writers (row writer via `SparkShreddingUtils`, Avro path via `Spark4VariantShreddingProvider`) return `allowNumericScaleChanges = true`. An integer `3` shredded into a decimal leaf comes back tagged DECIMAL; a decimal `3.0` shredded into a bigint leaf comes back LONG. Default inference triggers this on its own when a key holds `3` in one row and `3.25` in another. This is spec-legal but must be documented as expected drift. ### Test design One table, one variant column, one object per row with a key per input type, so a single inferred schema types every key independently. Observe each (key, layout, record type) at three layers: 1. Logical: `schema_of_variant`, `cast(v as string)`, `variant_get` into the declared type. 2. Binary: the value header (physical type id and width) via `VariantUtil`. 3. Physical: `typed_value` leaf vs `value` residual, and the Parquet primitive/logical type of the leaf. Axes: layout (unshredded, forced matching, forced mismatching, inferred); record type (AVRO, SPARK); boundary values per type; mixed-type keys (int+decimal coerces, int+string stays residual); non-Avro-identifier keys (Hudi declines them, Spark would shred); session time zone pinned. Differential oracle: the same DataFrame written through vanilla Spark Parquet shredding must match cell for cell. One pass of the full fixture through compaction, clustering, CDC and the bulk-insert row writer. The test emits a golden CSV checked into test resources; drift fails the build and the CSV is the published matrix. ### Related - #19442: FIXED `typed_value` broken on the Avro paths. - Avro path declines `timestamp-millis` leaves (stays residual) and its forced-schema DDL accepts only int, long, string, double, float, boolean, binary, decimal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
