voonhous commented on issue #20322: URL: https://github.com/apache/hudi/issues/20322#issuecomment-6164033028
Results from the first run of the matrix suite (`TestVariantTypeMatrix`, 10 tests, Spark 4.1 profile, both record types plus a vanilla Spark parquet oracle). The suite and the golden CSV are on branch `variant-type-matrix`; PR to follow. Full table at the bottom. **What held** - Every parse_json tag rule listed above came out as stated, on the literal and after an unshredded write on both legs. Strings that look like dates or ints stay STRING. - Hudi agrees with vanilla Spark parquet shredding cell for cell on the logical and binary layers, for every key, layout and leg. - Integer width is canonical after a shredded round trip: a raw INT8-encoded `5` stays INT8 in a residual and comes back INT1 from a typed leaf. - A mismatching forced schema sends every key to the per-field residual and the read equals the unshredded read byte for byte. - MOR update plus compaction, and row-writer bulk insert plus clustering, keep the logical layer of the whole fixture on both legs. - #19442 did not reproduce on this path: inferred FIXED decimal leaves, including decimal(38,38) and decimal(38,0), round-trip on the AVRO leg. **Re-typing, as predicted** - `mixed_int_dec` (3 in two rows, 3.25 in two): inference picks decimal(18,2); the int rows read back `DECIMAL(1,0)` with header DECIMAL4/s2 instead of BIGINT/INT1. `cast(v as string)` and `variant_get(..., 'int')` still return 3. - `dec_1_0` forced into a bigint leaf reads back BIGINT/INT1 instead of DECIMAL(1,0). - `mixed_dec_scale`: 1.5 comes back DECIMAL4/s2 after a decimal(18,2) leaf. Logically identical, visible only in the header. **Findings that need a decision** 1. Decimal typed_value leaves do not follow the shredding spec's physical-type table. The spec says decimal4 must be INT32 and decimal8 INT64 (BYTE_ARRAY or FIXED only for decimal16). Hudi writes FIXED_LEN_BYTE_ARRAY on the SPARK leg (minimal length when forced, 8 bytes for inferred decimal(18,s)) and BINARY on the AVRO leg, for every precision. Vanilla Spark writes INT32/INT64. Spark and Hudi read Hudi's files fine because parquet-mr decimal converters accept any encoding, but a reader that validates the shredding schema against the spec table will reject these leaves. Root cause is `HoodieRowParquetWriteSupport` always writing decimals in the legacy FIXED format, and parquet-avro's bytes decimal on the AVRO path. 2. A forced `tinyint` leaf fails the SPARK-leg write: `HoodieRowParquetWriteSupport.makeWriter` handles ShortType but has no ByteType branch, so `makeShreddedVariantWriter` throws `UnsupportedOperationException: Unsupported type: ByteType`. Inference never emits tinyint (ints widen to bigint), so the default path is unaffected. Pinned by a by-product test; the matrix carries those cells as `n/a`. 3. Hudi-only divergence: keys that are not Avro identifiers (`first-name`, `1abc`, non-ASCII) are dropped from the inferred schema and stay in the root residual on both legs. Vanilla Spark shreds them. Reads are correct either way. 4. The AVRO leg's forced DDL cannot express date, timestamp, timestamp_ntz, struct or array leaves (smallint is widened to int), so those cells are `n/a: not forceable on AVRO` and the key reads from the root residual. Inference does type them (as INT64 / INT32:DATE / INT64:TIMESTAMP(MICROS)), so this only affects declared schemas. **Spark-side observations (not Hudi)** - `-0.0` loses its sign in parse_json (BigDecimal) and reads back as `0`, DECIMAL(1,0). - `cast(binary variant as string)` renders the raw bytes, not base64. - Spark 4.1.1's `spark.sql.variant.forceShreddingSchemaForTest` throws ArrayIndexOutOfBoundsException in `ParquetWriteSupport.writeFields` on any file with four or more columns. The oracle writes one `(id, column)` file per variant column to work around it. - `pushVariantIntoScan` hoists `variant_get` out of a CASE, so a strict per-row cast on a mixed-type key fails for the other rows' type; the matrix uses `try_variant_get` for those keys. **Not covered yet** - Reading Hudi's shredded files with the vanilla Spark reader (the oracle compares Hudi-on-Hudi with Spark-on-Spark, not cross-reads). This is the check that would surface item 1 as a hard failure if Spark ever validates physical types. - CDC read of the fixture, UUID as a forced leaf, timestamp-millis leaves (not reachable from SQL since Hudi maps TimestampType to micros). <details> <summary>Full matrix: header tag and schema_of_variant after an unshredded write, inferred leaf and slot, forced leaf (SPARK leg; AVRO and PARQUET legs are in the CSV)</summary> | key | producer | header tag | schema_of_variant | inferred leaf (Hudi SPARK) | slot | forced type / leaf (SPARK) | |---|---|---|---|---|---|---| | tinyint_5 | cast/to_variant_object | INT1 | BIGINT | INT64 | typed | n/a: tinyint leaf fails the Hudi SPARK writer / absent | | smallint_300 | cast/to_variant_object | INT2 | BIGINT | INT64 | typed | smallint / INT32:INTEGER(16,true) | | int_70000 | cast/to_variant_object | INT4 | BIGINT | INT64 | typed | int / INT32 | | bigint_big | cast/to_variant_object | INT8 | BIGINT | INT64 | typed | bigint / INT64 | | float_0_1 | cast/to_variant_object | FLOAT | FLOAT | FLOAT | typed | float / FLOAT | | double_0_1 | cast/to_variant_object | DOUBLE | DOUBLE | DOUBLE | typed | double / DOUBLE | | double_nan | cast/to_variant_object | DOUBLE | DOUBLE | DOUBLE | typed | double / DOUBLE | | double_inf | cast/to_variant_object | DOUBLE | DOUBLE | DOUBLE | typed | double / DOUBLE | | decimal_10_2 | cast/to_variant_object | DECIMAL8/s2 | DECIMAL(10,2) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(10,2) / FIXED_LEN_BYTE_ARRAY(5):DECIMAL(10,2) | | decimal_38_10 | cast/to_variant_object | DECIMAL16/s10 | DECIMAL(38,10) | FIXED_LEN_BYTE_ARRAY(16):DECIMAL(38,10) | typed | decimal(38,10) / FIXED_LEN_BYTE_ARRAY(16):DECIMAL(38,10) | | string_s | cast/to_variant_object | STRING | STRING | BINARY:STRING | typed | string / BINARY:STRING | | binary_x0102 | cast/to_variant_object | BINARY | BINARY | BINARY | typed | binary / BINARY | | date_2024_01_15 | cast/to_variant_object | DATE | DATE | INT32:DATE | typed | date / INT32:DATE | | ts_epoch_micro | cast/to_variant_object | TIMESTAMP | TIMESTAMP | INT64:TIMESTAMP(MICROS,true) | typed | timestamp / INT64:TIMESTAMP(MICROS,true) | | ts_pre1970 | cast/to_variant_object | TIMESTAMP | TIMESTAMP | INT64:TIMESTAMP(MICROS,true) | typed | timestamp / INT64:TIMESTAMP(MICROS,true) | | ts_ntz | cast/to_variant_object | TIMESTAMP_NTZ | TIMESTAMP_NTZ | INT64:TIMESTAMP(MICROS,false) | typed | timestamp_ntz / INT64:TIMESTAMP(MICROS,false) | | struct_xy | cast/to_variant_object | OBJECT | OBJECT<x: BIGINT, y: STRING> | OBJECT | typed | struct<x: int, y: string> / OBJECT | | array_int | cast/to_variant_object | ARRAY | ARRAY<BIGINT> | LIST | typed | array<int> / LIST | | map_str_int | cast/to_variant_object | OBJECT | OBJECT<a: BIGINT, b: BIGINT> | OBJECT | typed | struct<a: int, b: int> / OBJECT | | int8_0 | parse_json | INT1 | BIGINT | INT64 | typed | n/a: tinyint leaf fails the Hudi SPARK writer / absent | | int8_127 | parse_json | INT1 | BIGINT | INT64 | typed | n/a: tinyint leaf fails the Hudi SPARK writer / absent | | int16_128 | parse_json | INT2 | BIGINT | INT64 | typed | smallint / INT32:INTEGER(16,true) | | int16_32767 | parse_json | INT2 | BIGINT | INT64 | typed | smallint / INT32:INTEGER(16,true) | | int32_32768 | parse_json | INT4 | BIGINT | INT64 | typed | int / INT32 | | int32_max | parse_json | INT4 | BIGINT | INT64 | typed | int / INT32 | | int64_min_above_int32 | parse_json | INT8 | BIGINT | INT64 | typed | bigint / INT64 | | int64_max | parse_json | INT8 | BIGINT | INT64 | typed | bigint / INT64 | | dec19_2pow63 | parse_json | DECIMAL16/s0 | DECIMAL(19,0) | FIXED_LEN_BYTE_ARRAY(16):DECIMAL(38,0) | typed | decimal(19,0) / FIXED_LEN_BYTE_ARRAY(9):DECIMAL(19,0) | | dbl_39digits | parse_json | DOUBLE | DOUBLE | DOUBLE | typed | double / DOUBLE | | dec_1_5 | parse_json | DECIMAL4/s1 | DECIMAL(2,1) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,1) | typed | decimal(2,1) / FIXED_LEN_BYTE_ARRAY(1):DECIMAL(2,1) | | dec_1_0 | parse_json | DECIMAL4/s1 | DECIMAL(1,0) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,1) | typed | bigint / INT64 | | dec_neg0_0 | parse_json | DECIMAL4/s1 | DECIMAL(1,0) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,1) | typed | decimal(1,1) / FIXED_LEN_BYTE_ARRAY(1):DECIMAL(1,1) | | dec_0_1 | parse_json | DECIMAL4/s1 | DECIMAL(1,1) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,1) | typed | decimal(1,1) / FIXED_LEN_BYTE_ARRAY(1):DECIMAL(1,1) | | dec_p38_s10 | parse_json | DECIMAL16/s10 | DECIMAL(38,10) | FIXED_LEN_BYTE_ARRAY(16):DECIMAL(38,10) | typed | decimal(38,10) / FIXED_LEN_BYTE_ARRAY(16):DECIMAL(38,10) | | dec_s38 | parse_json | DECIMAL16/s38 | DECIMAL(38,38) | FIXED_LEN_BYTE_ARRAY(16):DECIMAL(38,38) | typed | decimal(38,38) / FIXED_LEN_BYTE_ARRAY(16):DECIMAL(38,38) | | dbl_1e2 | parse_json | DOUBLE | DOUBLE | DOUBLE | typed | double / DOUBLE | | dbl_1_5E_3 | parse_json | DOUBLE | DOUBLE | DOUBLE | typed | double / DOUBLE | | str_s | parse_json | STRING | STRING | BINARY:STRING | typed | string / BINARY:STRING | | str_empty | parse_json | STRING | STRING | BINARY:STRING | typed | string / BINARY:STRING | | str_looks_like_date | parse_json | STRING | STRING | BINARY:STRING | typed | string / BINARY:STRING | | str_looks_like_int | parse_json | STRING | STRING | BINARY:STRING | typed | string / BINARY:STRING | | str_unicode | parse_json | STRING | STRING | BINARY:STRING | typed | string / BINARY:STRING | | bool_true | parse_json | BOOLEAN_TRUE | BOOLEAN | BOOLEAN | typed | boolean / BOOLEAN | | json_null | parse_json | NULL | VOID | residual-only | residual | - / absent | | obj_empty | parse_json | OBJECT | OBJECT<> | residual-only | residual | - / absent | | arr_empty | parse_json | ARRAY | ARRAY<VOID> | LIST | typed | array<bigint> / LIST | | arr_ints | parse_json | ARRAY | ARRAY<BIGINT> | LIST | typed | array<bigint> / LIST | | arr_mixed | parse_json | ARRAY | ARRAY<VARIANT> | LIST | typed | - / absent | | obj_nested | parse_json | OBJECT | OBJECT<x: OBJECT<y: BIGINT>> | OBJECT | typed | struct<x: struct<y: bigint>> / OBJECT | | mixed_int_dec [row 0] | parse_json | INT1 | BIGINT | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_int_dec [row 1] | parse_json | INT1 | BIGINT | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_int_dec [row 2] | parse_json | DECIMAL4/s2 | DECIMAL(3,2) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_int_dec [row 3] | parse_json | DECIMAL4/s2 | DECIMAL(3,2) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_int_str [row 0] | parse_json | INT1 | BIGINT | residual-only | residual | - / absent | | mixed_int_str [row 1] | parse_json | INT1 | BIGINT | residual-only | residual | - / absent | | mixed_int_str [row 2] | parse_json | STRING | STRING | residual-only | residual | - / absent | | mixed_int_str [row 3] | parse_json | STRING | STRING | residual-only | residual | - / absent | | mixed_dec_scale [row 0] | parse_json | DECIMAL4/s1 | DECIMAL(2,1) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_dec_scale [row 1] | parse_json | DECIMAL4/s1 | DECIMAL(2,1) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_dec_scale [row 2] | parse_json | DECIMAL4/s2 | DECIMAL(3,2) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_dec_scale [row 3] | parse_json | DECIMAL4/s2 | DECIMAL(3,2) | FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | typed | decimal(18,2) / FIXED_LEN_BYTE_ARRAY(8):DECIMAL(18,2) | | mixed_null_int [row 0] | parse_json | NULL | VOID | INT64 | residual | bigint / INT64 | | mixed_null_int [row 1] | parse_json | NULL | VOID | INT64 | residual | bigint / INT64 | | mixed_null_int [row 2] | parse_json | INT1 | BIGINT | INT64 | typed | bigint / INT64 | | mixed_null_int [row 3] | parse_json | INT1 | BIGINT | INT64 | typed | bigint / INT64 | | first-name | parse_json | STRING | STRING | absent | root-residual | - / absent | | 1abc | parse_json | INT1 | BIGINT | absent | root-residual | - / absent | | cl<e-acute> | parse_json | STRING | STRING | absent | root-residual | - / absent | | value | parse_json | INT1 | BIGINT | INT64 | typed | bigint / INT64 | | typed_value | parse_json | STRING | STRING | BINARY:STRING | typed | string / BINARY:STRING | | metadata | parse_json | INT1 | BIGINT | INT64 | typed | bigint / INT64 | | int64_5 | raw builder | INT8 | BIGINT | INT64 | typed | bigint / INT64 | | uuid_raw | raw builder | UUID | UUID | residual-only | residual | - / absent | </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
