andygrove opened a new issue, #280:
URL: https://github.com/apache/datafusion-spark/issues/280

   Part of #273 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `to_variant_object(expr)` converts a nested value into a `VARIANT`. The 
top-level type must be `ARRAY`, `MAP` or
   `STRUCT`, and nested values can be any type that can be cast to variant. 
Structs and maps become variant objects,
   which are unordered and sorted by key. Map keys must be strings. Any other 
input fails analysis with
   `DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION` (`srcType` = the input type, 
`targetType` = `"VARIANT"`). That includes
   scalars, intervals and maps with non-string keys. The function is 
null-intolerant, so SQL NULL input returns SQL
   NULL. Added in Spark 4.0.0.
   
   It is the only way to put structs and maps into a variant. `CAST(x AS 
VARIANT)` accepts atomic types and arrays of
   them, but rejects any struct or map at any depth. Both go through
   `VariantExpressionEvalUtils.castToVariant`, using a `VariantBuilder` that 
does not allow duplicate keys.
   
   ### Examples
   
   ```sql
   SELECT to_variant_object(named_struct('a', 1, 'b', 2));
   -- {"a":1,"b":2}
   SELECT to_variant_object(array(1, 2, 3));
   -- [1,2,3]
   SELECT to_variant_object(array(named_struct('a', 1)));
   -- [{"a":1}]
   SELECT to_variant_object(array(map('a', 2)));
   -- [{"a":2}]
   SELECT to_json(to_variant_object(map('z', 1, 'y', 2, 'x', 3)));
   -- {"x":3,"y":2,"z":1}
   SELECT to_json(to_variant_object(named_struct(
     'c', array('123', 'true', 'f'),
     'b', map('a', '123', 'b', 'true', 'c', 'f'),
     'd', map('a', named_struct('i', 132)),
     'a', named_struct('i', 0))));
   -- 
{"a":{"i":0},"b":{"a":"123","b":"true","c":"f"},"c":["123","true","f"],"d":{"a":{"i":132}}}
   SELECT to_variant_object(1);
   -- error DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION
   SELECT to_variant_object(map(1, 1));
   -- error DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION
   ```
   
   These come from the `ExpressionDescription` and 
`VariantExpressionSuite."cast to variant/to_variant_object"`. The
   suite uses literals, and the SQL above is equivalent.
   
   ### Edge cases to test
   
   - Element encoding (`buildVariant`):
     - TINYINT, SMALLINT, INT and BIGINT use the smallest of INT1, INT2, INT4 
and INT8 that fits the value. They do
       not follow the column type.
     - FLOAT stays FLOAT and DOUBLE stays DOUBLE.
     - DECIMAL uses DECIMAL4, DECIMAL8 or DECIMAL16, chosen by the value's 
precision and the column's scale. For
       example, a `DECIMAL(38,2)` value of `1.00` is stored as DECIMAL4 with 
scale 2.
     - STRING uses a short string up to 63 bytes, otherwise a long string. 
BINARY stays BINARY.
     - DATE is days. TIMESTAMP and TIMESTAMP_NTZ are microseconds.
     - Nested VARIANT values are copied with their keys re-added to the new 
dictionary.
   - How these render through `to_json` (from the suite): booleans and numbers 
print as Java `toString` (`4.0`,
     `5.0`), and decimals as plain strings. Binary `[1,2,3]` prints as 
`"AQID"`, and date 0 as `"1970-01-01"`.
     TIMESTAMP 0 prints as `"1970-01-01 00:00:00+00:00"` in UTC and 
`"1969-12-31 16:00:00-08:00"` in
     America/Los_Angeles, so it depends on `spark.sql.session.timeZone`. 
TIMESTAMP_NTZ 0 is always
     `"1970-01-01 00:00:00"`.
   - A NULL element, field or map value becomes a variant null 
(`[null,"a","b","c"]`). A top-level NULL is SQL NULL.
     An array of variants that holds NULLs works the same way.
   - Struct field order and map entry order are discarded, and keys are sorted 
with Java `String.compareTo` (UTF-16
     order). Struct field names become object keys exactly, case-sensitive.
   - Duplicate struct field names, such as `named_struct('a', 1, 'a', 2)`: the 
builder has `allowDuplicateKeys =
     false`, so by code reading this raises `VARIANT_DUPLICATE_KEY` directly, 
without the malformed-record wrapper
     that `parse_json` adds. Maps cannot hold duplicate keys under the default 
`spark.sql.mapKeyDedupPolicy`.
   - Map keys can be any string collation. Non-string keys fail at analysis, at 
any depth.
   - TIME (4.1+): `checkDataType` accepts every `DatetimeType`, which includes 
`TimeType`, but `buildVariant` has no
     TIME case. Check what Spark does with 
`to_variant_object(array(time'12:00:00'))` before deciding.
   - Intervals are rejected at analysis (SPARK-49985).
   - Exceeding the 128 MiB value or metadata limit raises `VARIANT_SIZE_LIMIT` 
from the builder.
   - Test literal and column inputs, empty arrays, maps and structs (`[]`, 
`{}`), and deep nesting.
   
   ### Differences across Spark versions
   
   - 3.x has no VARIANT.
   - No differences in `ToVariantObject` or `castToVariant` between 4.0.4, 
4.1.3 and 4.2.0. TIME types exist only
     from 4.1.
   
   ### Implementation notes
   
   - Spark: `ToVariantObject` in
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`,
 and
     `castToVariant` / `buildVariant` in `VariantExpressionEvalUtils.scala` in 
the same package.
     `VariantGet.checkDataType` is in the same file. The encoder is 
`common/variant/.../VariantBuilder.java`.
   - arrow-rs `parquet_variant_compute::cast_to_variant` covers most Arrow 
types, but its encoding differs from
     Spark's:
     - `Int32` becomes INT4 and `Decimal128` always becomes DECIMAL16, with no 
narrowing.
     - `ObjectBuilder` sorts keys by UTF-8 bytes.
     - The metadata builder may set `sorted_strings`.
     - It accepts types that Spark rejects (non-string map keys, unsigned 
integers, and others).
     Use it only as a building block. The type check and narrowing rules have 
to match Spark.
   - Spark's `DECIMAL` is always `Decimal128` in datafusion-spark, so the 
narrowing has to look at the value.
   - DataFusion core has no VARIANT support. This depends on the VARIANT Arrow 
representation proposed in the
     `parse_json` issue, and the same encoder can back `CAST(x AS VARIANT)`.
   - apache/datafusion-comet#5438 tracks native Variant support in Comet.
   - Put it in `crates/spark/src/function/variant/`. Add 
`crates/sqllogictest/test_files/variant/to_variant_object.slt`
     and document the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to