andygrove opened a new issue, #276:
URL: https://github.com/apache/datafusion-spark/issues/276

   Part of #273 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `variant_get(v, path[, type])` takes the sub-variant of `v` at `path` and 
casts it to `type`. `try_variant_get`
   does the same, but a failed cast returns NULL instead of raising an error.
   
   - `v` must be `VARIANT`. `path` is a `STRING` (any collation) and may be a 
column.
   - `type` must be a foldable string. It is parsed as DDL or JSON schema, so 
`'int'`, `'array<int>'` and
     `'struct<a int>'` all work. CHAR/VARCHAR are rejected.
   - Without `type` the result is `VARIANT`. The result is always nullable, 
whatever nullability `type` declares.
   - A NULL `v` or `path` returns NULL.
   - Allowed target types are numerics, boolean, string, binary, 
date/timestamp/timestamp_ntz, `VARIANT`, arrays of
     these, `MAP<STRING, ...>` and structs. Anything else fails analysis with
     `DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION`. That includes intervals and 
maps with non-string keys.
   - Both functions were added in Spark 4.0.0. They depend on the VARIANT Arrow 
representation proposed in the
     `parse_json` issue.
   
   The cast code (`VariantGet.cast`) is shared with `CAST(v AS type)` / 
`TRY_CAST` and the `v:path` extraction
   syntax, so implementing it here covers those too.
   
   ### Examples
   
   ```sql
   SELECT variant_get(parse_json('{"a": 1}'), '$.a', 'int');
   -- 1
   SELECT variant_get(parse_json('{"a": 1}'), '$.b', 'int');
   -- NULL
   SELECT variant_get(parse_json('[1, "2"]'), '$[1]', 'string');
   -- 2
   SELECT variant_get(parse_json('[1, "2"]'), '$[2]', 'string');
   -- NULL
   SELECT variant_get(parse_json('[1, "hello"]'), '$[1]');
   -- "hello"
   SELECT try_variant_get(parse_json('[1, "hello"]'), '$[1]', 'int');
   -- NULL
   SELECT variant_get(parse_json('{"a": 2147483648}'), '$.a', 'int');
   -- error INVALID_VARIANT_CAST: The variant value `2147483648` cannot be cast 
into `"INT"`. ...
   SELECT variant_get(parse_json('{"1": {"2": {"3": [4]}}}'), '$.1.2.3[0]', 
'int');
   -- 4
   SELECT variant_get(parse_json('{"1": {"2": {"3": [4]}}}'), '$.1.2.3[''0'']', 
'int');
   -- NULL
   SELECT variant_get(parse_json('1.10'), '$', 'string');
   -- 1.1
   SELECT try_variant_get(parse_json('[1, 2, 3, "hello"]'), '$', 'array<int>');
   -- [1,2,3,null]
   SELECT variant_get(parse_json('{"a": 1, "d": 2, "c": 3}'), '$', 'struct<a 
int, b int, c int>');
   -- {"a":1,"b":null,"c":3}
   ```
   
   These come from the `ExpressionDescription` and `VariantExpressionSuite`. 
The struct and array results there are
   asserted as rows, so the display form shown is the usual Spark rendering.
   
   ### Edge cases to test
   
   - Path grammar (`VariantPathParser`):
     - A path is `$` followed by any number of `.key`, `['key']`, `["key"]` or 
`[N]` segments, with no whitespace
       skipping.
     - `.key` takes any characters except `.` and `[`, so `$.zip code` and 
`$.fb:testid` are valid.
     - Quoted keys cannot contain their own quote character, and there are no 
escapes.
     - An empty key needs `['']` or `[""]`.
     - Invalid paths raise `INVALID_VARIANT_GET_PATH` with `` functionName = 
`variant_get` ``, even in
       `try_variant_get`. Examples are `""`, `.a`, `$1`, `$[-1]` and `$['"]`. 
For a literal path the error is raised
       at first evaluation, not at analysis.
     - `$[99999999999]` matches `\d+` and then `toInt` overflows. Pin down what 
Spark does there.
   - Lookups:
     - Key lookup is case-sensitive (`$.A` does not match `a`).
     - A key segment on a non-object returns NULL, and so does an index segment 
on a non-array or out of range.
       Cover objects with more than 32 fields (binary search) and more than 255 
elements or fields.
   - JSON null versus SQL NULL:
     - A variant null cast to any concrete type, including STRING, is SQL NULL.
     - With no `type` (VARIANT), the variant null is kept, so 
`is_variant_null(variant_get(..., '$.a'))` is true.
   - A VARIANT result is re-encoded so it only carries the dictionary keys it 
uses. It is not a byte slice of the
     input.
   - Scalar casts:
     - A typed literal is built for the variant's type, then cast with Spark 
`Cast` in TRY mode, but only when
       `Cast.canAnsiCast` allows the pair. A NULL result or a disallowed pair 
counts as an invalid cast.
     - `spark.sql.ansi.enabled` has no effect. Both settings throw for 
`variant_get`.
   - Spark test expectations to copy:
     - `"true"` to boolean is true; `" t "` to boolean is true.
     - `"true"` to int is an error.
     - `true` to `decimal(18,17)` is 1, and to `decimal(18,18)` is an error.
     - `128` to tinyint is an error.
     - `1.15` to `decimal(2,1)` is 1.2 (HALF_UP).
     - `2147483647.999` to int is 2147483647 (truncation).
     - `"inf"` and `"nan"` to double.
     - `1E308` to float is Infinity.
     - `"1970-03-01"` to date is 59.
   - Timestamps:
     - LONG and DECIMAL to TIMESTAMP are read as seconds with exact overflow 
checks: `9223372036854` works and
       `9223372036855` is an error.
     - Integer to TIMESTAMP_NTZ is an error.
     - String to timestamp uses `spark.sql.session.timeZone`.
   - Decimal to string strips trailing zeros (`-1.00` gives `-1`). Object or 
array to string gives the compact JSON
     (`Variant.toJson`). Doubles print like Java `Double.toString`, for example 
`1.0E19`. Object or array to any
     other scalar is an error.
   - UUID values (from external data) can only be cast to STRING.
   - Nested targets:
     - Struct fields are matched by exact name. Missing fields are NULL and 
extra fields are ignored.
     - Maps take every object field.
     - Arrays cast element by element.
     - `variant_get` fails on the first bad element. `try_variant_get` sets 
only that element to NULL, for example
       `Row(1, null)`.
     - The error's `value` is the JSON of the failing sub-value (`"hello"`, 
`{}`), not the whole input.
   - Mix literal and column `path` arguments, and test `v` coming from 
`parse_json` and from columns.
   
   ### Differences across Spark versions
   
   - 3.x has no VARIANT.
   - 4.0.4, 4.1.3 and 4.2.0 have the same `VariantGet`, except that 4.2.0 
matches CHAR/VARCHAR by type class instead
     of by pattern. The behavior is the same.
   
   ### Implementation notes
   
   - Spark: `VariantGet`, `VariantGet.cast`, `VariantPathParser` and the 
builders in
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`.
 Value
     access is in 
`common/variant/src/main/java/org/apache/spark/types/variant/Variant.java` 
(`getFieldByKey`,
     `getElementAtIndex`, `toJson`).
   - arrow-rs `parquet_variant_compute::variant_get` is a different design:
     - Its `VariantPath` syntax has no `$` and treats quoted brackets as fields.
     - Casting follows Arrow `CastOptions`, not Spark's rules.
     - Its JSON output formats floats differently (`{f}`, so `1e10` prints as 
`10000000000`).
     Reuse `parquet_variant::Variant` for decoding and the existing Spark cast 
kernels in datafusion-spark for
     scalar conversions, but write the path parser, the cast dispatch and 
`toJson` to match Spark.
   - DataFusion core has no VARIANT support.
   - apache/datafusion-comet#5438 tracks native Variant support in Comet.
   - Put this in `crates/spark/src/function/variant/` with the other VARIANT 
functions. Because `type` determines the
     return type, the UDF needs `return_field_from_args`.
   - Add `crates/sqllogictest/test_files/variant/variant_get.slt` and document 
both names with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to