andygrove opened a new issue, #276:
URL: https://github.com/apache/datafusion-spark/issues/276
Part of #273 and apache/datafusion-comet#7300.
### What the function does
`variant_get(v, path[, type])` takes the sub-variant of `v` at `path` and
casts it to `type`. `try_variant_get`
does the same, but a failed cast returns NULL instead of raising an error.
- `v` must be `VARIANT`. `path` is a `STRING` (any collation) and may be a
column.
- `type` must be a foldable string. It is parsed as DDL or JSON schema, so
`'int'`, `'array<int>'` and
`'struct<a int>'` all work. CHAR/VARCHAR are rejected.
- Without `type` the result is `VARIANT`. The result is always nullable,
whatever nullability `type` declares.
- A NULL `v` or `path` returns NULL.
- Allowed target types are numerics, boolean, string, binary,
date/timestamp/timestamp_ntz, `VARIANT`, arrays of
these, `MAP<STRING, ...>` and structs. Anything else fails analysis with
`DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION`. That includes intervals and
maps with non-string keys.
- Both functions were added in Spark 4.0.0. They depend on the VARIANT Arrow
representation proposed in the
`parse_json` issue.
The cast code (`VariantGet.cast`) is shared with `CAST(v AS type)` /
`TRY_CAST` and the `v:path` extraction
syntax, so implementing it here covers those too.
### Examples
```sql
SELECT variant_get(parse_json('{"a": 1}'), '$.a', 'int');
-- 1
SELECT variant_get(parse_json('{"a": 1}'), '$.b', 'int');
-- NULL
SELECT variant_get(parse_json('[1, "2"]'), '$[1]', 'string');
-- 2
SELECT variant_get(parse_json('[1, "2"]'), '$[2]', 'string');
-- NULL
SELECT variant_get(parse_json('[1, "hello"]'), '$[1]');
-- "hello"
SELECT try_variant_get(parse_json('[1, "hello"]'), '$[1]', 'int');
-- NULL
SELECT variant_get(parse_json('{"a": 2147483648}'), '$.a', 'int');
-- error INVALID_VARIANT_CAST: The variant value `2147483648` cannot be cast
into `"INT"`. ...
SELECT variant_get(parse_json('{"1": {"2": {"3": [4]}}}'), '$.1.2.3[0]',
'int');
-- 4
SELECT variant_get(parse_json('{"1": {"2": {"3": [4]}}}'), '$.1.2.3[''0'']',
'int');
-- NULL
SELECT variant_get(parse_json('1.10'), '$', 'string');
-- 1.1
SELECT try_variant_get(parse_json('[1, 2, 3, "hello"]'), '$', 'array<int>');
-- [1,2,3,null]
SELECT variant_get(parse_json('{"a": 1, "d": 2, "c": 3}'), '$', 'struct<a
int, b int, c int>');
-- {"a":1,"b":null,"c":3}
```
These come from the `ExpressionDescription` and `VariantExpressionSuite`.
The struct and array results there are
asserted as rows, so the display form shown is the usual Spark rendering.
### Edge cases to test
- Path grammar (`VariantPathParser`):
- A path is `$` followed by any number of `.key`, `['key']`, `["key"]` or
`[N]` segments, with no whitespace
skipping.
- `.key` takes any characters except `.` and `[`, so `$.zip code` and
`$.fb:testid` are valid.
- Quoted keys cannot contain their own quote character, and there are no
escapes.
- An empty key needs `['']` or `[""]`.
- Invalid paths raise `INVALID_VARIANT_GET_PATH` with `` functionName =
`variant_get` ``, even in
`try_variant_get`. Examples are `""`, `.a`, `$1`, `$[-1]` and `$['"]`.
For a literal path the error is raised
at first evaluation, not at analysis.
- `$[99999999999]` matches `\d+` and then `toInt` overflows. Pin down what
Spark does there.
- Lookups:
- Key lookup is case-sensitive (`$.A` does not match `a`).
- A key segment on a non-object returns NULL, and so does an index segment
on a non-array or out of range.
Cover objects with more than 32 fields (binary search) and more than 255
elements or fields.
- JSON null versus SQL NULL:
- A variant null cast to any concrete type, including STRING, is SQL NULL.
- With no `type` (VARIANT), the variant null is kept, so
`is_variant_null(variant_get(..., '$.a'))` is true.
- A VARIANT result is re-encoded so it only carries the dictionary keys it
uses. It is not a byte slice of the
input.
- Scalar casts:
- A typed literal is built for the variant's type, then cast with Spark
`Cast` in TRY mode, but only when
`Cast.canAnsiCast` allows the pair. A NULL result or a disallowed pair
counts as an invalid cast.
- `spark.sql.ansi.enabled` has no effect. Both settings throw for
`variant_get`.
- Spark test expectations to copy:
- `"true"` to boolean is true; `" t "` to boolean is true.
- `"true"` to int is an error.
- `true` to `decimal(18,17)` is 1, and to `decimal(18,18)` is an error.
- `128` to tinyint is an error.
- `1.15` to `decimal(2,1)` is 1.2 (HALF_UP).
- `2147483647.999` to int is 2147483647 (truncation).
- `"inf"` and `"nan"` to double.
- `1E308` to float is Infinity.
- `"1970-03-01"` to date is 59.
- Timestamps:
- LONG and DECIMAL to TIMESTAMP are read as seconds with exact overflow
checks: `9223372036854` works and
`9223372036855` is an error.
- Integer to TIMESTAMP_NTZ is an error.
- String to timestamp uses `spark.sql.session.timeZone`.
- Decimal to string strips trailing zeros (`-1.00` gives `-1`). Object or
array to string gives the compact JSON
(`Variant.toJson`). Doubles print like Java `Double.toString`, for example
`1.0E19`. Object or array to any
other scalar is an error.
- UUID values (from external data) can only be cast to STRING.
- Nested targets:
- Struct fields are matched by exact name. Missing fields are NULL and
extra fields are ignored.
- Maps take every object field.
- Arrays cast element by element.
- `variant_get` fails on the first bad element. `try_variant_get` sets
only that element to NULL, for example
`Row(1, null)`.
- The error's `value` is the JSON of the failing sub-value (`"hello"`,
`{}`), not the whole input.
- Mix literal and column `path` arguments, and test `v` coming from
`parse_json` and from columns.
### Differences across Spark versions
- 3.x has no VARIANT.
- 4.0.4, 4.1.3 and 4.2.0 have the same `VariantGet`, except that 4.2.0
matches CHAR/VARCHAR by type class instead
of by pattern. The behavior is the same.
### Implementation notes
- Spark: `VariantGet`, `VariantGet.cast`, `VariantPathParser` and the
builders in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`.
Value
access is in
`common/variant/src/main/java/org/apache/spark/types/variant/Variant.java`
(`getFieldByKey`,
`getElementAtIndex`, `toJson`).
- arrow-rs `parquet_variant_compute::variant_get` is a different design:
- Its `VariantPath` syntax has no `$` and treats quoted brackets as fields.
- Casting follows Arrow `CastOptions`, not Spark's rules.
- Its JSON output formats floats differently (`{f}`, so `1e10` prints as
`10000000000`).
Reuse `parquet_variant::Variant` for decoding and the existing Spark cast
kernels in datafusion-spark for
scalar conversions, but write the path parser, the cast dispatch and
`toJson` to match Spark.
- DataFusion core has no VARIANT support.
- apache/datafusion-comet#5438 tracks native Variant support in Comet.
- Put this in `crates/spark/src/function/variant/` with the other VARIANT
functions. Because `type` determines the
return type, the UDF needs `return_field_from_args`.
- Add `crates/sqllogictest/test_files/variant/variant_get.slt` and document
both names with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]