andygrove opened a new issue, #278:
URL: https://github.com/apache/datafusion-spark/issues/278
Part of #273 and apache/datafusion-comet#7300.
### What the function does
`schema_of_variant(v)` returns a `STRING` describing the schema of a single
VARIANT value. The format is Spark's
SQL type syntax, except that objects print as `OBJECT<...>` instead of
`STRUCT<...>`. Variant objects are unordered
bags of fields, unlike structs. The argument must be `VARIANT` and the
result uses the default string type (and
collation). SQL NULL input returns NULL through `StaticInvoke` null
propagation. Added in Spark 4.0.0.
Type mapping (`SchemaOfVariant.schemaOf`):
| Variant type | Result |
| --- | --- |
| null | `VOID` |
| boolean | `BOOLEAN` |
| INT1, INT2, INT4, INT8 | `BIGINT` |
| string | `STRING` |
| double | `DOUBLE` |
| float | `FLOAT` |
| decimal | `DECIMAL(p,s)`, from the value after trailing zeros are stripped
|
| date, timestamp, timestamp_ntz | `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ` |
| binary | `BINARY` |
| uuid | `UUID` |
| object | `OBJECT<name: type, ...>` in key order |
| array | `ARRAY<merged element type>` |
Array elements are merged with `JsonInferSchema.compatibleType(t1, t2,
VariantType)`, so incompatible types become
`VARIANT`.
### Examples
```sql
SELECT schema_of_variant(parse_json('null'));
-- VOID
SELECT schema_of_variant(parse_json('[{"b":true,"a":0}]'));
-- ARRAY<OBJECT<a: BIGINT, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('1.0')),
schema_of_variant(parse_json('10.10')), schema_of_variant(parse_json('-0.0'));
-- DECIMAL(1,0) DECIMAL(3,1) DECIMAL(1,0)
SELECT schema_of_variant(parse_json('9223372036854775808'));
-- DECIMAL(19,0)
SELECT
schema_of_variant(parse_json('9999999999999999999.99999999999999999999'));
-- DOUBLE
SELECT schema_of_variant(parse_json('"2000-01-01"'));
-- STRING
SELECT schema_of_variant(parse_json('[]'));
-- ARRAY<VOID>
SELECT schema_of_variant(parse_json('[null, 1, 1.1]'));
-- ARRAY<DECIMAL(21,1)>
SELECT schema_of_variant(parse_json('[1.1,
11111111111111111111111111111111111111]'));
-- ARRAY<DOUBLE>
SELECT schema_of_variant(parse_json('[1, "1"]'));
-- ARRAY<VARIANT>
SELECT schema_of_variant(parse_json('[{"c": ""}, {"a": null}, {"b": 1}]'));
-- ARRAY<OBJECT<a: VOID, b: BIGINT, c: STRING>>
SELECT schema_of_variant(parse_json('[{"a": 1, "b": null}, {"b": true, "a":
1E0}]'));
-- ARRAY<OBJECT<a: DOUBLE, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('{"STRUCT": {"!special!": true}}'));
-- OBJECT<STRUCT: OBJECT<`!special!`: BOOLEAN>>
```
These come from the `ExpressionDescription`, `VariantEndToEndSuite` and
`VariantSuite`.
### Edge cases to test
- Merge table from `VariantExpressionSuite."schema_of_variant - schema
merge"`:
| Pair | Result |
| --- | --- |
| `BIGINT`/`DOUBLE` | `DOUBLE` |
| `BIGINT`/`FLOAT` | `FLOAT` |
| `BIGINT`/`DECIMAL(6,3)` | `DECIMAL(23,3)` |
| `DOUBLE`/`FLOAT` | `DOUBLE` |
| `DOUBLE`/`DECIMAL` | `DOUBLE` |
| `FLOAT`/`DECIMAL` | `DOUBLE` |
| `DATE`/`TIMESTAMP` | `TIMESTAMP` |
| `DATE`/`TIMESTAMP_NTZ` | `TIMESTAMP_NTZ` |
| `TIMESTAMP`/`TIMESTAMP_NTZ` | `TIMESTAMP` |
| `ARRAY<BIGINT>`/`ARRAY<DOUBLE>` | `ARRAY<DOUBLE>` |
| `OBJECT<a: STRING>`/`OBJECT<a: BOOLEAN, b: BIGINT>` | `OBJECT<a:
VARIANT, b: BIGINT>` |
`VOID` merged with anything gives the other type, and every other pair
gives `VARIANT`, including
`STRING`/`BIGINT` and `BINARY`/`STRING`.
- Decimal merging: the scale is `max(s1, s2)` and the integer range is
`max(p - s)`. When the total exceeds 38 the
result is `DOUBLE`. An integral type is widened to `DECIMAL(20,0)` first
(`[null, 1, 1.0]` gives
`DECIMAL(20,0)`, and `[123456.789, 123.456789]` gives `DECIMAL(12,6)`).
- Decimal precision comes from `stripTrailingZeros`, with a negative scale
normalized to 0: `1.00` gives
`DECIMAL(1,0)` and `10.00` gives `DECIMAL(2,0)`. It does not come from the
stored scale.
- Field names are quoted with backticks unless they match
`[a-zA-Z_][a-zA-Z0-9_]*`, and embedded backticks are
doubled (``OBJECT<`it's`: ...>``). Keywords like `STRUCT` are not quoted.
- Struct merging inside arrays first goes through
`TypeCoercion.findTightestCommonType`. For structs of equal
length, that compares names with `spark.sql.caseSensitive` (default false)
and keeps the first name. Only if it
fails does Spark fall back to a case-sensitive sorted merge. By code
reading, `[{"a":1},{"A":2}]` gives
`ARRAY<OBJECT<a: BIGINT>>` by default. Verify this before asserting it.
- Object fields must be strictly increasing by Java `String.compareTo`
(UTF-16 order). Otherwise Spark throws
`MALFORMED_VARIANT`, and duplicates hit the same check. Variants written
by other tools that sort by UTF-8 bytes
can trip this.
- An unknown primitive type id raises `UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT`
(id 25 in the suite).
- UUID shows as `UUID`, and merging it with a variant null keeps `UUID`
(`ARRAY<UUID>`). These values can only come
from external data.
- Test NULL input, column input, and deep nesting.
### Differences across Spark versions
- 3.x has no VARIANT.
- No logic changes between 4.0.4, 4.1.3 and 4.2.0. 4.2 adds
`GEOMETRY`/`GEOGRAPHY` cases to
`findTightestCommonType`, but variants never produce those types.
### Implementation notes
- Spark: `SchemaOfVariant` (`schemaOf`, `mergeSchema`, `printSchema`) in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`.
The merge rules are `JsonInferSchema.compatibleType` in
`.../catalyst/json/JsonInferSchema.scala` and
`TypeCoercion.findTightestCommonType` /
`TypeCoercionHelper.findTypeForComplex` in `.../catalyst/analysis/`.
The printer is `DataType.sql`, with `QuotingUtils.quoteIfNeeded` for field
names.
- DataFusion core and arrow-rs have nothing equivalent. Reuse
`parquet_variant::Variant` to walk values and write
the Spark type lattice and printer directly. Share them with
`schema_of_variant_agg`, and with
`schema_of_json` / `from_json`'s merge logic if they land.
- This depends on the VARIANT Arrow representation in the `parse_json` issue.
- apache/datafusion-comet#5438 tracks native Variant support in Comet.
- Put it in `crates/spark/src/function/variant/`. Add
`crates/sqllogictest/test_files/variant/schema_of_variant.slt`
and document the function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]