andygrove opened a new issue, #275:
URL: https://github.com/apache/datafusion-spark/issues/275
Part of #273 and apache/datafusion-comet#7300.
### What the function does
`parse_json(jsonStr)` parses a JSON string into a `VARIANT` value. If the
input is not valid JSON, it raises an
error. `try_parse_json(jsonStr)` does the same but returns NULL for
malformed input, duplicate keys, and the size
limit. Both take exactly one `STRING` argument (any collation, including
trim collations). SQL `NULL` input
returns SQL `NULL`. `parse_json` is non-nullable for non-null input and
`try_parse_json` is nullable. Both were
added in Spark 4.0.0 (`variant_funcs`). Spark 3.x has no `VARIANT` type.
This is the first VARIANT function, so this issue also covers how
datafusion-spark represents the type. Every
other variant function (`variant_get`, `is_variant_null`,
`schema_of_variant`, `schema_of_variant_agg`,
`to_variant_object`) needs that representation.
A Spark VARIANT is two binaries. Spark implements the Parquet
`VariantEncoding.md` spec as of parquet-format
commit `37b6e8b8` (see `common/variant/README.md`):
- **metadata**: a header byte (version `1` in the low 4 bits, bit 4 =
`sorted_strings`, bits 6-7 =
`offset_size - 1`), the dictionary size, `size + 1` offsets, then the
UTF-8 key bytes.
- **value**: a header byte whose low 2 bits are the basic type (primitive,
short string, object, array) and whose
upper 6 bits are type info, followed by the content.
### Examples
```sql
SELECT parse_json('{"a":1,"b":0.8}');
-- {"a":1,"b":0.8}
SELECT try_parse_json('{"a":1,');
-- NULL
SELECT to_json(parse_json('[0.0, 1.00, 1.10, 1.23]'));
-- [0,1,1.1,1.23]
SELECT to_json(parse_json('1.0E10'));
-- 1.0E10
SELECT parse_json('{"a": 1, "b": 2, "a": "3"}');
-- error MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION (cause:
VARIANT_DUPLICATE_KEY)
SELECT try_parse_json('{"a": 1, "b": 2, "a": "3"}');
-- NULL
SET spark.sql.variant.allowDuplicateKeys=true;
SELECT to_json(parse_json('{"c": [], "b": 0, "a": null, "a": {"x": 0, "x":
1}, "b": 1, "b": 2, "c": [3]}'));
-- {"a":{"x":1},"b":2,"c":[3]}
SELECT try_parse_json('{1:2}'), try_parse_json('{"a":[a,b,c]}');
-- NULL NULL
```
These come from the `ExpressionDescription`, `VariantEndToEndSuite` and
`VariantExpressionEvalUtilsSuite`.
### Edge cases to test
Byte-level expectations are in `VariantExpressionEvalUtilsSuite."parseJson
positive"`. Port them as unit tests.
- Integers use the smallest of INT1, INT2, INT4 and INT8: `127` is INT1,
`128` is INT2, `-32769` is INT4 and
`2147483648` is INT8. An integer outside `i64` (for example
`9223372036854775808`) becomes DECIMAL16 with scale
0 if it has at most 38 digits. Otherwise it becomes DOUBLE (`1` followed
by 38 zeros is DOUBLE `1E38`).
- Non-integers become a decimal only if the text contains just `-`, `.` and
digits and fits precision/scale 38.
The original scale is kept: `1.0` is stored as DECIMAL4 with scale 1 and
unscaled 10. DECIMAL4 is used when
precision and scale are both <= 9, DECIMAL8 when they are <= 18, and
DECIMAL16 otherwise (`99999.9999` is
DECIMAL4, `99999.99999` is DECIMAL8). Anything with an exponent (`1e10`,
`1E0`) is DOUBLE, and so are values
over 38 digits.
- Strings of up to 63 UTF-8 bytes use the short-string header. Longer
strings use LONG_STR with a 4-byte length.
- Objects:
- Field ids and offsets are sorted by key, but the field data stays in
document order. For
`{"z": 1, "y": 2, "x": "3"}` the ids are `2,1,0` and the offsets are
`4,2,0,6`.
- Keys are compared with Java `String.compareTo`, which is UTF-16
code-unit order, not UTF-8 byte order. Test
keys that mix supplementary characters (U+10000 and above) with
U+E000-U+FFFF.
- Id and offset widths are the minimum that fits, and the size field
becomes 4 bytes above 255 fields.
- The metadata dictionary is in first-seen document order, without
duplicates. `sorted_strings` is never set,
and the offset width is the minimum that fits. Spark writes `{}` and
scalars with metadata `[0x01, 0, 0]`.
- Duplicate keys:
- `spark.sql.variant.allowDuplicateKeys` is internal and defaults to
`false`. The `VARIANT_DUPLICATE_KEY` error
is wrapped as `MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION` with
`badRecord` set to the input and
`failFastMode` = `FAILFAST`.
- With `true`, the last occurrence wins and the dead bytes are compacted.
- Malformed input such as `""`, `[`, `+1`, `1a`, single quotes, `NaN` and
leading zeros gives the same
`MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION` error. Spark uses a
default Jackson `JsonFactory`, so all the
lenient features are off.
- Trailing content: Spark calls `nextToken()` once and never checks for end
of input. By code reading,
`'{"a":1} x'` may parse successfully. Confirm this against Spark before
asserting it.
- Size limit: value and metadata are each limited to 128 MiB (16 MiB under
`SPARK_TESTING`). Going over raises
`VARIANT_SIZE_LIMIT` ("Cannot build variant bigger than 128.0 MiB in
`parse_json`."), and `try_parse_json`
returns NULL. Jackson's `StreamReadConstraints` defaults (string length,
number length, nesting depth) can fail
first, with the malformed-record error instead.
- Lone UTF-16 surrogates such as `"\uD835"` and `{"\uDC00":1}` are rejected
with the malformed-record error when
`spark.sql.variant.validateUnicodeInJsonParsing` is true (internal,
default true). With `false` they are
silently replaced by U+FFFD. Valid surrogate pairs are accepted.
- Test scalar and column inputs, NULL, the empty string, whitespace-only
input, and deep nesting.
### Differences across Spark versions
- 3.4 and 3.5 have no VARIANT.
- 4.0.4, 4.1.3 and 4.2.0 share the same `ParseJson` and builder logic.
- Only the 4.2.0 tree checked here has the lone-surrogate validation and its
config (tagged `.version("4.3.0")`).
4.0.4 and 4.1.3 accept lone surrogates and replace them with U+FFFD.
- Jackson is 2.18.6 in 4.0 and 2.21.2 in 4.1 and 4.2. That can change the
parser constraint limits.
### Implementation notes
- Spark: `ParseJson` and `ParseJsonExpressionBuilder` /
`TryParseJsonExpressionBuilder` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`.
`VariantExpressionEvalUtils.parseJson` is in the same package. The encoder
is
`common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java`,
and the constants and
`SIZE_LIMIT` are in `VariantUtil.java`.
- Arrow representation:
- Use the canonical Arrow extension type `arrow.parquet.variant`, which
arrow-rs implements as
`parquet_variant_compute::VariantType`. It is a `Struct` with `metadata`
and `value` binary children (`Binary`,
`LargeBinary` or `BinaryView`), plus an optional `typed_value` child
when shredded.
- Spark's own `ArrowUtils` maps `VariantType` to `Struct<value: Binary not
null, metadata: Binary not null>`.
The `metadata` child carries field metadata `variant=true` and there is
no extension name. Engines that
exchange Arrow with Spark have to translate between the two.
- The UDF sets the extension in `return_field_from_args`.
- arrow-rs has `parquet-variant`, `parquet-variant-compute` and
`parquet-variant-json`, but its encoding is not
byte-identical to Spark's. The local arrow-rs is 58.2. datafusion-spark is
on arrow 60, so use the matching
crate versions.
- `parquet-variant-json::append_json` uses `serde_json`. Non-integers and
integers outside `i64` become DOUBLE,
never decimal.
- Duplicate keys silently keep the last value.
- Keys are visited in `serde_json::Map` order, which is sorted unless the
`preserve_order` feature is unified in.
That changes the dictionary order.
- `serde_json` stops at nesting depth 128 and rejects trailing content.
- `WritableMetadataBuilder` sets `sorted_strings` when keys happen to
arrive in sorted order, including any
single-key dictionary. Spark never sets it.
- `ObjectBuilder::finish` sorts by Rust `str` byte order, not UTF-16 order.
- When duplicates are allowed, the old bytes are left in place, so sizes
differ.
- I found no size limit in the builder.
- Spark reads any valid encoding (`VariantExpressionSuite."to_json valid
input"`), so readers only need logical
equivalence. Field order is the exception. Spark's `schema_of_variant`
validates sorting with UTF-16 compare
and throws `MALFORMED_VARIANT`, and `getFieldByKey` binary-searches
objects with 32 or more fields. A writer
that sorts by UTF-8 bytes can therefore produce values that Spark rejects.
The size limit also depends on
Spark's exact byte layout.
- Recommendation:
- Reuse `parquet_variant::Variant` for decoding.
- Write a Spark-specific JSON-to-variant encoder in a new
`crates/spark/src/function/variant/` module, or a
configured `VariantBuilder` with Spark's number classification,
document-order dictionary, UTF-16 key
ordering, duplicate handling, cleared `sorted_strings` and the 128 MiB
limit.
- Use a JSON tokenizer that exposes the raw number text.
- DataFusion core has no VARIANT support or `parse_json`.
- apache/datafusion-comet#5438 tracks native Variant support in Comet.
- Add `crates/sqllogictest/test_files/variant/parse_json.slt`. It will need
`to_json` or a cast to string to
display values. Document both names with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]