andygrove opened a new issue, #275:
URL: https://github.com/apache/datafusion-spark/issues/275

   Part of #273 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `parse_json(jsonStr)` parses a JSON string into a `VARIANT` value. If the 
input is not valid JSON, it raises an
   error. `try_parse_json(jsonStr)` does the same but returns NULL for 
malformed input, duplicate keys, and the size
   limit. Both take exactly one `STRING` argument (any collation, including 
trim collations). SQL `NULL` input
   returns SQL `NULL`. `parse_json` is non-nullable for non-null input and 
`try_parse_json` is nullable. Both were
   added in Spark 4.0.0 (`variant_funcs`). Spark 3.x has no `VARIANT` type.
   
   This is the first VARIANT function, so this issue also covers how 
datafusion-spark represents the type. Every
   other variant function (`variant_get`, `is_variant_null`, 
`schema_of_variant`, `schema_of_variant_agg`,
   `to_variant_object`) needs that representation.
   
   A Spark VARIANT is two binaries. Spark implements the Parquet 
`VariantEncoding.md` spec as of parquet-format
   commit `37b6e8b8` (see `common/variant/README.md`):
   
   - **metadata**: a header byte (version `1` in the low 4 bits, bit 4 = 
`sorted_strings`, bits 6-7 =
     `offset_size - 1`), the dictionary size, `size + 1` offsets, then the 
UTF-8 key bytes.
   - **value**: a header byte whose low 2 bits are the basic type (primitive, 
short string, object, array) and whose
     upper 6 bits are type info, followed by the content.
   
   ### Examples
   
   ```sql
   SELECT parse_json('{"a":1,"b":0.8}');
   -- {"a":1,"b":0.8}
   SELECT try_parse_json('{"a":1,');
   -- NULL
   SELECT to_json(parse_json('[0.0, 1.00, 1.10, 1.23]'));
   -- [0,1,1.1,1.23]
   SELECT to_json(parse_json('1.0E10'));
   -- 1.0E10
   SELECT parse_json('{"a": 1, "b": 2, "a": "3"}');
   -- error MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION (cause: 
VARIANT_DUPLICATE_KEY)
   SELECT try_parse_json('{"a": 1, "b": 2, "a": "3"}');
   -- NULL
   SET spark.sql.variant.allowDuplicateKeys=true;
   SELECT to_json(parse_json('{"c": [], "b": 0, "a": null, "a": {"x": 0, "x": 
1}, "b": 1, "b": 2, "c": [3]}'));
   -- {"a":{"x":1},"b":2,"c":[3]}
   SELECT try_parse_json('{1:2}'), try_parse_json('{"a":[a,b,c]}');
   -- NULL  NULL
   ```
   
   These come from the `ExpressionDescription`, `VariantEndToEndSuite` and 
`VariantExpressionEvalUtilsSuite`.
   
   ### Edge cases to test
   
   Byte-level expectations are in `VariantExpressionEvalUtilsSuite."parseJson 
positive"`. Port them as unit tests.
   
   - Integers use the smallest of INT1, INT2, INT4 and INT8: `127` is INT1, 
`128` is INT2, `-32769` is INT4 and
     `2147483648` is INT8. An integer outside `i64` (for example 
`9223372036854775808`) becomes DECIMAL16 with scale
     0 if it has at most 38 digits. Otherwise it becomes DOUBLE (`1` followed 
by 38 zeros is DOUBLE `1E38`).
   - Non-integers become a decimal only if the text contains just `-`, `.` and 
digits and fits precision/scale 38.
     The original scale is kept: `1.0` is stored as DECIMAL4 with scale 1 and 
unscaled 10. DECIMAL4 is used when
     precision and scale are both <= 9, DECIMAL8 when they are <= 18, and 
DECIMAL16 otherwise (`99999.9999` is
     DECIMAL4, `99999.99999` is DECIMAL8). Anything with an exponent (`1e10`, 
`1E0`) is DOUBLE, and so are values
     over 38 digits.
   - Strings of up to 63 UTF-8 bytes use the short-string header. Longer 
strings use LONG_STR with a 4-byte length.
   - Objects:
     - Field ids and offsets are sorted by key, but the field data stays in 
document order. For
       `{"z": 1, "y": 2, "x": "3"}` the ids are `2,1,0` and the offsets are 
`4,2,0,6`.
     - Keys are compared with Java `String.compareTo`, which is UTF-16 
code-unit order, not UTF-8 byte order. Test
       keys that mix supplementary characters (U+10000 and above) with 
U+E000-U+FFFF.
     - Id and offset widths are the minimum that fits, and the size field 
becomes 4 bytes above 255 fields.
   - The metadata dictionary is in first-seen document order, without 
duplicates. `sorted_strings` is never set,
     and the offset width is the minimum that fits. Spark writes `{}` and 
scalars with metadata `[0x01, 0, 0]`.
   - Duplicate keys:
     - `spark.sql.variant.allowDuplicateKeys` is internal and defaults to 
`false`. The `VARIANT_DUPLICATE_KEY` error
       is wrapped as `MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION` with 
`badRecord` set to the input and
       `failFastMode` = `FAILFAST`.
     - With `true`, the last occurrence wins and the dead bytes are compacted.
   - Malformed input such as `""`, `[`, `+1`, `1a`, single quotes, `NaN` and 
leading zeros gives the same
     `MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION` error. Spark uses a 
default Jackson `JsonFactory`, so all the
     lenient features are off.
   - Trailing content: Spark calls `nextToken()` once and never checks for end 
of input. By code reading,
     `'{"a":1} x'` may parse successfully. Confirm this against Spark before 
asserting it.
   - Size limit: value and metadata are each limited to 128 MiB (16 MiB under 
`SPARK_TESTING`). Going over raises
     `VARIANT_SIZE_LIMIT` ("Cannot build variant bigger than 128.0 MiB in 
`parse_json`."), and `try_parse_json`
     returns NULL. Jackson's `StreamReadConstraints` defaults (string length, 
number length, nesting depth) can fail
     first, with the malformed-record error instead.
   - Lone UTF-16 surrogates such as `"\uD835"` and `{"\uDC00":1}` are rejected 
with the malformed-record error when
     `spark.sql.variant.validateUnicodeInJsonParsing` is true (internal, 
default true). With `false` they are
     silently replaced by U+FFFD. Valid surrogate pairs are accepted.
   - Test scalar and column inputs, NULL, the empty string, whitespace-only 
input, and deep nesting.
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5 have no VARIANT.
   - 4.0.4, 4.1.3 and 4.2.0 share the same `ParseJson` and builder logic.
   - Only the 4.2.0 tree checked here has the lone-surrogate validation and its 
config (tagged `.version("4.3.0")`).
     4.0.4 and 4.1.3 accept lone surrogates and replace them with U+FFFD.
   - Jackson is 2.18.6 in 4.0 and 2.21.2 in 4.1 and 4.2. That can change the 
parser constraint limits.
   
   ### Implementation notes
   
   - Spark: `ParseJson` and `ParseJsonExpressionBuilder` / 
`TryParseJsonExpressionBuilder` in
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`.
     `VariantExpressionEvalUtils.parseJson` is in the same package. The encoder 
is
     
`common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java`,
 and the constants and
     `SIZE_LIMIT` are in `VariantUtil.java`.
   - Arrow representation:
     - Use the canonical Arrow extension type `arrow.parquet.variant`, which 
arrow-rs implements as
       `parquet_variant_compute::VariantType`. It is a `Struct` with `metadata` 
and `value` binary children (`Binary`,
       `LargeBinary` or `BinaryView`), plus an optional `typed_value` child 
when shredded.
     - Spark's own `ArrowUtils` maps `VariantType` to `Struct<value: Binary not 
null, metadata: Binary not null>`.
       The `metadata` child carries field metadata `variant=true` and there is 
no extension name. Engines that
       exchange Arrow with Spark have to translate between the two.
     - The UDF sets the extension in `return_field_from_args`.
   - arrow-rs has `parquet-variant`, `parquet-variant-compute` and 
`parquet-variant-json`, but its encoding is not
     byte-identical to Spark's. The local arrow-rs is 58.2. datafusion-spark is 
on arrow 60, so use the matching
     crate versions.
     - `parquet-variant-json::append_json` uses `serde_json`. Non-integers and 
integers outside `i64` become DOUBLE,
       never decimal.
     - Duplicate keys silently keep the last value.
     - Keys are visited in `serde_json::Map` order, which is sorted unless the 
`preserve_order` feature is unified in.
       That changes the dictionary order.
     - `serde_json` stops at nesting depth 128 and rejects trailing content.
     - `WritableMetadataBuilder` sets `sorted_strings` when keys happen to 
arrive in sorted order, including any
       single-key dictionary. Spark never sets it.
     - `ObjectBuilder::finish` sorts by Rust `str` byte order, not UTF-16 order.
     - When duplicates are allowed, the old bytes are left in place, so sizes 
differ.
     - I found no size limit in the builder.
   - Spark reads any valid encoding (`VariantExpressionSuite."to_json valid 
input"`), so readers only need logical
     equivalence. Field order is the exception. Spark's `schema_of_variant` 
validates sorting with UTF-16 compare
     and throws `MALFORMED_VARIANT`, and `getFieldByKey` binary-searches 
objects with 32 or more fields. A writer
     that sorts by UTF-8 bytes can therefore produce values that Spark rejects. 
The size limit also depends on
     Spark's exact byte layout.
   - Recommendation:
     - Reuse `parquet_variant::Variant` for decoding.
     - Write a Spark-specific JSON-to-variant encoder in a new 
`crates/spark/src/function/variant/` module, or a
       configured `VariantBuilder` with Spark's number classification, 
document-order dictionary, UTF-16 key
       ordering, duplicate handling, cleared `sorted_strings` and the 128 MiB 
limit.
     - Use a JSON tokenizer that exposes the raw number text.
   - DataFusion core has no VARIANT support or `parse_json`.
   - apache/datafusion-comet#5438 tracks native Variant support in Comet.
   - Add `crates/sqllogictest/test_files/variant/parse_json.slt`. It will need 
`to_json` or a cast to string to
     display values. Document both names with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to