andygrove opened a new issue, #243:
URL: https://github.com/apache/datafusion-spark/issues/243
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`json_object_keys(json_object)` returns the keys of the outermost JSON
object as an `ARRAY<STRING>`, in document
order. It returns NULL for NULL input, an empty string, any valid JSON whose
top-level value is not an object, and
malformed JSON. The result is nullable.
The argument must be a string (`ExpectsInputTypes`, so no implicit cast;
`json_object_keys(200)` fails analysis with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`). From 4.0 the input is
`StringTypeWithCollation(supportsTrimCollation =
true)` and the element type is the default string type. Available since
Spark 3.1.0. No aliases.
The implementation uses the lenient Jackson `SharedFactory`
(`ALLOW_SINGLE_QUOTES`, `ALLOW_UNESCAPED_CONTROL_CHARS`).
It requires the first token to be `START_OBJECT`, then loops `while
(nextValue() != null && currentName() != null)`,
collecting `currentName()` and calling `skipChildren()`. Any `IOException`
gives NULL.
### Examples
```sql
SELECT json_object_keys('{}');
-- []
SELECT json_object_keys('{"key": "value"}');
-- ["key"]
SELECT json_object_keys('{"f1":"abc","f2":{"f3":"a", "f4":"b"}}');
-- ["f1","f2"]
SELECT json_object_keys(null);
-- NULL
SELECT json_object_keys('');
-- NULL
SELECT json_object_keys('{"key": "value", "key2": 2}');
-- ["key","key2"]
SELECT json_object_keys('{"key":[1,2,3,{"key":"value"},[1,2,3]]}');
-- ["key"]
SELECT json_object_keys('{"k1": [1, 2, {"key": 5}], "k2": {"key2": [1,
2]}}');
-- ["k1","k2"]
SELECT json_object_keys('{[1,2]}');
-- NULL
SELECT json_object_keys('{"key": 45, "random_string"}');
-- NULL
SELECT json_object_keys('[1, 2, 3]');
-- NULL
```
The first three come from the `ExpressionDescription`. The others come from
`json-functions.sql.out`.
### Edge cases to test
- NULL, empty string, and whitespace-only input give NULL. `{}` gives an
empty array, not NULL.
- Top-level arrays (`[]`, `[{"key": "JSON"}]`) and scalars give NULL.
- Only top-level keys are returned. Nested objects and arrays are skipped.
- Malformed JSON anywhere before the closing `}` of the outer object gives
NULL, even after some keys have been
collected (`{"key": 45, "random_string"}`).
- Duplicate keys. Jackson does not detect duplicates by default, so by code
reading `{"a":1,"a":2}` gives `["a","a"]`.
Do not deduplicate the keys or sort them.
- Order is document order. A `serde_json::Map` without `preserve_order`
sorts keys, so it would be wrong here.
- Lenient syntax. Single-quoted keys (`{'a': 1}`) and raw control characters
inside strings are accepted. Unquoted
keys, comments, and leading-zero numbers are not.
- Keys with escapes (`{"a\"b": 1}`, `{"é": 1}`) come back decoded. Also test
multi-byte UTF-8 keys.
- Trailing content. The loop ends at the outer `END_OBJECT`, and Spark never
checks for EOF. By code reading,
`json_object_keys('{"a":1} x')` returns `["a"]`.
- Jackson stream limits (3.5+). A number longer than 1000 digits, or nesting
deeper than 1000, anywhere before the end
of the outer object gives NULL.
- Scalar and column input.
- The function is not affected by ANSI mode.
### Differences across Spark versions
- The behaviour is the same from 3.4 through 4.2, apart from the Jackson
upgrade. 3.4 uses Jackson 2.14.2 with no
number or nesting limits, and 3.5 and later have those limits.
- 4.0 turns the expression into a `RuntimeReplaceable` that calls
`JsonExpressionUtils.jsonObjectKeys` (no behaviour
change) and adds collation support.
- The type-mismatch error parameter `paramIndex` changed from `"1"` (3.x) to
`"first"` (4.0+).
### Implementation notes
- Spark: `JsonObjectKeys` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala`.
From 4.0 the logic is in
`sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionUtils.java`
(`jsonObjectKeys`).
- DataFusion core has no JSON functions. Do not confuse this with `map_keys`.
- In datafusion-spark, `crates/spark/src/function/json/json_tuple.rs` parses
into `serde_json::Value`. That would
lose duplicate keys and reorder them, reject single quotes, and reject
trailing content. Use a streaming lenient
tokenizer instead, ideally one shared with `get_json_object`,
`json_array_length` and `json_tuple`.
- Build the result as a `ListArray` of `Utf8`. The element can be
non-nullable in practice, but Spark declares
`ArrayType(StringType)` with `containsNull = true`.
- Put it in `crates/spark/src/function/json/`, add
`crates/sqllogictest/test_files/json/json_object_keys.slt`, and
document it with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]