andygrove opened a new issue, #243:
URL: https://github.com/apache/datafusion-spark/issues/243

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `json_object_keys(json_object)` returns the keys of the outermost JSON 
object as an `ARRAY<STRING>`, in document
   order. It returns NULL for NULL input, an empty string, any valid JSON whose 
top-level value is not an object, and
   malformed JSON. The result is nullable.
   
   The argument must be a string (`ExpectsInputTypes`, so no implicit cast; 
`json_object_keys(200)` fails analysis with
   `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`). From 4.0 the input is 
`StringTypeWithCollation(supportsTrimCollation =
   true)` and the element type is the default string type. Available since 
Spark 3.1.0. No aliases.
   
   The implementation uses the lenient Jackson `SharedFactory` 
(`ALLOW_SINGLE_QUOTES`, `ALLOW_UNESCAPED_CONTROL_CHARS`).
   It requires the first token to be `START_OBJECT`, then loops `while 
(nextValue() != null && currentName() != null)`,
   collecting `currentName()` and calling `skipChildren()`. Any `IOException` 
gives NULL.
   
   ### Examples
   
   ```sql
   SELECT json_object_keys('{}');
   -- []
   SELECT json_object_keys('{"key": "value"}');
   -- ["key"]
   SELECT json_object_keys('{"f1":"abc","f2":{"f3":"a", "f4":"b"}}');
   -- ["f1","f2"]
   SELECT json_object_keys(null);
   -- NULL
   SELECT json_object_keys('');
   -- NULL
   SELECT json_object_keys('{"key": "value", "key2": 2}');
   -- ["key","key2"]
   SELECT json_object_keys('{"key":[1,2,3,{"key":"value"},[1,2,3]]}');
   -- ["key"]
   SELECT json_object_keys('{"k1": [1, 2, {"key": 5}], "k2": {"key2": [1, 
2]}}');
   -- ["k1","k2"]
   SELECT json_object_keys('{[1,2]}');
   -- NULL
   SELECT json_object_keys('{"key": 45, "random_string"}');
   -- NULL
   SELECT json_object_keys('[1, 2, 3]');
   -- NULL
   ```
   
   The first three come from the `ExpressionDescription`. The others come from 
`json-functions.sql.out`.
   
   ### Edge cases to test
   
   - NULL, empty string, and whitespace-only input give NULL. `{}` gives an 
empty array, not NULL.
   - Top-level arrays (`[]`, `[{"key": "JSON"}]`) and scalars give NULL.
   - Only top-level keys are returned. Nested objects and arrays are skipped.
   - Malformed JSON anywhere before the closing `}` of the outer object gives 
NULL, even after some keys have been
     collected (`{"key": 45, "random_string"}`).
   - Duplicate keys. Jackson does not detect duplicates by default, so by code 
reading `{"a":1,"a":2}` gives `["a","a"]`.
     Do not deduplicate the keys or sort them.
   - Order is document order. A `serde_json::Map` without `preserve_order` 
sorts keys, so it would be wrong here.
   - Lenient syntax. Single-quoted keys (`{'a': 1}`) and raw control characters 
inside strings are accepted. Unquoted
     keys, comments, and leading-zero numbers are not.
   - Keys with escapes (`{"a\"b": 1}`, `{"é": 1}`) come back decoded. Also test 
multi-byte UTF-8 keys.
   - Trailing content. The loop ends at the outer `END_OBJECT`, and Spark never 
checks for EOF. By code reading,
     `json_object_keys('{"a":1} x')` returns `["a"]`.
   - Jackson stream limits (3.5+). A number longer than 1000 digits, or nesting 
deeper than 1000, anywhere before the end
     of the outer object gives NULL.
   - Scalar and column input.
   - The function is not affected by ANSI mode.
   
   ### Differences across Spark versions
   
   - The behaviour is the same from 3.4 through 4.2, apart from the Jackson 
upgrade. 3.4 uses Jackson 2.14.2 with no
     number or nesting limits, and 3.5 and later have those limits.
   - 4.0 turns the expression into a `RuntimeReplaceable` that calls 
`JsonExpressionUtils.jsonObjectKeys` (no behaviour
     change) and adds collation support.
   - The type-mismatch error parameter `paramIndex` changed from `"1"` (3.x) to 
`"first"` (4.0+).
   
   ### Implementation notes
   
   - Spark: `JsonObjectKeys` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala`.
     From 4.0 the logic is in
     
`sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionUtils.java`
     (`jsonObjectKeys`).
   - DataFusion core has no JSON functions. Do not confuse this with `map_keys`.
   - In datafusion-spark, `crates/spark/src/function/json/json_tuple.rs` parses 
into `serde_json::Value`. That would
     lose duplicate keys and reorder them, reject single quotes, and reject 
trailing content. Use a streaming lenient
     tokenizer instead, ideally one shared with `get_json_object`, 
`json_array_length` and `json_tuple`.
   - Build the result as a `ListArray` of `Utf8`. The element can be 
non-nullable in practice, but Spark declares
     `ArrayType(StringType)` with `containsNull = true`.
   - Put it in `crates/spark/src/function/json/`, add 
`crates/sqllogictest/test_files/json/json_object_keys.slt`, and
     document it with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to