Yaoxuan Wu created SPARK-60785:
----------------------------------
Summary: get_json_object returns the string 'null' instead of NULL
when an array index path reaches a JSON null
Key: SPARK-60785
URL: https://issues.apache.org/jira/browse/SPARK-60785
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0
Reporter: Yaoxuan Wu
{code:sql}
SELECT get_json_object('{"a": null}', '$.a'), -- NULL
get_json_object('[null, 1]', '$[0]'), -- 'null' (expected NULL)
get_json_object('{"a": [null]}', '$.a[0]'), -- 'null' (expected NULL)
get_json_object('["null"]', '$[0]'); -- 'null' (the JSON
string "null")
{code}
A field path that ends at a JSON null returns SQL NULL, but an array-index path
that ends at a JSON null returns the 4-character string {{'null'}}, which
cannot be told apart from the JSON string {{"null"}}. So e.g.
{{get_json_object(to_json(arr), '$[0]')}} differs from {{CAST(arr[0] AS
STRING)}} exactly for NULL elements.
Cause: in {{GetJsonObjectEvaluator.evaluatePath}}
({{JsonExpressionEvalUtils.scala}}) only the {{(FIELD_NAME, Named(name) ::
xs)}} case checks {{p.nextToken() != JsonToken.VALUE_NULL}} and returns no
result for a null. The {{(START_ARRAY, Subscript :: Index(idx) :: xs)}} case
does not, so the null token reaches the {{(_, Nil)}} case, which copies it to
the output as the text {{null}} ({{g.copyCurrentStructure(p)}}). Same code on
master.
Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
