[
https://issues.apache.org/jira/browse/SPARK-60785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60785:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> get_json_object returns the string 'null' instead of NULL when an array index
> path reaches a JSON null
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60785
> URL: https://issues.apache.org/jira/browse/SPARK-60785
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> {code:sql}
> SELECT get_json_object('{"a": null}', '$.a'), -- NULL
> get_json_object('[null, 1]', '$[0]'), -- 'null' (expected
> NULL)
> get_json_object('{"a": [null]}', '$.a[0]'), -- 'null' (expected
> NULL)
> get_json_object('["null"]', '$[0]'); -- 'null' (the JSON
> string "null")
> {code}
> A field path that ends at a JSON null returns SQL NULL, but an array-index
> path that ends at a JSON null returns the 4-character string {{'null'}},
> which cannot be told apart from the JSON string {{"null"}}. So e.g.
> {{get_json_object(to_json(arr), '$[0]')}} differs from {{CAST(arr[0] AS
> STRING)}} exactly for NULL elements.
> Cause: in {{GetJsonObjectEvaluator.evaluatePath}}
> ({{JsonExpressionEvalUtils.scala}}) only the {{(FIELD_NAME, Named(name) ::
> xs)}} case checks {{p.nextToken() != JsonToken.VALUE_NULL}} and returns no
> result for a null. The {{(START_ARRAY, Subscript :: Index(idx) :: xs)}} case
> does not, so the null token reaches the {{(_, Nil)}} case, which copies it to
> the output as the text {{null}} ({{g.copyCurrentStructure(p)}}). Same code on
> master.
> Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
