[ 
https://issues.apache.org/jira/browse/SPARK-61831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61831:
-----------------------------------
    Labels: pull-request-available  (was: )

> Cast from VARIANT to a type with a non-nullable nested field reads a JSON 
> null or missing key as 0
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61831
>                 URL: https://issues.apache.org/jira/browse/SPARK-61831
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.3.1
>            Reporter: Linhong Liu
>            Priority: Major
>              Labels: pull-request-available
>
> A cast from VARIANT to a type with a non-nullable nested slot, for example
> {{CAST(v AS STRUCT<user_id: BIGINT NOT NULL, country: STRING>)}} (also 
> {{v::STRUCT<...>}}, {{try_cast}}, and the same
> struct nested in an ARRAY or MAP), is accepted, and its result type keeps 
> {{user_id NOT NULL}}. When the JSON value is
> {{null}} or the key is missing, the cast still puts NULL into that field. 
> Everything downstream trusts the NOT NULL type,
> so the NULL is read as 0: {{IS NULL}} is false, {{coalesce}} is skipped, and 
> an INSERT into a NOT NULL column stores 0.
> {code:sql}
> SELECT event_id, p.user_id, p.user_id IS NULL FROM (
>   SELECT DISTINCT event_id, payload::STRUCT<user_id: BIGINT NOT NULL> AS p 
> FROM (
>     SELECT 1 AS event_id, parse_json('{"user_id": 42}') AS payload UNION ALL
>     SELECT 2, parse_json('{"user_id": null}') UNION ALL
>     SELECT 3, parse_json('{}')));
> -- returns (2, 0, false) and (3, 0, false); the user_id of rows 2 and 3 is 
> NULL
> {code}
> {{Cast.canAnsiCast}} and {{Cast.canCast}} accept any VARIANT target that 
> {{VariantGet.checkDataType}} accepts, which ignores
> nullability, while {{VariantGet.cast}} leaves a struct field NULL when the 
> key is missing or its value is null.
> {{variant_get}} avoids this by typing itself {{targetType.asNullable}}. 
> Elsewhere Cast already rejects a nullable source
> slot for a non-nullable target ({{resolvableNullability}}), including a 
> nested VARIANT.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to