[ 
https://issues.apache.org/jira/browse/SPARK-60871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uroš Bojanić updated SPARK-60871:
---------------------------------
    Fix Version/s: 4.4.0

> Explicitly collated variant_get target and JSON RETURNING types are ignored 
> in comparisons
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60871
>                 URL: https://issues.apache.org/jira/browse/SPARK-60871
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Stevo Mitric
>            Assignee: Stevo Mitric
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 5.0.0, 4.4.0
>
>
> {{variant_get}} / {{try_variant_get}} with a collated target type, and 
> {{JSON_VALUE}} / {{JSON_QUERY}} / {{JSON_ARRAY}} / {{JSON_OBJECT}} with a 
> collated {{RETURNING}} type, ignore that collation in comparisons. The result 
> takes the collation of the arguments, so it gets cast back to UTF8_BINARY.
> {code:sql}
> CREATE TABLE t (id INT, v VARIANT) USING parquet;
> INSERT INTO t SELECT id, parse_json(j) FROM VALUES
>   (1, '{"status": "Open"}'), (2, '{"status": "open"}'), (3, '{"status": 
> "OPEN"}'), (4, '{"status": "closed"}') AS x(id, j);
> SELECT id FROM t WHERE variant_get(v, '$.status', 'STRING COLLATE 
> UTF8_LCASE') = 'open';
> -- returns 2, expected 1, 2, 3
> {code}
> Related: runtime Bloom filters hash the raw bytes of the join key, so on a 
> non-binary collated key that isn't wrapped in a collation key (an expression 
> key, or the semi join of an IN subquery) they can drop rows that the join 
> matches.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to