[
https://issues.apache.org/jira/browse/SPARK-60871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić updated SPARK-60871:
---------------------------------
Fix Version/s: 4.4.0
> Explicitly collated variant_get target and JSON RETURNING types are ignored
> in comparisons
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-60871
> URL: https://issues.apache.org/jira/browse/SPARK-60871
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Stevo Mitric
> Assignee: Stevo Mitric
> Priority: Major
> Labels: pull-request-available
> Fix For: 5.0.0, 4.4.0
>
>
> {{variant_get}} / {{try_variant_get}} with a collated target type, and
> {{JSON_VALUE}} / {{JSON_QUERY}} / {{JSON_ARRAY}} / {{JSON_OBJECT}} with a
> collated {{RETURNING}} type, ignore that collation in comparisons. The result
> takes the collation of the arguments, so it gets cast back to UTF8_BINARY.
> {code:sql}
> CREATE TABLE t (id INT, v VARIANT) USING parquet;
> INSERT INTO t SELECT id, parse_json(j) FROM VALUES
> (1, '{"status": "Open"}'), (2, '{"status": "open"}'), (3, '{"status":
> "OPEN"}'), (4, '{"status": "closed"}') AS x(id, j);
> SELECT id FROM t WHERE variant_get(v, '$.status', 'STRING COLLATE
> UTF8_LCASE') = 'open';
> -- returns 2, expected 1, 2, 3
> {code}
> Related: runtime Bloom filters hash the raw bytes of the join key, so on a
> non-binary collated key that isn't wrapped in a collation key (an expression
> key, or the semi join of an IN subquery) they can drop rows that the join
> matches.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]