Wan Kun created SPARK-41940:
-------------------------------
Summary: Infer IsNotNull constraints for complex expression join
keys
Key: SPARK-41940
URL: https://issues.apache.org/jira/browse/SPARK-41940
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.4.0
Reporter: Wan Kun
Infer IsNotNull constraints for complex join expressions could help filter a
lot of rows before join.
For example,
{code:sql}
CREATE TABLE t(id int, text string) using parquet;
SELECT *
FROM t as t1
LEFT JOIN t as t2
ON t1.text = get_json_object(t2.text, '$.not_exists_col');
{code}
we can infer *IsNotNull(get_json_object(t2.text, '$.not_exists_col'))*
constraint alone with *IsNotNull(t2.text)* to filter a lot of rows for the
right side of the join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
