[
https://issues.apache.org/jira/browse/SPARK-59749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59749:
-----------------------------------
Labels: pull-request-available (was: )
> ASOF JOIN MATCH_CONDITION rejects empty STRUCT operands accepted by the
> comparison operator
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-59749
> URL: https://issues.apache.org/jira/browse/SPARK-59749
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
>
> Inside a MATCH_CONDITION, comparing two empty structs (STRUCT<>) fails with
> ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE. The same comparison outside ASOF
> works: two empty structs compare as equal. The error says the operands must
> be orderable, but STRUCT<> is orderable. ASOF adds its own check that rejects
> any type that contains an empty struct.
> SET spark.sql.join.asofJoin.enabled = true;
> -- fails:
> SELECT * FROM VALUES (named_struct()) AS l(s)
> ASOF JOIN VALUES (named_struct()) AS r(s)
> MATCH_CONDITION (l.s >= r.s); --
> ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE: "STRUCT<>" and "STRUCT<>"
> -- works:
> SELECT named_struct() >= named_struct(); -- true
> The same check also rejects nested empty structs, for example STRUCT<x:
> STRUCT<>> and ARRAY<STRUCT<>>.
> Expected: MATCH_CONDITION accepts empty struct operands, nested or not, same
> as the comparison operator. An empty struct paired with a non-empty struct
> stays rejected.
> The check does not depend on ANSI mode.
> Impact: an as-of condition that is a legal comparison cannot be written.
> Umbrella: SPARK-59526
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
