[
https://issues.apache.org/jira/browse/SPARK-59438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Zdravic updated SPARK-59438:
---------------------------------
Description:
The planner-stage path for ASOF JOIN — AsOfJoinSelection
(SparkStrategies.scala) selecting SortMergeAsOfJoinExec — is exercised
end-to-end by the SQL golden suite (join-asof-*.sql) and by
SortMergeAsOfJoinSuite / AsOfJoinSortMergeSQLSuite.
Two planner-stage behaviors, however, have no test:
# Physical plan shape (required distribution/ordering). SortMergeAsOfJoinExec
requires AllTuples (a single partition) when there are no equi-keys, and
clustered+sorted children otherwise. No test asserts the operator's physical
requirements: that a no-equi-key ASOF forces a single-partition exchange, and
that no redundant shuffle/sort is added when the child already satisfies them.
Sibling joins cover this in PlannerSuite; ASOF has nothing there.
# EqualNullSafe (<=>) in ON. The strategy deliberately excludes <=> from the
equi-keys and routes it to the residual condition. The resulting behavior —
NULL keys do match (unlike =), and with no equi-key both sides collapse to one
partition — is untested. (The = counterpart, "NULL never matches", is covered.)
This ticket adds the missing planner-stage tests. No production behavior
changes.
Umbrella: SPARK-59298
was:
The planner-stage path for ASOF JOIN — AsOfJoinSelection
(SparkStrategies.scala) selecting SortMergeAsOfJoinExec — is exercised
end-to-end by the SQL golden suite (join-asof-*.sql) and by
SortMergeAsOfJoinSuite / AsOfJoinSortMergeSQLSuite.
Two planner-stage behaviors, however, have no test:
# Physical plan shape (required distribution/ordering). SortMergeAsOfJoinExec
requires AllTuples (a single partition) when there are no equi-keys, and
clustered+sorted children otherwise. No test asserts the operator's physical
requirements: that a no-equi-key ASOF forces a single-partition exchange, and
that no redundant shuffle/sort is added when the child already satisfies them.
Sibling joins cover this in PlannerSuite; ASOF has nothing there.
# EqualNullSafe (<=>) in ON. The strategy deliberately excludes <=> from the
equi-keys and routes it to the residual condition. The resulting behavior —
NULL keys do match (unlike =), and with no equi-key both sides collapse to one
partition — is untested. (The = counterpart, "NULL never matches", is covered.)
This ticket adds the missing planner-stage tests. No production behavior
changes.
UMBRELLA: SPARK-59298
> Add planner-stage test coverage for ASOF JOIN sort-merge operator
> -----------------------------------------------------------------
>
> Key: SPARK-59438
> URL: https://issues.apache.org/jira/browse/SPARK-59438
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Luka Zdravic
> Priority: Minor
>
> The planner-stage path for ASOF JOIN — AsOfJoinSelection
> (SparkStrategies.scala) selecting SortMergeAsOfJoinExec — is exercised
> end-to-end by the SQL golden suite (join-asof-*.sql) and by
> SortMergeAsOfJoinSuite / AsOfJoinSortMergeSQLSuite.
> Two planner-stage behaviors, however, have no test:
> # Physical plan shape (required distribution/ordering).
> SortMergeAsOfJoinExec requires AllTuples (a single partition) when there are
> no equi-keys, and clustered+sorted children otherwise. No test asserts the
> operator's physical requirements: that a no-equi-key ASOF forces a
> single-partition exchange, and that no redundant shuffle/sort is added when
> the child already satisfies them. Sibling joins cover this in PlannerSuite;
> ASOF has nothing there.
> # EqualNullSafe (<=>) in ON. The strategy deliberately excludes <=> from the
> equi-keys and routes it to the residual condition. The resulting behavior —
> NULL keys do match (unlike =), and with no equi-key both sides collapse to
> one partition — is untested. (The = counterpart, "NULL never matches", is
> covered.)
> This ticket adds the missing planner-stage tests. No production behavior
> changes.
>
> Umbrella: SPARK-59298
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
