[
https://issues.apache.org/jira/browse/SPARK-50228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-50228:
-----------------------------
Description:
Such conversions should be NOT considered as an optimization, but it is needed
for correctness. So, we cannot run the substitutions as an optimization rule,
it should be applied before any.
We could modify RewriteCollationJoin and take into account DPP, but we might
miss something else. Fxing ALL results of optimizations in RewriteCollationJoin
introduces some dependencies.
> Move the RewriteCollationJoin rule to FinishAnalysis
> ----------------------------------------------------
>
> Key: SPARK-50228
> URL: https://issues.apache.org/jira/browse/SPARK-50228
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Such conversions should be NOT considered as an optimization, but it is
> needed for correctness. So, we cannot run the substitutions as an
> optimization rule, it should be applied before any.
> We could modify RewriteCollationJoin and take into account DPP, but we might
> miss something else. Fxing ALL results of optimizations in
> RewriteCollationJoin introduces some dependencies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
