Allison Wang created SPARK-33472:
------------------------------------
Summary: IllegalArgumentException when applying
RemoveRedundantSorts before EnsureRequirements
Key: SPARK-33472
URL: https://issues.apache.org/jira/browse/SPARK-33472
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.8, 3.0.2, 3.1.0
Reporter: Allison Wang
`RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check
whether a sort node is redundant. Currently, it is added before
`EnsureRequirements`. Since `PartitioningCollection` requires left and right
partitioning to have the same number of partitions, which is not necessarily
true before `EnsureRequirements`, the rule can fail with the following
exception:
{{{{IllegalArgumentException: requirement failed: PartitioningCollection
requires all of its partitionings have the same numPartitions.}}}}
We should switch the order between these two rules to satisfy the requirement
when instantiating `PartitioningCollection`.
{{{{}}}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
