This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 3e81d53ddf [fix](Nereids) record wrong best plan properties #23973
(#24232)
3e81d53ddf is described below
commit 3e81d53ddf1c47b59f945fc9337df8e356a6071e
Author: morrySnow <[email protected]>
AuthorDate: Tue Sep 12 16:05:55 2023 +0800
[fix](Nereids) record wrong best plan properties #23973 (#24232)
pick from master:
PR: #23973
commit-id: b2ca28139553cacd0f5f2e492e30d3a37ce26bed
when output meet order by not meet distribution. we use a trick way to
do enforce by set current output to any. but when we do enforce later,
we still use the old output. So when we do choose best plan, we could
not find the older output's plan, since we have replace it by any.
For example:
```
lowest Plan(cost, properties, plan, childrenRequires)
18.0 ANY
id:138#4 cost=0 [0/0/0/] estRows=4 children=[@0 ]
(plan=PhysicalWindow[139]@4 ( windowFrameGroup=(Funcs=[row_number()
WindowSpec(PARTITION BY b#1, a#0 ROWS BETWEEN UNBOUNDED_PRECEDING AND
CURRENT_ROW) AS `r1`#2], PartitionKeys=[b#1, a#0], OrderKeys=[],
WindowFrame=WindowFrame(ROWS, UNBOUNDED_PRECEDING, CURRENT_ROW)),
requiredProperties=[DistributionSpecHash ( orderedShuffledColumns=[1, 0],
shuffleType=REQUIRE, tableId=-1, selectedIndexId=-1, partitionIds=[],
equivalenceExprIds=[[ [...]
[DistributionSpecHash ( orderedShuffledColumns=[0],
shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297,
partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0}
) Order: ([b#1 asc, a#0 asc])]
32.01171875 DistributionSpecHash ( orderedShuffledColumns=[1],
shuffleType=REQUIRE, tableId=-1, selectedIndexId=-1, partitionIds=[],
equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} ) Order: ([b#1 asc])
id:161#4 cost=14 [4/4/4/] estRows=4 children=[@4 ]
(plan=PhysicalQuickSort[162]@4 ( orderKeys=[b#1 asc], phase=LOCAL_SORT,
stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0],
shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297,
partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0}
) Order: ([b#1 asc, a#0 asc])]
32.01171875 DistributionSpecHash ( orderedShuffledColumns=[1],
shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1,
partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} )
Order: ([b#1 asc])
id:161#4 cost=14 [4/4/4/] estRows=4 children=[@4 ]
(plan=PhysicalQuickSort[162]@4 ( orderKeys=[b#1 asc], phase=LOCAL_SORT,
stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[1],
shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1,
partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} )
Order: ([])]
18.01171875 DistributionSpecHash ( orderedShuffledColumns=[1],
shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1,
partitionIds=[], equivalenceExprIds=[[1]], exprIdToEquivalenceSet={1=0} )
Order: ([])
id:157#4 cost=0 [0/0/0/] estRows=4 children=[@4 ]
(plan=PhysicalDistribute[158]@4 ( distributionSpec=DistributionSpecHash (
orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1,
selectedIndexId=-1, partitionIds=[], equivalenceExprIds=[[1]],
exprIdToEquivalenceSet={1=0} ), stats=null ))
[DistributionSpecHash ( orderedShuffledColumns=[0],
shuffleType=NATURAL, tableId=3547296, selectedIndexId=3547297,
partitionIds=[3547295], equivalenceExprIds=[[0]], exprIdToEquivalenceSet={0=0}
) Order: ([b#1 asc, a#0 asc])]
```
the last one require a natural shuffle type property from this group.
but this property already been removed when we do
enforceDistributionButMeetSort. So, such exception will be thrown
```
Caused by: org.apache.doris.nereids.exceptions.AnalysisException: Failed to
choose best plan
at
org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:340)
~[classes/:?]
at
org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:323)
~[classes/:?]
... 18 more
Caused by: org.apache.doris.nereids.exceptions.AnalysisException:
lowestCostPlans with physicalProperties(DistributionSpecHash (
orderedShuffledColumns=[0], shuffleType=NATURAL, tableId=3547296,
selectedIndexId=3547297, partitionIds=[3547295], equivalenceExprIds=[[0]],
exprIdToEquivalenceSet={0=0} ) Order: ([b#1 asc, a#0 asc])) doesn't exist in
root group
at
org.apache.doris.nereids.NereidsPlanner.lambda$chooseBestPlan$1(NereidsPlanner.java:318)
~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?]
at
org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:317)
~[classes/:?]
at
org.apache.doris.nereids.NereidsPlanner.chooseBestPlan(NereidsPlanner.java:323)
~[classes/:?]
... 18 more
```
---
.../apache/doris/nereids/properties/EnforceMissingPropertiesHelper.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/EnforceMissingPropertiesHelper.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/EnforceMissingPropertiesHelper.java
index 1c45db6d4c..d548e3254c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/EnforceMissingPropertiesHelper.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/EnforceMissingPropertiesHelper.java
@@ -87,7 +87,7 @@ public class EnforceMissingPropertiesHelper {
groupExpression.getOwnerGroup()
.replaceBestPlanProperty(
output, PhysicalProperties.ANY,
groupExpression.getCostValueByProperties(output));
- return enforceSortAndDistribution(output, request);
+ return enforceSortAndDistribution(PhysicalProperties.ANY, request);
}
private PhysicalProperties enforceGlobalSort(PhysicalProperties
oldOutputProperty, PhysicalProperties required) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
