tkalkirill commented on code in PR #13389:
URL: https://github.com/apache/ignite/pull/13389#discussion_r3842284962
##########
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/integration/QueryBlockingTaskExecutorIntegrationTest.java:
##########
@@ -61,8 +61,7 @@ public void testJoinRehash() throws Exception {
assertQuery(sql)
.withParams("region0")
- .matches(QueryChecker.containsSubPlan("IgniteMergeJoin"))
-
.matches(QueryChecker.containsSubPlan("IgniteExchange(distribution=[affinity"))
+
.matches(QueryChecker.matches(".*IgniteMergeJoin.*IgniteExchange\\(distribution=\\[affinity.*"))
Review Comment:
Out of curiosity, why did you rewrite it for beauty or practicality?
##########
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/integration/LitmusCheckIntegrationTest.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ignite.internal.processors.query.calcite.integration;
+
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.core.config.Configurator;
+import org.junit.Test;
+
+import static org.apache.logging.log4j.Level.DEBUG;
+
+/** Calcite litmus related tests. */
+public class LitmusCheckIntegrationTest extends AbstractBasicIntegrationTest {
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ super.beforeTestsStarted();
+
+ setCalciteLoggerDebugLevel();
Review Comment:
Do you think this is necessary?
##########
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/AggregatePlannerTest.java:
##########
@@ -440,10 +440,6 @@ public void colocated() throws Exception {
.and(hasDistribution(IgniteDistributions.affinity(0, null,
"hash")))),
algo.rulesToDisable);
- // TODO: https://issues.apache.org/jira/browse/IGNITE-16334 Eventually
planner skips optimal join plan.
Review Comment:
Why did you delete the link? Does that mean the problem with the deleted
link has been fixed?
##########
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteNestedLoopJoin.java:
##########
@@ -88,12 +88,16 @@ public IgniteNestedLoopJoin(RelInput input) {
if (Double.isInfinite(rightCnt))
return costFactory.makeInfiniteCost();
- double rows = leftCnt * rightCnt;
-
double rightSize = rightCnt * getRight().getRowType().getFieldCount()
* IgniteCost.AVERAGE_FIELD_SIZE;
- return costFactory.makeCost(rows,
- rows * (IgniteCost.ROW_COMPARISON_COST +
IgniteCost.ROW_PASS_THROUGH_COST), 0, rightSize, 0);
+ double rowCnt = mq.getRowCount(this);
+
+ RelOptCost cost = costFactory.makeCost(rowCnt,
Review Comment:
Could we keep the CPU cost proportional to `leftCnt * rightCnt`? A
nested-loop join still compares every input pair, while the output row count
may be orders of magnitude smaller. Multiplying it by 10 does not preserve the
actual complexity and may make NLJ incorrectly cheaper than a hash join.
##########
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/JoinRowCountEstimationTest.java:
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.query.calcite.planner;
+
+import java.math.BigDecimal;
+import java.util.function.Predicate;
+import java.util.regex.Pattern;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.sql.SqlExplainFormat;
+import org.apache.calcite.sql.SqlExplainLevel;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.ignite.internal.processors.query.calcite.schema.IgniteSchema;
+import
org.apache.ignite.internal.processors.query.calcite.trait.IgniteDistributions;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.hamcrest.BaseMatcher;
+import org.hamcrest.CoreMatchers;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import static
org.apache.ignite.internal.processors.query.calcite.metadata.IgniteMdSelectivity.COMPARISON_SELECTIVITY;
+import static
org.apache.ignite.internal.processors.query.calcite.metadata.IgniteMdSelectivity.EQUALS_SELECTIVITY;
+import static
org.apache.ignite.internal.processors.query.calcite.metadata.IgniteMdSelectivity.IS_NOT_NULL_SELECTIVITY;
+
+/**
+ * Tests to check row count estimation for join relation.
+ */
+public class JoinRowCountEstimationTest extends AbstractPlannerTest {
+ /** */
+ private static final int CATALOG_SALES_SIZE = 1_441_548;
Review Comment:
Can you indicate in the comment or in the name what is meant by the size in
all these constants?
##########
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdRowCount.java:
##########
@@ -72,48 +88,263 @@ public class IgniteMdRowCount extends RelMdRowCount {
mq.getRowCount(rel.getLeft()));
}
+ JoinInfo joinInfo = rel.analyzeCondition();
+
+ if (joinInfo.pairs().isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
// Row count estimates of 0 will be rounded up to 1.
// So, use maxRowCount where the product is very small.
- final Double left = mq.getRowCount(rel.getLeft());
- final Double right = mq.getRowCount(rel.getRight());
+ final Double leftRowCnt = mq.getRowCount(rel.getLeft());
+ final Double rightRowCnt = mq.getRowCount(rel.getRight());
- if (left == null || right == null)
+ if (leftRowCnt == null || rightRowCnt == null)
return null;
- if (left <= 1D || right <= 1D) {
+ if (leftRowCnt <= 1D || rightRowCnt <= 1D) {
Double max = mq.getMaxRowCount(rel);
if (max != null && max <= 1D)
return max;
}
- JoinInfo joinInfo = rel.analyzeCondition();
+ Map<Integer, KeyColumnOrigin> columnsFromLeft = resolveOrigins(mq,
rel.getLeft(), joinInfo.leftKeys);
+ Map<Integer, KeyColumnOrigin> columnsFromRight = resolveOrigins(mq,
rel.getRight(), joinInfo.rightKeys);
+
+ if (columnsFromLeft.isEmpty() || columnsFromRight.isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
+ Map<TablesPair, JoinContext> joinCtxts = new HashMap<>();
+ for (IntPair joinKeys : joinInfo.pairs()) {
+ KeyColumnOrigin leftKey = columnsFromLeft.get(joinKeys.source);
+ KeyColumnOrigin rightKey = columnsFromRight.get(joinKeys.target);
+
+ if (leftKey == null || rightKey == null) {
+ continue;
+ }
+
+ joinCtxts.computeIfAbsent(
+ new TablesPair(
+ leftKey.origin.getOriginTable(),
+ rightKey.origin.getOriginTable()
+ ),
+ key -> {
+ IgniteTable leftTable = key.left.unwrap(IgniteTable.class);
+ IgniteTable rightTable =
key.right.unwrap(IgniteTable.class);
+
+ assert leftTable != null && rightTable != null;
+
+ int leftPkSize = leftTable.distribution().getKeys().size();
+ int rightPkSize =
rightTable.distribution().getKeys().size();
+
+ return new JoinContext(leftPkSize, rightPkSize);
+ }
+ ).countKeys(leftKey, rightKey);
+ }
+
+ if (joinCtxts.isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
+ Iterator<JoinContext> it = joinCtxts.values().iterator();
+ JoinContext joinCtx = it.next();
+ while (it.hasNext()) {
+ JoinContext nextCtx = it.next();
+ if (nextCtx.joinType().strength > joinCtx.joinType().strength) {
+ joinCtx = nextCtx;
+ }
+
+ if (joinCtx.joinType().strength ==
JoiningRelationType.PK_ON_PK.strength) {
+ break;
+ }
+ }
+
+ if (joinCtx.joinType() == JoiningRelationType.UNKNOWN) {
+ // Use crude estimation instead.
+ return crudeEstimation(mq, joinInfo, rel, leftRowCnt, rightRowCnt);
+ }
+
+ double postFiltrationAdjustment = 1.0;
+
+ switch (rel.getJoinType()) {
+ case INNER:
+ case SEMI:
+ // Extra join keys as well as non-equi conditions serves as
post-filtration,
+ // therefore we need to adjust final result with a little
factor.
+ if (joinCtxts.size() != 1 || !joinInfo.isEqui())
+ postFiltrationAdjustment = NON_EQUI_COEFF;
+
+ break;
+ default:
+ break;
+ }
+
+ double baseRowCnt = 0.0;
+ Double percentageAdjustment = null;
+ if (joinCtx.joinType() == JoiningRelationType.PK_ON_PK) {
+ postFiltrationAdjustment = EQUI_COEFF;
+
+ if (rel.getJoinType() == JoinRelType.INNER || rel.getJoinType() ==
JoinRelType.SEMI) {
+ // Assume we have two fact tables SALES and RETURNS sharing
the same primary key. Every item
+ // can be sold, but only items which were sold can be returned
back, therefore
+ // size(SALES) > size(RETURNS). When joining SALES on RETURNS
by primary key, the estimated
+ // result size will be the same as the size of the smallest
table (RETURNS in this case),
+ // adjusted by the percentage of rows of the biggest table
(SALES in this case; percentage
+ // adjustment is required to account for predicates pushed
down to the table, e.g. we are
+ // interested in returns of items with certain category)
+ if (leftRowCnt > rightRowCnt) {
+ baseRowCnt = rightRowCnt;
+ percentageAdjustment =
mq.getPercentageOriginalRows(rel.getLeft());
+ }
+ else {
+ baseRowCnt = leftRowCnt;
+ percentageAdjustment =
mq.getPercentageOriginalRows(rel.getRight());
+ }
+ }
+ else if (rel.getJoinType() == JoinRelType.LEFT) {
+ baseRowCnt = leftRowCnt;
+ }
+ else if (rel.getJoinType() == JoinRelType.RIGHT) {
+ baseRowCnt = rightRowCnt;
+ }
+ else if (rel.getJoinType() == JoinRelType.FULL) {
+ Double selectivity = mq.getSelectivity(rel,
rel.getCondition());
+
+ // Fall-back.
+ if (selectivity == null) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel,
rel.getCondition());
+ }
+
+ baseRowCnt = rightRowCnt + leftRowCnt;
+ percentageAdjustment = 1.0 - selectivity;
+ }
+ }
+ else if (joinCtx.joinType() == JoiningRelationType.FK_ON_PK) {
+ // For foreign key joins the base table is the one which is joined
by non-primary key columns.
+ if (rel.getJoinType() == JoinRelType.INNER || rel.getJoinType() ==
JoinRelType.SEMI) {
+ Double selectivity = mq.getSelectivity(rel,
rel.getCondition());
+ baseRowCnt = leftRowCnt * selectivity;
+ percentageAdjustment =
mq.getPercentageOriginalRows(rel.getRight());
+ }
+ else if (rel.getJoinType() == JoinRelType.LEFT ||
rel.getJoinType() == JoinRelType.RIGHT) {
+ baseRowCnt = leftRowCnt;
+ }
+ else if (rel.getJoinType() == JoinRelType.FULL) {
+ Double selectivity = mq.getSelectivity(rel,
rel.getCondition());
+
+ // Fall-back.
+ if (selectivity == null) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel,
rel.getCondition());
+ }
+
+ baseRowCnt = rightRowCnt + leftRowCnt;
+ percentageAdjustment = 1.0 - selectivity;
+ }
+ }
+ else { // PK_ON_FK
+ if (rel.getJoinType() == JoinRelType.INNER || rel.getJoinType() ==
JoinRelType.SEMI) {
+ baseRowCnt = rightRowCnt;
Review Comment:
The reversed `FK_ON_PK` branch applies join selectivity, while this
`PK_ON_FK` branch does not. Thus, commuting the same inner join produces
different row-count estimates. Could both orientations use the same formula
relative to the FK side?
##########
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdRowCount.java:
##########
@@ -72,48 +88,263 @@ public class IgniteMdRowCount extends RelMdRowCount {
mq.getRowCount(rel.getLeft()));
}
+ JoinInfo joinInfo = rel.analyzeCondition();
+
+ if (joinInfo.pairs().isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
// Row count estimates of 0 will be rounded up to 1.
// So, use maxRowCount where the product is very small.
- final Double left = mq.getRowCount(rel.getLeft());
- final Double right = mq.getRowCount(rel.getRight());
+ final Double leftRowCnt = mq.getRowCount(rel.getLeft());
+ final Double rightRowCnt = mq.getRowCount(rel.getRight());
- if (left == null || right == null)
+ if (leftRowCnt == null || rightRowCnt == null)
return null;
- if (left <= 1D || right <= 1D) {
+ if (leftRowCnt <= 1D || rightRowCnt <= 1D) {
Double max = mq.getMaxRowCount(rel);
if (max != null && max <= 1D)
return max;
}
- JoinInfo joinInfo = rel.analyzeCondition();
+ Map<Integer, KeyColumnOrigin> columnsFromLeft = resolveOrigins(mq,
rel.getLeft(), joinInfo.leftKeys);
+ Map<Integer, KeyColumnOrigin> columnsFromRight = resolveOrigins(mq,
rel.getRight(), joinInfo.rightKeys);
+
+ if (columnsFromLeft.isEmpty() || columnsFromRight.isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
+ Map<TablesPair, JoinContext> joinCtxts = new HashMap<>();
+ for (IntPair joinKeys : joinInfo.pairs()) {
+ KeyColumnOrigin leftKey = columnsFromLeft.get(joinKeys.source);
+ KeyColumnOrigin rightKey = columnsFromRight.get(joinKeys.target);
+
+ if (leftKey == null || rightKey == null) {
+ continue;
+ }
+
+ joinCtxts.computeIfAbsent(
+ new TablesPair(
+ leftKey.origin.getOriginTable(),
+ rightKey.origin.getOriginTable()
+ ),
+ key -> {
+ IgniteTable leftTable = key.left.unwrap(IgniteTable.class);
+ IgniteTable rightTable =
key.right.unwrap(IgniteTable.class);
+
+ assert leftTable != null && rightTable != null;
+
+ int leftPkSize = leftTable.distribution().getKeys().size();
+ int rightPkSize =
rightTable.distribution().getKeys().size();
+
+ return new JoinContext(leftPkSize, rightPkSize);
+ }
+ ).countKeys(leftKey, rightKey);
+ }
+
+ if (joinCtxts.isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
+ Iterator<JoinContext> it = joinCtxts.values().iterator();
+ JoinContext joinCtx = it.next();
+ while (it.hasNext()) {
+ JoinContext nextCtx = it.next();
+ if (nextCtx.joinType().strength > joinCtx.joinType().strength) {
+ joinCtx = nextCtx;
+ }
+
+ if (joinCtx.joinType().strength ==
JoiningRelationType.PK_ON_PK.strength) {
+ break;
+ }
+ }
+
+ if (joinCtx.joinType() == JoiningRelationType.UNKNOWN) {
+ // Use crude estimation instead.
+ return crudeEstimation(mq, joinInfo, rel, leftRowCnt, rightRowCnt);
+ }
+
+ double postFiltrationAdjustment = 1.0;
+
+ switch (rel.getJoinType()) {
+ case INNER:
+ case SEMI:
+ // Extra join keys as well as non-equi conditions serves as
post-filtration,
+ // therefore we need to adjust final result with a little
factor.
+ if (joinCtxts.size() != 1 || !joinInfo.isEqui())
+ postFiltrationAdjustment = NON_EQUI_COEFF;
+
+ break;
+ default:
+ break;
+ }
+
+ double baseRowCnt = 0.0;
+ Double percentageAdjustment = null;
+ if (joinCtx.joinType() == JoiningRelationType.PK_ON_PK) {
+ postFiltrationAdjustment = EQUI_COEFF;
Review Comment:
This coefficient is also applied to outer joins, so a LEFT/RIGHT join may be
estimated below the row count of its preserved input, which is impossible.
Could it be limited to INNER/SEMI joins or clamped to the preserved-side row
count?
##########
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdRowCount.java:
##########
@@ -72,48 +88,263 @@ public class IgniteMdRowCount extends RelMdRowCount {
mq.getRowCount(rel.getLeft()));
}
+ JoinInfo joinInfo = rel.analyzeCondition();
+
+ if (joinInfo.pairs().isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
// Row count estimates of 0 will be rounded up to 1.
// So, use maxRowCount where the product is very small.
- final Double left = mq.getRowCount(rel.getLeft());
- final Double right = mq.getRowCount(rel.getRight());
+ final Double leftRowCnt = mq.getRowCount(rel.getLeft());
+ final Double rightRowCnt = mq.getRowCount(rel.getRight());
- if (left == null || right == null)
+ if (leftRowCnt == null || rightRowCnt == null)
return null;
- if (left <= 1D || right <= 1D) {
+ if (leftRowCnt <= 1D || rightRowCnt <= 1D) {
Double max = mq.getMaxRowCount(rel);
if (max != null && max <= 1D)
return max;
}
- JoinInfo joinInfo = rel.analyzeCondition();
+ Map<Integer, KeyColumnOrigin> columnsFromLeft = resolveOrigins(mq,
rel.getLeft(), joinInfo.leftKeys);
+ Map<Integer, KeyColumnOrigin> columnsFromRight = resolveOrigins(mq,
rel.getRight(), joinInfo.rightKeys);
+
+ if (columnsFromLeft.isEmpty() || columnsFromRight.isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
+ Map<TablesPair, JoinContext> joinCtxts = new HashMap<>();
+ for (IntPair joinKeys : joinInfo.pairs()) {
+ KeyColumnOrigin leftKey = columnsFromLeft.get(joinKeys.source);
+ KeyColumnOrigin rightKey = columnsFromRight.get(joinKeys.target);
+
+ if (leftKey == null || rightKey == null) {
+ continue;
+ }
+
+ joinCtxts.computeIfAbsent(
+ new TablesPair(
+ leftKey.origin.getOriginTable(),
+ rightKey.origin.getOriginTable()
+ ),
+ key -> {
+ IgniteTable leftTable = key.left.unwrap(IgniteTable.class);
+ IgniteTable rightTable =
key.right.unwrap(IgniteTable.class);
+
+ assert leftTable != null && rightTable != null;
+
+ int leftPkSize = leftTable.distribution().getKeys().size();
+ int rightPkSize =
rightTable.distribution().getKeys().size();
+
+ return new JoinContext(leftPkSize, rightPkSize);
+ }
+ ).countKeys(leftKey, rightKey);
+ }
+
+ if (joinCtxts.isEmpty()) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
+ }
+
+ Iterator<JoinContext> it = joinCtxts.values().iterator();
+ JoinContext joinCtx = it.next();
+ while (it.hasNext()) {
+ JoinContext nextCtx = it.next();
+ if (nextCtx.joinType().strength > joinCtx.joinType().strength) {
+ joinCtx = nextCtx;
+ }
+
+ if (joinCtx.joinType().strength ==
JoiningRelationType.PK_ON_PK.strength) {
+ break;
+ }
+ }
+
+ if (joinCtx.joinType() == JoiningRelationType.UNKNOWN) {
+ // Use crude estimation instead.
+ return crudeEstimation(mq, joinInfo, rel, leftRowCnt, rightRowCnt);
+ }
+
+ double postFiltrationAdjustment = 1.0;
+
+ switch (rel.getJoinType()) {
+ case INNER:
+ case SEMI:
+ // Extra join keys as well as non-equi conditions serves as
post-filtration,
+ // therefore we need to adjust final result with a little
factor.
+ if (joinCtxts.size() != 1 || !joinInfo.isEqui())
+ postFiltrationAdjustment = NON_EQUI_COEFF;
+
+ break;
+ default:
+ break;
+ }
+
+ double baseRowCnt = 0.0;
+ Double percentageAdjustment = null;
+ if (joinCtx.joinType() == JoiningRelationType.PK_ON_PK) {
+ postFiltrationAdjustment = EQUI_COEFF;
+
+ if (rel.getJoinType() == JoinRelType.INNER || rel.getJoinType() ==
JoinRelType.SEMI) {
+ // Assume we have two fact tables SALES and RETURNS sharing
the same primary key. Every item
+ // can be sold, but only items which were sold can be returned
back, therefore
+ // size(SALES) > size(RETURNS). When joining SALES on RETURNS
by primary key, the estimated
+ // result size will be the same as the size of the smallest
table (RETURNS in this case),
+ // adjusted by the percentage of rows of the biggest table
(SALES in this case; percentage
+ // adjustment is required to account for predicates pushed
down to the table, e.g. we are
+ // interested in returns of items with certain category)
+ if (leftRowCnt > rightRowCnt) {
+ baseRowCnt = rightRowCnt;
+ percentageAdjustment =
mq.getPercentageOriginalRows(rel.getLeft());
+ }
+ else {
+ baseRowCnt = leftRowCnt;
+ percentageAdjustment =
mq.getPercentageOriginalRows(rel.getRight());
+ }
+ }
+ else if (rel.getJoinType() == JoinRelType.LEFT) {
+ baseRowCnt = leftRowCnt;
+ }
+ else if (rel.getJoinType() == JoinRelType.RIGHT) {
+ baseRowCnt = rightRowCnt;
+ }
+ else if (rel.getJoinType() == JoinRelType.FULL) {
+ Double selectivity = mq.getSelectivity(rel,
rel.getCondition());
+
+ // Fall-back.
+ if (selectivity == null) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel,
rel.getCondition());
+ }
+
+ baseRowCnt = rightRowCnt + leftRowCnt;
+ percentageAdjustment = 1.0 - selectivity;
+ }
+ }
+ else if (joinCtx.joinType() == JoiningRelationType.FK_ON_PK) {
+ // For foreign key joins the base table is the one which is joined
by non-primary key columns.
+ if (rel.getJoinType() == JoinRelType.INNER || rel.getJoinType() ==
JoinRelType.SEMI) {
+ Double selectivity = mq.getSelectivity(rel,
rel.getCondition());
+ baseRowCnt = leftRowCnt * selectivity;
+ percentageAdjustment =
mq.getPercentageOriginalRows(rel.getRight());
+ }
+ else if (rel.getJoinType() == JoinRelType.LEFT ||
rel.getJoinType() == JoinRelType.RIGHT) {
+ baseRowCnt = leftRowCnt;
+ }
+ else if (rel.getJoinType() == JoinRelType.FULL) {
+ Double selectivity = mq.getSelectivity(rel,
rel.getCondition());
+
+ // Fall-back.
+ if (selectivity == null) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel,
rel.getCondition());
+ }
+
+ baseRowCnt = rightRowCnt + leftRowCnt;
+ percentageAdjustment = 1.0 - selectivity;
+ }
+ }
+ else { // PK_ON_FK
+ if (rel.getJoinType() == JoinRelType.INNER || rel.getJoinType() ==
JoinRelType.SEMI) {
+ baseRowCnt = rightRowCnt;
+ percentageAdjustment =
mq.getPercentageOriginalRows(rel.getLeft());
+ }
+ else if (rel.getJoinType() == JoinRelType.RIGHT ||
rel.getJoinType() == JoinRelType.LEFT) {
+ baseRowCnt = rightRowCnt;
+ }
+ else if (rel.getJoinType() == JoinRelType.FULL) {
+ Double selectivity = mq.getSelectivity(rel,
rel.getCondition());
+
+ // Fall-back.
+ if (selectivity == null) {
+ // Fall-back to calcite's implementation.
+ return RelMdUtil.getJoinRowCount(mq, rel,
rel.getCondition());
+ }
+
+ baseRowCnt = rightRowCnt + leftRowCnt;
+ percentageAdjustment = 1.0 - selectivity;
+ }
+ }
+
+ if (percentageAdjustment == null) {
+ // No info, let's be conservative.
+ percentageAdjustment = 1.0;
+ }
+
+ return baseRowCnt * percentageAdjustment * postFiltrationAdjustment;
+ }
+ /** */
+ private static Map<Integer, KeyColumnOrigin>
resolveOrigins(RelMetadataQuery mq, RelNode joinShoulder, ImmutableIntList
keys) {
+ GridLeanMap<Integer, KeyColumnOrigin> origins = new GridLeanMap<>();
+
+ for (int i : keys) {
+ if (origins.containsKey(i)) {
+ continue;
+ }
+
+ RelColumnOrigin origin = mq.getColumnOrigin(joinShoulder, i);
+ if (origin == null) {
+ continue;
+ }
+
+ IgniteTable table =
origin.getOriginTable().unwrap(IgniteTable.class);
+ if (table == null || !table.distribution().function().affinity())
+ continue;
+
+ // Keys can relate to affinity not pk, just assumption here.
+ ImmutableIntList distKeys = table.distribution().getKeys();
+
+ int idx = distKeys.indexOf(origin.getOriginColumnOrdinal());
+
+ origins.put(i, new KeyColumnOrigin(origin, idx));
+ }
+
+ return origins;
+ }
+
+ /**
+ * @param origin
+ * @param positionInKey
+ */
+ private record KeyColumnOrigin(RelColumnOrigin origin, int positionInKey)
{ }
+
+ /** This part of estimation is applicable for distributions different from
hash, i.e. broadcast, single. */
+ private static double crudeEstimation(RelMetadataQuery mq, JoinInfo
joinInfo, Join rel, Double leftRowCnt, Double rightRowCnt) {
ImmutableIntList leftKeys = joinInfo.leftKeys;
ImmutableIntList rightKeys = joinInfo.rightKeys;
- double selectivity = mq.getSelectivity(rel, rel.getCondition());
+ Double selectivity = mq.getSelectivity(rel, rel.getCondition());
+
+ if (selectivity == null) {
+ // Fall-back to calcite's implementation.
+ RelMdUtil.getJoinRowCount(mq, rel, rel.getCondition());
Review Comment:
The fallback result is ignored here, so execution continues with
`selectivity == null` and later fails during unboxing. This should `return
RelMdUtil.getJoinRowCount(...);` crudeEstimation may also need to return
`Double`.
##########
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdRowCount.java:
##########
@@ -17,42 +17,58 @@
package org.apache.ignite.internal.processors.query.calcite.metadata;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.core.Intersect;
import org.apache.calcite.rel.core.Join;
import org.apache.calcite.rel.core.JoinInfo;
import org.apache.calcite.rel.core.JoinRelType;
import org.apache.calcite.rel.core.Minus;
import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.metadata.BuiltInMetadata;
import org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
import org.apache.calcite.rel.metadata.RelMdRowCount;
import org.apache.calcite.rel.metadata.RelMdUtil;
import org.apache.calcite.rel.metadata.RelMetadataProvider;
import org.apache.calcite.rel.metadata.RelMetadataQuery;
import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.util.BuiltInMethod;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.calcite.util.ImmutableIntList;
import org.apache.calcite.util.Util;
+import org.apache.calcite.util.mapping.IntPair;
import org.apache.ignite.internal.processors.query.calcite.rel.IgniteAggregate;
import org.apache.ignite.internal.processors.query.calcite.rel.IgniteLimit;
import
org.apache.ignite.internal.processors.query.calcite.rel.IgniteSortedIndexSpool;
import
org.apache.ignite.internal.processors.query.calcite.rel.IgniteTableModify;
+import org.apache.ignite.internal.processors.query.calcite.schema.IgniteTable;
+import org.apache.ignite.internal.util.GridLeanMap;
import org.apache.ignite.internal.util.typedef.F;
import org.jetbrains.annotations.Nullable;
import static org.apache.calcite.util.NumberUtil.multiply;
+
/** */
@SuppressWarnings("unused") // actually all methods are used by runtime
generated classes
public class IgniteMdRowCount extends RelMdRowCount {
+ /** */
+ private static final double NON_EQUI_COEFF = 0.7;
+
+ /** */
+ public static final double EQUI_COEFF = 0.8;
+
/** */
public static final RelMetadataProvider SOURCE =
- ReflectiveRelMetadataProvider.reflectiveSource(
- BuiltInMethod.ROW_COUNT.method, new IgniteMdRowCount());
+ ReflectiveRelMetadataProvider.reflectiveSource(new IgniteMdRowCount(),
BuiltInMetadata.RowCount.Handler.class);
/** {@inheritDoc} */
@Override public Double getRowCount(Join rel, RelMetadataQuery mq) {
- return rel.estimateRowCount(mq);
+ return joinRowCount(mq, rel);
Review Comment:
This bypasses `IgniteCorrelatedNestedLoopJoin.estimateRowCount()`, which
removes the join-condition selectivity because it is already applied by an
external filter. The generic estimator may therefore count the condition twice.
Could we preserve the specialized CNL handling here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
