github-actions[bot] commented on code in PR #69871:
URL: https://github.com/apache/doris/pull/69871#discussion_r4421163854
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -883,6 +919,112 @@ public static PhysicalPlan chooseBestPlan(Group
rootGroup, PhysicalProperties ph
}
}
+ /**
+ * Choose the best physical plan considering USE_MV hint as a hard
constraint.
+ * If USE_MV hint is present with rewrite-success MVs, the returned plan
must
+ * contain those MVs. Otherwise, falls back to standard cost-based
selection.
+ *
+ * @param rootGroup the root memo group
+ * @param physicalProperties required physical properties
+ * @param cascadesContext the cascades context
+ * @return physical plan respecting USE_MV hard constraint
+ */
+ public static PhysicalPlan chooseBestPlanConsideringUseMv(Group rootGroup,
+ PhysicalProperties physicalProperties, CascadesContext
cascadesContext)
+ throws AnalysisException {
+ StatementContext stmtCtx = cascadesContext.getStatementContext();
+ Set<List<String>> requiredUseMvIdentifiers =
computeRequiredUseMvIdentifiers(stmtCtx);
+ if (requiredUseMvIdentifiers.isEmpty()) {
+ return chooseBestPlan(rootGroup, physicalProperties,
cascadesContext);
+ }
+ Map<Group, Set<List<String>>> mvReachableCache = new HashMap<>();
+ computeReachableMvIdentifiers(rootGroup, mvReachableCache,
requiredUseMvIdentifiers);
+ Set<List<String>> rootReachable = mvReachableCache.get(rootGroup);
+ // Drop the required MVs which the rewrite can not produce, and mark
them as UnUsed,
+ // USE_MV is best-effort and should not fail the query.
+ requiredUseMvIdentifiers = dropUnreachableUseMvIdentifiers(stmtCtx,
requiredUseMvIdentifiers,
+ rootReachable == null ? ImmutableSet.of() : rootReachable);
+ if (requiredUseMvIdentifiers.isEmpty()) {
+ return chooseBestPlan(rootGroup, physicalProperties,
cascadesContext);
+ }
+ return chooseBestPlanConsideringRequiredUseMv(rootGroup,
physicalProperties,
+ requiredUseMvIdentifiers, mvReachableCache, cascadesContext);
+ }
+
+ /**
+ * Choose the best plan which contains the required USE_MV
materializations.
+ * <p>
+ * Several hinted materializations may be alternatives of each other and
can not be chosen at the
+ * same time, so only the single required materialization is enforced. The
cost model already makes
+ * the hinted materializations the cheapest ones, so the cost based plan
is used for the others.
+ */
+ private static PhysicalPlan chooseBestPlanConsideringRequiredUseMv(Group
rootGroup,
+ PhysicalProperties physicalProperties, Set<List<String>>
requiredUseMvIdentifiers,
+ Map<Group, Set<List<String>>> mvReachableCache, CascadesContext
cascadesContext)
+ throws AnalysisException {
+ if (requiredUseMvIdentifiers.size() == 1) {
+ PhysicalPlan physicalPlan =
chooseBestPlanWithRequiredMaterializations(rootGroup,
+ physicalProperties, requiredUseMvIdentifiers,
mvReachableCache, cascadesContext);
+ if (providesProperties(physicalPlan.getPhysicalProperties(),
physicalProperties)) {
+ return physicalPlan;
+ }
+ // The forced plan does not provide the required output properties
anymore, for example a
+ // materialization replaced one side of a colocate join and both
sides are shuffled now.
+ PhysicalPlan enforced = addExchange(physicalPlan,
physicalProperties.getDistributionSpec());
+ if
(!enforced.getPhysicalProperties().getOrderSpec().satisfy(physicalProperties.getOrderSpec()))
{
+ enforced = addSort(enforced,
physicalProperties.getOrderSpec());
+ }
+ return enforced;
+ }
+ return chooseBestPlan(rootGroup, physicalProperties, cascadesContext);
Review Comment:
[P2] Enforce a reachable alternative for multiple-MV hints
This fallback can choose none of the hinted MVs even when both rewrites
succeed. For example, use `NOT_IN_RBO`, base `A(dt,k,v) DUPLICATE KEY(dt,k)`,
and refreshed m1/m2 each defined as `SELECT k,dt,v FROM A WHERE dt>=0`. For
`SELECT /*+ USE_MV(m1,m2) */ k,dt,v FROM A WHERE dt=1`, the memo contains
`Filter(dt=1) -> Scan(A)` and each valid `Project -> Filter(m.dt=1) ->
Scan(mi)` replacement.
Zero scan cost does not make the whole MV plan cheapest: A's dt prefix
avoids one `filter_cost_factor` that each MV's leading k does not. Set that
factor large enough to outweigh the finite base scan cost and the base
alternative remains cheaper. Both MV identifiers enter the required/reachable
sets, but size>1 reaches this unconditional `chooseBestPlan`, selecting A and
reporting both unused. Adding a valid alternative therefore disables the new
hard constraint. Select the cheapest plan containing at least one reachable
hinted alternative instead of dropping enforcement for the entire set.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -1006,6 +1148,432 @@ private PhysicalPlan chooseBestPlanWithRequiredGroups(
return physicalPlan;
}
+ /**
+ * Compute the set of USE_MV hint qualifiers that successfully rewrote.
+ * Marks USE_MV hint parameters that did NOT rewrite-success as UnUsed.
+ */
+ private static Set<List<String>>
computeRequiredUseMvIdentifiers(StatementContext statementContext) {
+ Optional<UseMvHint> useMvHint =
statementContext.getUseMvHint("USE_MV");
+ if (!useMvHint.isPresent()) {
+ return ImmutableSet.of();
+ }
+ Set<List<String>> useMvQualifiers =
useMvHint.get().getUseMvTableColumnMap().keySet();
+ Set<List<String>> rewrittenSuccessSet =
statementContext.getMaterializationRewrittenSuccessSet();
+ Set<List<String>> required = new HashSet<>(useMvQualifiers);
+ required.retainAll(rewrittenSuccessSet);
+ // Mark USE_MV qualifiers that did not rewrite-success as UnUsed
+ Set<List<String>> notRewritten = new HashSet<>(useMvQualifiers);
+ notRewritten.removeAll(rewrittenSuccessSet);
+ for (List<String> qualifier : notRewritten) {
+ // Update hint map to mark as not-chosen (false) with failure
reason
+ useMvHint.get().getUseMvTableColumnMap().put(qualifier, false);
+ }
+ if (!notRewritten.isEmpty()) {
+ // If some USE_MV hints failed to rewrite, set status to UNUSED
with reason
+ useMvHint.get().setStatus(Hint.HintStatus.UNUSED);
+ useMvHint.get().setErrorMessage("MV(s) not rewrite-success after
CBO: " + notRewritten);
+ }
+ return required;
+ }
+
+ /**
+ * Extract the MV qualifier(s) from a scan plan if it matches a required
USE_MV identifier.
+ * Returns the set of MV qualifiers this plan represents.
+ */
+ private static Set<List<String>> getScanMvIdentifiers(Plan plan,
+ Set<List<String>> requiredUseMvIdentifiers) {
+ Set<List<String>> result = new HashSet<>();
+ if (plan instanceof PhysicalOlapScan) {
+ PhysicalOlapScan scan = (PhysicalOlapScan) plan;
+ OlapTable table = scan.getTable();
+ if (table.getBaseIndexId() != scan.getSelectedIndexId()) {
+ List<String> qualifier = new ArrayList<>();
+ qualifier.add(table.getDatabase().getCatalog().getName());
+ qualifier.add(table.getDatabase().getFullName());
+ qualifier.add(table.getName());
+
qualifier.add(table.getIndexNameById(scan.getSelectedIndexId()));
+ if (requiredUseMvIdentifiers.contains(qualifier)) {
+ result.add(qualifier);
+ }
+ }
+ }
+ if (plan instanceof PhysicalCatalogRelation) {
+ PhysicalCatalogRelation rel = (PhysicalCatalogRelation) plan;
+ if (rel.getTable() instanceof MTMV) {
+ List<String> qualifier = rel.getTable().getFullQualifiers();
+ if (requiredUseMvIdentifiers.contains(qualifier)) {
+ result.add(qualifier);
+ }
+ }
+ // Also check for sync MV via OlapScan (PhysicalOlapScan is a
PhysicalCatalogRelation)
+ if (rel instanceof PhysicalOlapScan) {
+ PhysicalOlapScan scan = (PhysicalOlapScan) rel;
+ OlapTable table = scan.getTable();
+ if (table.getBaseIndexId() != scan.getSelectedIndexId()) {
+ List<String> qualifier = new ArrayList<>();
+ qualifier.add(table.getDatabase().getCatalog().getName());
+ qualifier.add(table.getDatabase().getFullName());
+ qualifier.add(table.getName());
+
qualifier.add(table.getIndexNameById(scan.getSelectedIndexId()));
+ if (requiredUseMvIdentifiers.contains(qualifier)) {
+ result.add(qualifier);
+ }
+ }
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Memoized bottom-up DFS: compute all MV identifiers reachable from this
group's subtree
+ * through physical GroupExpressions (including enforcers).
+ */
+ private static void computeReachableMvIdentifiers(Group group,
+ Map<Group, Set<List<String>>> cache, Set<List<String>>
requiredUseMvIdentifiers) {
+ if (cache.containsKey(group)) {
+ return;
+ }
+ Set<List<String>> reachable = new HashSet<>();
+ // Put in cache BEFORE recursing to break cycles
+ cache.put(group, reachable);
+
+ for (GroupExpression ge : group.getPhysicalExpressions()) {
+ Set<List<String>> geReachable = getScanMvIdentifiers(ge.getPlan(),
requiredUseMvIdentifiers);
+ for (Group child : ge.children()) {
+ computeReachableMvIdentifiers(child, cache,
requiredUseMvIdentifiers);
+ geReachable.addAll(cache.get(child));
+ }
+ reachable.addAll(geReachable);
+ }
+ for (GroupExpression enforcer : group.getEnforcers().values()) {
+ Set<List<String>> enforcerReachable = new HashSet<>();
+ for (Group child : enforcer.children()) {
+ computeReachableMvIdentifiers(child, cache,
requiredUseMvIdentifiers);
+ enforcerReachable.addAll(cache.get(child));
+ }
+ reachable.addAll(enforcerReachable);
+ }
+ }
+
+ /**
+ * Choose the cheapest physical plan that contains all required MV
identifiers.
+ * At each group, picks the cheapest physical GroupExpression whose own +
children's
+ * reachable sets collectively cover all remaining required MV
identifiers, then
+ * distributes the remaining requirements among children and recurses.
+ */
+ private static PhysicalPlan chooseBestPlanWithRequiredMaterializations(
+ Group group, PhysicalProperties properties,
+ Set<List<String>> remaining, Map<Group, Set<List<String>>>
reachableCache,
+ CascadesContext cascadesContext) {
+ remaining = new HashSet<>(remaining);
+
+ // Remove MVs that this group's own scan expressions cover
+ Set<List<String>> ownMvs = new HashSet<>();
+ for (GroupExpression ge : group.getPhysicalExpressions()) {
+ ownMvs.addAll(getScanMvIdentifiers(ge.getPlan(), remaining));
+ }
+ remaining.removeAll(ownMvs);
+
+ if (remaining.isEmpty()) {
+ return chooseBestPlan(group, properties, cascadesContext);
+ }
+
+ GroupExpression chosenGE = null;
+ double chosenCost = Double.MAX_VALUE;
+
+ // Check all physical GEs in this group
+ for (GroupExpression ge : group.getPhysicalExpressions()) {
+ if (!ge.getLowestCostTable().containsKey(properties)) {
+ continue;
+ }
+ double geCost = ge.getCostByProperties(properties);
+ if (geCost >= chosenCost) {
+ continue;
+ }
+ Set<List<String>> childReachable = new HashSet<>();
+ for (Group child : ge.children()) {
+ Set<List<String>> cr = reachableCache.get(child);
+ if (cr != null) {
+ childReachable.addAll(cr);
+ }
+ }
+ if (childReachable.containsAll(remaining)) {
+ chosenGE = ge;
+ chosenCost = geCost;
+ }
+ }
+
+ // Also check enforcers
+ for (GroupExpression enforcer : group.getEnforcers().values()) {
+ if (!enforcer.getLowestCostTable().containsKey(properties)) {
+ continue;
+ }
+ double eCost = enforcer.getCostByProperties(properties);
+ if (eCost >= chosenCost) {
+ continue;
+ }
+ Set<List<String>> childReachable = new HashSet<>();
+ for (Group child : enforcer.children()) {
+ Set<List<String>> cr = reachableCache.get(child);
+ if (cr != null) {
+ childReachable.addAll(cr);
+ }
+ }
+ if (childReachable.containsAll(remaining)) {
+ chosenGE = enforcer;
+ chosenCost = eCost;
+ }
+ }
+
+ if (chosenGE == null) {
+ // The required materialization can not be produced with the
required properties, for example
+ // the hinted materialization is not costed for the required
distribution, so fall back to the
+ // cost based plan of this subtree instead of failing the query.
+ markUseMvNotChosen(cascadesContext.getStatementContext(),
remaining);
+ return chooseBestPlan(group, properties, cascadesContext);
+ }
+
+ // Handle enforcer marking (same as chooseBestPlan)
+ if ((chosenGE.getPlan() instanceof PhysicalDistribute
+ && group.getEnforcerSpecs().containsKey(
+ ((PhysicalDistribute<?>)
chosenGE.getPlan()).getDistributionSpec()))
+ || group.getEnforcers().containsKey(chosenGE)) {
+ group.addChosenEnforcerId(chosenGE.getId().asInt());
+ group.addChosenEnforcerProperties(properties);
+ } else {
+ group.setChosenProperties(properties);
+ group.setChosenGroupExpressionId(chosenGE.getId().asInt());
+ }
+
+ // Distribute remaining among children and recurse
+ List<PhysicalProperties> inputProps =
chosenGE.getInputPropertiesList(properties);
+ List<Plan> planChildren = Lists.newArrayList();
+ for (int i = 0; i < chosenGE.arity(); i++) {
+ Group child = chosenGE.child(i);
+ Set<List<String>> childReachable =
reachableCache.getOrDefault(child, new HashSet<>());
+ Set<List<String>> childRemaining = new HashSet<>(remaining);
+ childRemaining.retainAll(childReachable);
+ planChildren.add(chooseBestPlanWithRequiredMaterializations(
+ child, inputProps.get(i), childRemaining, reachableCache,
cascadesContext));
+ }
+
+ // The children were chosen by the materialization reachability
instead of the cost, so the
+ // property contract of the costed group expression can be stale: a
child can be replaced by a
+ // materialization which is laid out by another relation. Re-derive
the actual children outputs
+ // and enforce the missing properties again, instead of keeping the
costed contract.
+ boolean contractHolds = childrenProvideProperties(inputProps,
planChildren);
+ if (!contractHolds) {
+ enforceChildrenProperties(inputProps, planChildren);
+ }
+
+ Plan plan = chosenGE.getPlan().withChildren(planChildren);
+ if (!(plan instanceof PhysicalPlan)) {
+ throw new AnalysisException("Result plan must be PhysicalPlan");
+ }
+ plan = plan.withGroupExpression(Optional.of(chosenGE));
+ PhysicalProperties outputProperties = contractHolds
+ ? chosenGE.getOutputProperties(properties)
+ : deriveOutputProperties(plan, planChildren, cascadesContext);
+ return ((PhysicalPlan)
plan).withPhysicalPropertiesAndStats(outputProperties, group.getStatistics());
+ }
+
+ /**
+ * Check whether the actual physical properties provide the required ones.
+ * <p>
+ * {@link DistributionSpecHash#satisfy} and {@link
DistributionSpecHash#equals} compare the shuffle
+ * type and the shuffle columns only, they ignore the relation, the index
and the partitions. A
+ * materialization which replaces a relation is laid out by its own table,
so it compares equal to
+ * the distribution of the replaced relation even though the two are not
co-located. The layout
+ * identity must be compared explicitly when the property regulation of
the cost based selection is
+ * by-passed.
+ */
+ private static boolean providesProperties(PhysicalProperties actual,
PhysicalProperties required) {
+ return providesDistribution(actual.getDistributionSpec(),
required.getDistributionSpec())
+ && actual.getOrderSpec().satisfy(required.getOrderSpec());
+ }
+
+ private static boolean providesDistribution(DistributionSpec actual,
DistributionSpec required) {
+ if (!actual.satisfy(required)) {
+ return false;
+ }
+ if (!(required instanceof DistributionSpecHash) || !(actual instanceof
DistributionSpecHash)) {
+ return true;
+ }
+ DistributionSpecHash requiredHash = (DistributionSpecHash) required;
+ ShuffleType requiredShuffleType = requiredHash.getShuffleType();
+ if (requiredShuffleType != ShuffleType.NATURAL && requiredShuffleType
!= ShuffleType.STORAGE_BUCKETED) {
+ // The required distribution does not identify a storage layout.
+ return true;
+ }
+ DistributionSpecHash actualHash = (DistributionSpecHash) actual;
+ return actualHash.getTableId() == requiredHash.getTableId()
+ && actualHash.getSelectedIndexId() ==
requiredHash.getSelectedIndexId()
+ &&
actualHash.getPartitionIds().equals(requiredHash.getPartitionIds());
+ }
+
+ /**
+ * Check whether the actual distribution delivers the shuffle columns of
the required one. Unlike
+ * {@link #providesDistribution}, a distribution with another storage
layout is accepted, because a
+ * single child only propagates its distribution to the group expression,
so replacing it with a
+ * materialization does not need a data movement as long as the shuffle
columns keep matching.
+ */
+ private static boolean providesShuffleColumns(DistributionSpec actual,
DistributionSpec required) {
+ if (!(required instanceof DistributionSpecHash) || !(actual instanceof
DistributionSpecHash)) {
+ return actual.satisfy(required);
+ }
+ return actual.satisfy(((DistributionSpecHash)
required).withShuffleType(ShuffleType.REQUIRE));
+ }
+
+ /**
+ * Check whether every forced child provides the properties the memo
costed the group expression
+ * with.
+ */
+ private static boolean childrenProvideProperties(List<PhysicalProperties>
inputProps,
+ List<Plan> planChildren) {
+ for (int i = 0; i < planChildren.size(); i++) {
+ if
(!providesProperties(getPhysicalProperties(planChildren.get(i)),
inputProps.get(i))) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Enforce the input properties of the group expression on the forced
children plans, the same way
+ * {@link org.apache.doris.nereids.properties.ChildrenPropertiesRegulator}
does it during the cost
+ * based selection.
+ */
+ private static void enforceChildrenProperties(List<PhysicalProperties>
inputProps,
+ List<Plan> planChildren) {
+ boolean multiChild = planChildren.size() > 1;
+ for (int i = 0; i < planChildren.size(); i++) {
+ PhysicalProperties required = inputProps.get(i);
+ PhysicalProperties actual =
getPhysicalProperties(planChildren.get(i));
+ if (!providesProperties(actual, required)) {
+ // Every hash distributed child of a join or of a set
operation is realigned, not only
+ // the replaced one: the data layout pairing of the costed
plan does not hold anymore
+ // once one child of such a group expression is replaced by a
materialization.
+ if (multiChild && required.getDistributionSpec() instanceof
DistributionSpecHash) {
+ planChildren.set(i, addExchange(planChildren.get(i),
required.getDistributionSpec()));
+ } else if (!providesShuffleColumns(
+ actual.getDistributionSpec(),
required.getDistributionSpec())) {
+ planChildren.set(i, addExchange(planChildren.get(i),
required.getDistributionSpec()));
+ }
+ }
+ if
(!getPhysicalProperties(planChildren.get(i)).getOrderSpec().satisfy(required.getOrderSpec()))
{
+ planChildren.set(i, addSort(planChildren.get(i),
required.getOrderSpec()));
+ }
+ }
+ }
+
+ /**
+ * Add an exchange which delivers the required distribution. A hash
distribution without a shuffle
+ * type is delivered by a hash shuffle, the same way
+ * {@link
org.apache.doris.nereids.properties.EnforceMissingPropertiesHelper} does it.
+ */
+ private static PhysicalPlan addExchange(Plan child, DistributionSpec
required) {
+ DistributionSpec exchangeDistribution = required instanceof
DistributionSpecHash
+ ? ((DistributionSpecHash)
required).withShuffleType(ShuffleType.EXECUTION_BUCKETED)
+ : required;
+ return new PhysicalDistribute<>(exchangeDistribution,
Optional.empty(), child.getLogicalProperties(),
+ new PhysicalProperties(exchangeDistribution),
child.getStats(), child);
+ }
+
+ private static PhysicalPlan addSort(Plan child, OrderSpec requiredOrder) {
+ SortPhase sortPhase = requiredOrder instanceof MustLocalSortOrderSpec
+ ? SortPhase.LOCAL_SORT : SortPhase.MERGE_SORT;
Review Comment:
[P2] Restore local ordering before constructing a merge sort
An ordinary `OrderSpec` does not imply that the child is already a local
sort. A reachable example is a twice-used materialized CTE containing
`row_number() OVER (PARTITION BY k ORDER BY v)`, with both consumers retaining
the row number. Use `NOT_IN_RBO`, `enable_cte_materialize=true`,
`inline_cte_referenced_threshold=1`, `cte_inline_mode=-1`, base `A(dt,k,v)
DUPLICATE KEY(dt,k) HASH(k)`, and refreshed `m AS SELECT k,dt,v FROM A WHERE
dt>=0 HASH(k)`. A sufficiently large finite `filter_cost_factor` keeps the base
filter cheaper because only A matches the dt prefix.
The reduced producer is `CTEProducer -> Window(k,v) -> LocalSort(k,v) ->
Filter(dt=1) -> Scan(A)`. Forcing m propagates its different storage identity
through the unary nodes. CTEAnchor records the original producer's actual
`NATURAL(A)+Order(k,v)` input, then `enforceChildrenProperties` treats the
anchor's two children as paired inputs and inserts an exchange above the
producer. That drops its order, so this helper builds `MERGE_SORT -> Exchange
-> CTEProducer -> Window(...)`.
`visitPhysicalCTEProducer` returns the window's fragment, whose root is
`AnalyticEvalNode` (or an upper projection's SelectNode).
`visitPhysicalQuickSort` casts the exchange's immediate child to `SortNode`, so
translation throws `ClassCastException`. Respect CTEAnchor's producer contract
and create a real local sort when restoring missing order; MERGE_SORT requires
an already sorted input.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -1006,6 +1148,432 @@ private PhysicalPlan chooseBestPlanWithRequiredGroups(
return physicalPlan;
}
+ /**
+ * Compute the set of USE_MV hint qualifiers that successfully rewrote.
+ * Marks USE_MV hint parameters that did NOT rewrite-success as UnUsed.
+ */
+ private static Set<List<String>>
computeRequiredUseMvIdentifiers(StatementContext statementContext) {
+ Optional<UseMvHint> useMvHint =
statementContext.getUseMvHint("USE_MV");
+ if (!useMvHint.isPresent()) {
+ return ImmutableSet.of();
+ }
+ Set<List<String>> useMvQualifiers =
useMvHint.get().getUseMvTableColumnMap().keySet();
+ Set<List<String>> rewrittenSuccessSet =
statementContext.getMaterializationRewrittenSuccessSet();
+ Set<List<String>> required = new HashSet<>(useMvQualifiers);
+ required.retainAll(rewrittenSuccessSet);
+ // Mark USE_MV qualifiers that did not rewrite-success as UnUsed
+ Set<List<String>> notRewritten = new HashSet<>(useMvQualifiers);
+ notRewritten.removeAll(rewrittenSuccessSet);
+ for (List<String> qualifier : notRewritten) {
+ // Update hint map to mark as not-chosen (false) with failure
reason
+ useMvHint.get().getUseMvTableColumnMap().put(qualifier, false);
+ }
+ if (!notRewritten.isEmpty()) {
+ // If some USE_MV hints failed to rewrite, set status to UNUSED
with reason
+ useMvHint.get().setStatus(Hint.HintStatus.UNUSED);
+ useMvHint.get().setErrorMessage("MV(s) not rewrite-success after
CBO: " + notRewritten);
+ }
+ return required;
+ }
+
+ /**
+ * Extract the MV qualifier(s) from a scan plan if it matches a required
USE_MV identifier.
+ * Returns the set of MV qualifiers this plan represents.
+ */
+ private static Set<List<String>> getScanMvIdentifiers(Plan plan,
+ Set<List<String>> requiredUseMvIdentifiers) {
+ Set<List<String>> result = new HashSet<>();
+ if (plan instanceof PhysicalOlapScan) {
+ PhysicalOlapScan scan = (PhysicalOlapScan) plan;
+ OlapTable table = scan.getTable();
+ if (table.getBaseIndexId() != scan.getSelectedIndexId()) {
+ List<String> qualifier = new ArrayList<>();
+ qualifier.add(table.getDatabase().getCatalog().getName());
+ qualifier.add(table.getDatabase().getFullName());
+ qualifier.add(table.getName());
+
qualifier.add(table.getIndexNameById(scan.getSelectedIndexId()));
+ if (requiredUseMvIdentifiers.contains(qualifier)) {
+ result.add(qualifier);
+ }
+ }
+ }
+ if (plan instanceof PhysicalCatalogRelation) {
+ PhysicalCatalogRelation rel = (PhysicalCatalogRelation) plan;
+ if (rel.getTable() instanceof MTMV) {
+ List<String> qualifier = rel.getTable().getFullQualifiers();
+ if (requiredUseMvIdentifiers.contains(qualifier)) {
+ result.add(qualifier);
+ }
+ }
+ // Also check for sync MV via OlapScan (PhysicalOlapScan is a
PhysicalCatalogRelation)
+ if (rel instanceof PhysicalOlapScan) {
+ PhysicalOlapScan scan = (PhysicalOlapScan) rel;
+ OlapTable table = scan.getTable();
+ if (table.getBaseIndexId() != scan.getSelectedIndexId()) {
+ List<String> qualifier = new ArrayList<>();
+ qualifier.add(table.getDatabase().getCatalog().getName());
+ qualifier.add(table.getDatabase().getFullName());
+ qualifier.add(table.getName());
+
qualifier.add(table.getIndexNameById(scan.getSelectedIndexId()));
+ if (requiredUseMvIdentifiers.contains(qualifier)) {
+ result.add(qualifier);
+ }
+ }
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Memoized bottom-up DFS: compute all MV identifiers reachable from this
group's subtree
+ * through physical GroupExpressions (including enforcers).
+ */
+ private static void computeReachableMvIdentifiers(Group group,
+ Map<Group, Set<List<String>>> cache, Set<List<String>>
requiredUseMvIdentifiers) {
+ if (cache.containsKey(group)) {
+ return;
+ }
+ Set<List<String>> reachable = new HashSet<>();
+ // Put in cache BEFORE recursing to break cycles
+ cache.put(group, reachable);
+
+ for (GroupExpression ge : group.getPhysicalExpressions()) {
+ Set<List<String>> geReachable = getScanMvIdentifiers(ge.getPlan(),
requiredUseMvIdentifiers);
+ for (Group child : ge.children()) {
+ computeReachableMvIdentifiers(child, cache,
requiredUseMvIdentifiers);
+ geReachable.addAll(cache.get(child));
+ }
+ reachable.addAll(geReachable);
+ }
+ for (GroupExpression enforcer : group.getEnforcers().values()) {
+ Set<List<String>> enforcerReachable = new HashSet<>();
+ for (Group child : enforcer.children()) {
+ computeReachableMvIdentifiers(child, cache,
requiredUseMvIdentifiers);
+ enforcerReachable.addAll(cache.get(child));
+ }
+ reachable.addAll(enforcerReachable);
+ }
+ }
+
+ /**
+ * Choose the cheapest physical plan that contains all required MV
identifiers.
+ * At each group, picks the cheapest physical GroupExpression whose own +
children's
+ * reachable sets collectively cover all remaining required MV
identifiers, then
+ * distributes the remaining requirements among children and recurses.
+ */
+ private static PhysicalPlan chooseBestPlanWithRequiredMaterializations(
+ Group group, PhysicalProperties properties,
+ Set<List<String>> remaining, Map<Group, Set<List<String>>>
reachableCache,
+ CascadesContext cascadesContext) {
+ remaining = new HashSet<>(remaining);
+
+ // Remove MVs that this group's own scan expressions cover
+ Set<List<String>> ownMvs = new HashSet<>();
+ for (GroupExpression ge : group.getPhysicalExpressions()) {
+ ownMvs.addAll(getScanMvIdentifiers(ge.getPlan(), remaining));
+ }
+ remaining.removeAll(ownMvs);
+
+ if (remaining.isEmpty()) {
+ return chooseBestPlan(group, properties, cascadesContext);
+ }
+
+ GroupExpression chosenGE = null;
+ double chosenCost = Double.MAX_VALUE;
+
+ // Check all physical GEs in this group
+ for (GroupExpression ge : group.getPhysicalExpressions()) {
+ if (!ge.getLowestCostTable().containsKey(properties)) {
+ continue;
+ }
+ double geCost = ge.getCostByProperties(properties);
+ if (geCost >= chosenCost) {
+ continue;
+ }
+ Set<List<String>> childReachable = new HashSet<>();
+ for (Group child : ge.children()) {
+ Set<List<String>> cr = reachableCache.get(child);
+ if (cr != null) {
+ childReachable.addAll(cr);
+ }
+ }
+ if (childReachable.containsAll(remaining)) {
+ chosenGE = ge;
+ chosenCost = geCost;
+ }
+ }
+
+ // Also check enforcers
+ for (GroupExpression enforcer : group.getEnforcers().values()) {
+ if (!enforcer.getLowestCostTable().containsKey(properties)) {
+ continue;
+ }
+ double eCost = enforcer.getCostByProperties(properties);
+ if (eCost >= chosenCost) {
+ continue;
+ }
+ Set<List<String>> childReachable = new HashSet<>();
+ for (Group child : enforcer.children()) {
+ Set<List<String>> cr = reachableCache.get(child);
+ if (cr != null) {
+ childReachable.addAll(cr);
+ }
+ }
+ if (childReachable.containsAll(remaining)) {
+ chosenGE = enforcer;
+ chosenCost = eCost;
+ }
+ }
+
+ if (chosenGE == null) {
+ // The required materialization can not be produced with the
required properties, for example
+ // the hinted materialization is not costed for the required
distribution, so fall back to the
+ // cost based plan of this subtree instead of failing the query.
+ markUseMvNotChosen(cascadesContext.getStatementContext(),
remaining);
+ return chooseBestPlan(group, properties, cascadesContext);
+ }
+
+ // Handle enforcer marking (same as chooseBestPlan)
+ if ((chosenGE.getPlan() instanceof PhysicalDistribute
+ && group.getEnforcerSpecs().containsKey(
+ ((PhysicalDistribute<?>)
chosenGE.getPlan()).getDistributionSpec()))
+ || group.getEnforcers().containsKey(chosenGE)) {
+ group.addChosenEnforcerId(chosenGE.getId().asInt());
+ group.addChosenEnforcerProperties(properties);
+ } else {
+ group.setChosenProperties(properties);
+ group.setChosenGroupExpressionId(chosenGE.getId().asInt());
+ }
+
+ // Distribute remaining among children and recurse
+ List<PhysicalProperties> inputProps =
chosenGE.getInputPropertiesList(properties);
+ List<Plan> planChildren = Lists.newArrayList();
+ for (int i = 0; i < chosenGE.arity(); i++) {
+ Group child = chosenGE.child(i);
+ Set<List<String>> childReachable =
reachableCache.getOrDefault(child, new HashSet<>());
+ Set<List<String>> childRemaining = new HashSet<>(remaining);
+ childRemaining.retainAll(childReachable);
+ planChildren.add(chooseBestPlanWithRequiredMaterializations(
+ child, inputProps.get(i), childRemaining, reachableCache,
cascadesContext));
Review Comment:
[P2] Rebuild enforcers using the hinted MV's available properties
Reusing a cheapest enforcer's input property can discard a valid single
hinted MV. Set `NOT_IN_RBO` and `enable_parallel_result_sink=false`; use base
`A(dt,k,v) DUPLICATE KEY(dt,k) HASH(k)` and refreshed `m AS SELECT k,dt,v FROM
A WHERE dt>=0 HASH(v)`. For `SELECT /*+ USE_MV(m) */ k,dt,v FROM A WHERE dt=1`,
choose a finite `filter_cost_factor` large enough that A's dt-prefix saving
keeps the base plan cheaper.
The ResultSink requires GATHER. Its shared gather enforcer retains the
cheapest input `NATURAL(k)` from the base plan; original MV expressions are
recorded under ANY/native `NATURAL(v)`, and higher-cost distribution
enforcement can be pruned by `CostAndEnforcerJob`. Reachability sees m through
the enforcer's self-group, but this recursion requests the retained
`NATURAL(k)`. The hinted Project fails the `containsKey(properties)` check, and
selection falls back to `Gather -> Filter -> Scan(A)`, reporting m unused.
`Gather -> Project -> Filter -> Scan(m)` would satisfy the same parent
contract. Select the hinted alternative on its available ANY/native property
and rebuild the exchange, rather than requiring it to reproduce the cheapest
base's storage hash.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -322,27 +338,47 @@ private Plan planWithoutLock(
LOG.info("{}\nMemo is null",
ConnectContext.get().getQueryIdentifier());
}
}
- Set<Integer> requiredGroupIds = cascadesContext.getConnectContext()
- .getSessionVariable().getRequiredGroupIds();
+
+ // Compute USE_MV hard constraint: USE_MV specified qualifiers that
rewrite-successfully
+ Set<List<String>> requiredUseMvIdentifiers =
computeRequiredUseMvIdentifiers(statementContext);
+ Map<Group, Set<List<String>>> mvReachableCache = new HashMap<>();
+ if (!requiredUseMvIdentifiers.isEmpty()) {
+ computeReachableMvIdentifiers(getRoot(), mvReachableCache,
requiredUseMvIdentifiers);
+ // Drop the required MVs which the rewrite can not produce, and
mark them as UnUsed,
+ // USE_MV is best-effort and should not fail the query.
+ Set<List<String>> rootReachable = mvReachableCache.get(getRoot());
+ requiredUseMvIdentifiers =
dropUnreachableUseMvIdentifiers(statementContext,
+ requiredUseMvIdentifiers,
+ rootReachable == null ? ImmutableSet.of() : rootReachable);
+ }
PhysicalPlan physicalPlan;
- if (!requiredGroupIds.isEmpty()) {
- Map<Group, Set<Integer>> reachableCache = new HashMap<>();
- computeReachableGroupIds(getRoot(), reachableCache);
- // Validate all required groups are reachable from root
- Set<Integer> rootReachable = reachableCache.get(getRoot());
- Set<Integer> unreachable = new HashSet<>(requiredGroupIds);
- unreachable.removeAll(rootReachable);
- if (!unreachable.isEmpty()) {
- throw new AnalysisException("Required group IDs not found in
memo: " + unreachable);
- }
- physicalPlan = chooseBestPlanWithRequiredGroups(
- getRoot(), requireProperties, requiredGroupIds,
reachableCache);
+ if (!requiredUseMvIdentifiers.isEmpty()) {
+ physicalPlan = chooseBestPlanConsideringRequiredUseMv(getRoot(),
requireProperties,
Review Comment:
[P3] Populate the plan cost on the USE_MV selection path
This branch bypasses `chooseNthPlan` and `chooseBestPlanWithRequiredGroups`,
the two instance methods that assign the planner's `cost` field. The new static
helpers never assign it, so `EXPLAIN OPTIMIZED PLAN` prints the default `cost =
0.0` for every reachable USE_MV selection. For example, with refreshed `m AS
SELECT k FROM t` and `NOT_IN_RBO`, `SELECT /*+ USE_MV(m) */ k+1 FROM t` selects
`Project(k+1) -> Scan(m)`; `visitPhysicalProject` charges a positive cost even
though the hinted scan costs zero. Preserve the selected plan's estimate,
including any synthesized enforcement, so the existing EXPLAIN cost remains
accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
