Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24657 )
Change subject: IMPALA-13534: Implement runtime filters on CTEs ...................................................................... Patch Set 8: (2 comments) http://gerrit.cloudera.org:8080/#/c/24657/8/be/src/exec/cte-consumer-node.cc File be/src/exec/cte-consumer-node.cc: http://gerrit.cloudera.org:8080/#/c/24657/8/be/src/exec/cte-consumer-node.cc@249 PS8, Line 249: if (!filter_ctxs_.empty()) { : if (!filters_waited_) { : filters_waited_ = true; : WaitForRuntimeFilters(state, filter_ctxs_); : } : FilterRowBatch(output_batch); : } One other difference vs the scanners is that HdfsScanner implements CheckFiltersEffectiveness() to disable less useful runtime filters. I think there are some queries where that would make a difference. http://gerrit.cloudera.org:8080/#/c/24657/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/24657/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@1457 PS8, Line 1457: /** : * Assigns pending runtime filters to a CTE consumer node. Filters targeting the CTE : * consumer's output tuple are assigned to 'cteNode'. This is correct because the CTE : * producer materializes data before the hash join build executes, so filters must be : * applied at the consumer level when reading from the LocalExchanger. Assignment to : * scan nodes inside the producer is prevented by the blocking mechanism in : * generateFiltersRecursive(). : */ : private void assignRuntimeFiltersToCTEConsumer(PlannerContext ctx, : CTEConsumerNode cteNode) { : Preconditions.checkState(cteNode.getTupleIds().size() == 1); : TupleId tid = cteNode.getTupleIds().get(0); : if (!runtimeFiltersByTid_.containsKey(tid)) return; : Analyzer analyzer = ctx.getRootAnalyzer(); : boolean disableRowRuntimeFiltering = : ctx.getQueryOptions().isDisable_row_runtime_filtering(); : TRuntimeFilterMode runtimeFilterMode = ctx.getQueryOptions().getRuntime_filter_mode(); : : for (RuntimeFilter filter : runtimeFiltersByTid_.get(tid)) { : if (filter.isFinalized()) continue; : if (cteProducerBlockedFilters_.containsKey(filter.getFilterId())) continue; : // CTE consumer filters are never partition-bound, so skip in row-filtering mode. : if (disableRowRuntimeFiltering) continue; : Expr targetExpr = computeTargetExpr(filter, tid, analyzer); : if (targetExpr == null) continue; : boolean isLocalTarget = isLocalTargetForPlanNode(filter, cteNode); : if (runtimeFilterMode == TRuntimeFilterMode.LOCAL && !isLocalTarget) continue; : RuntimeFilter.RuntimeFilterTarget target = : new RuntimeFilter.RuntimeFilterTarget(cteNode, targetExpr, isLocalTarget); : filter.addTarget(target); : } : } One thought here: I was looking at some profiles, and in the non-CTE plans, the min_max filters are before the bloom filters. CTE plans don't seem to follow that. Sometimes min_max filters eliminate whole files (suggesting they could be selective). We can't eliminate whole files, but min_max filters would be faster to evaluate than bloom filters. We could change our code to make sure that the min_max filters come first. -- To view, visit http://gerrit.cloudera.org:8080/24657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic877fb590187826f828da6a27bf274465c381e8e Gerrit-Change-Number: 24657 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Aleksandr Efimov <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Thu, 13 Aug 2026 21:38:53 +0000 Gerrit-HasComments: Yes
