sunchao commented on code in PR #56575:
URL: https://github.com/apache/spark/pull/56575#discussion_r3450352127
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -155,6 +155,12 @@ case class AdaptiveSparkPlanExec(
private def postStageCreationRules(outputsColumnar: Boolean) = Seq(
ApplyColumnarRulesAndInsertTransitions(
context.session.sessionState.columnarRules, outputsColumnar),
+ // Mirror `QueryExecution.preparations`: materialize any surviving
`RuntimeReplaceable` after
+ // columnar/native conversion and before codegen. AQE re-optimizes each
stage (which can
+ // re-insert a surviving `RuntimeReplaceable`, e.g. via
`OptimizeCsvJsonExprs`), and
+ // `AdaptiveSparkPlanExec` is a `LeafExecNode` that the outer
`preparations` rule can't reach,
+ // so the materialization must also run here.
+ MaterializeRuntimeReplaceable,
Review Comment:
[P2] This fixes AQE exchange and result stages, but `newQueryStage`'s
`InMemoryTableScanLike` branch still deliberately skips
`postStageCreationRules`, so this materialization rule never reaches
table-cache scans. `InMemoryScans` copies logical filters into
`InMemoryTableScanExec.predicates`; after the scan is wrapped in
`TableCacheQueryStageExec` (a leaf), later result-stage traversal cannot reach
that hidden plan. A surviving predicate wrapper whose replacement is supported
by `CachedBatchSerializer.buildFilter` (for example, `EqualTo(attr, literal)`)
is therefore left unrecognized, disabling cached-batch pruning even though the
separate `FilterExec` still produces correct rows.
Could we apply `MaterializeRuntimeReplaceable` explicitly to the optimized
`InMemoryTableScanLike` before wrapping it, and add an AQE cache/filter test
that inspects `TableCacheQueryStageExec.plan`? The current repartition test
covers only the result-stage path, and `finalPlan.exists` does not traverse
query-stage plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
