felipepessoto commented on code in PR #13510:
URL: https://github.com/apache/gluten/pull/13510#discussion_r4606302517
##########
backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsTracker.scala:
##########
@@ -115,12 +137,46 @@ object GlutenDeltaJobStatsTracker extends Logging {
new GlutenDeltaJobStatsFallbackTracker(tracker)
}
+ private def createStatsPlan(
+ dataCols: Seq[Attribute],
+ aggregates: Seq[AggregateExpression]): SparkPlan = {
+ val statsAttrs =
aggregates.flatMap(_.aggregateFunction.aggBufferAttributes)
+ val statsResultAttrs =
aggregates.flatMap(_.aggregateFunction.inputAggBufferAttributes)
+ val aggOp = SortAggregateExec(
+ None,
+ isStreaming = false,
+ None,
+ Seq.empty,
+ aggregates,
+ statsAttrs,
+ 0,
+ statsResultAttrs,
+ StatisticsInputNode(dataCols)
+ )
+ val projOp = ProjectExec(statsResultAttrs, aggOp)
+ val offloads = Seq(OffloadOthers()).map(_.toStrcitRule())
+ val validatorBuilder: GlutenConfig => Validator = conf =>
+ Validators.newValidator(conf, offloads)
+ val config = GlutenConfig.get
+ val transformRule =
+ HeuristicTransform.WithRewrites(validatorBuilder(config),
Seq(PullOutPreProject), offloads)
+ ColumnarCollapseTransformStages(config)(transformRule(projOp))
Review Comment:
You’re right to question whether we still need this. The original motivation
was the NTZ statistics-support gap, which #12967 addressed. I haven’t
identified another unsupported statistics pattern under default settings, so
I’m not sure we should proceed with this PR in its current form.
The remaining reproductions are configuration-driven. For example, disabling
native aggregation while keeping native writing enabled makes the statistics
planner leave a Spark aggregate, but the tracker still assumes a fully native
plan. That causes a column-count mismatch; disabling projection instead causes
the unchecked `ProjectExec` cast to fail. Explicitly enabling NTZ validation
also reproduces rejection in the direct statistics test.
This PR checks the complete stage—not just its root—and uses Spark
statistics when it cannot fully offload, while retaining native file writing.
It therefore handles those rejected plans safely rather than adding missing
native support.
Before proceeding, could we clarify whether ordinary-query offload switches
should also govern the writer’s internal statistics? If yes, this fixes crashes
for that configuration. If not, statistics-specific planning rules would
address those cases more directly, and we could defer the general fallback
change. I’ve updated #12538 to distinguish the historical motivation from this
remaining question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]