wecharyu commented on code in PR #12703:
URL: https://github.com/apache/gluten/pull/12703#discussion_r3869824808
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala:
##########
@@ -145,8 +147,13 @@ case class
GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark:
// case 2: check whether fallback exists and decide whether increase heap
memory
// and decrease offheap memory.
- val fallenNodeCnt = planNodes.count(p => !p.isInstanceOf[GlutenPlan])
- val totalCount = planNodes.size
+ val countedPlanNodes =
planNodes.filterNot(GlutenExplainUtils.isFallbackInsensitivePlan)
+ val fallenNodeCnt = countedPlanNodes.count {
+ case _: GlutenPlan => false
+ case i: InMemoryTableScanExec => !PlanUtil.isGlutenTableCache(i)
+ case _ => true
+ }
+ val totalCount = countedPlanNodes.size
if (1.0 * fallenNodeCnt / totalCount >=
glutenConf.autoAdjustStageFallenNodeThreshold) {
Review Comment:
`totalCount == 0` means `fallenNode == 0`, then the fallenNodeCnt /
totalCount is `NaN` rather than `Infinity`, and `NaN` compare is always false
so no auto-adjustment is triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
