Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18931#discussion_r136741637
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -149,14 +149,146 @@ trait CodegenSupport extends SparkPlan {
ctx.freshNamePrefix = parent.variablePrefix
val evaluated = evaluateRequiredVariables(output, inputVars,
parent.usedInputs)
+
+ // Under certain conditions, we can put the logic to consume the rows
of this operator into
+ // another function. So we can prevent a generated function too long
to be optimized by JIT.
+ // The conditions:
+ // 1. The parent uses all variables in output. we can't defer variable
evaluation when consume
--- End diff --
E.g., a `ProjectExec` node doesn't necessarily evaluate all its output
variables before continuing `doConsume` of its parent node. It can defer the
evaluation until the variables are needed in the parent node's consume logic.
Once a variable's evaluation is deferred, and if we create a consume
function, the variable will be evaluated in the function. But now the
references of this variable is out of scope.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
