MaxGekk commented on code in PR #59225:
URL: https://github.com/apache/spark/pull/59225#discussion_r4203636838
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:
##########
@@ -1188,6 +1485,215 @@ class CodegenContext extends Logging {
}
}
+ /**
+ * What whole stage codegen does with an expression it could split, which
+ * `WholeStageCodegenExec.doCodeGen` sets for each generation of a stage.
Read here rather than
+ * the conf, so that every split of a generation follows the decision
`doCodeGen` made from the
+ * stage's own conf.
+ */
+ private[sql] var wholeStageSplit: WholeStageSplit = WholeStageSplit.Off
+
+ /**
+ * Under `WholeStageSplit.Record`, the expressions
[[splitExpressionsWithSources]] might have
+ * split, by the index their code is marked with.
+ */
+ private[sql] val wholeStageSplitsRecorded =
mutable.ArrayBuffer.empty[WholeStageSplitKey]
+
+ /**
+ * Under `WholeStageSplit.Record`, a digest of the stage's source with the
marks of the recorded
+ * expressions, which `WholeStageCodegenExec.generate` sets before it strips
them: the marks say
+ * which expression each index in [[wholeStageSplitsRecorded]] is, so a
decision made for these
+ * indices holds for this digest and no other.
+ */
+ private[sql] var wholeStageSplitDigest: String = ""
+
+ /**
+ * How many expressions with each sources [[splitExpressionsWithSources]]
has met in this
+ * generation, for [[WholeStageSplitKey.occurrence]].
+ */
+ private val wholeStageSplitOccurrences =
mutable.HashMap.empty[Seq[Seq[Expression]], Int]
+
+ /**
+ * Under `WholeStageSplit.Split`, how many blocks
[[splitExpressionsWithSources]] has turned into
+ * methods in this generation: none, and the code is the code in one piece,
whatever the split
Review Comment:
All four reworded in 1edcb55e4b7:
- `wholeStageBlocksSplit`: with no block split, the code is the code in one
piece but for the helpers of a discarded subexpression pass, which only a split
generation removes.
- `splitsSomething`: the code in one piece is kept as it is when no split is
compiled. The description now says its class is cached for `doExecute` only
where it compiled; where it failed, `doExecute` compiles it again.
- `logHugeMethod` says its caller holds the report in a trial. The `trial`
doc lists the metric updates and huge-method reports as held, and says an
expected failure is logged at DEBUG.
- The `reportOrHold` doc, the `trial` doc and the description are narrowed
to the metrics and the huge-method reports. They also say the
`spark.sql.codegen.logLevel` dump is made for every compile, a trial's
included. I kept the dump unheld, as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
