cloud-fan commented on code in PR #56977:
URL: https://github.com/apache/spark/pull/56977#discussion_r3524909079
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala:
##########
@@ -210,6 +213,33 @@ case class ColumnarToRowExec(child: SparkPlan)
copy(child = newChild)
}
+object ColumnarToRowExec {
+ /**
+ * Releases the current batch (if any) and fetches the next one from the
input iterator,
+ * bumping the batch/row metrics. Returns null when the input is exhausted.
This is called by
+ * the generated code of [[ColumnarToRowExec]] (through the static
forwarder), so the
+ * type-independent per-batch bookkeeping is compiled once per JVM instead
of being re-emitted
+ * into every stage's generated `nextBatch` method.
+ */
+ def advanceBatch(
Review Comment:
Non-blocking: now that `advanceBatch` is extracted as a pure static method,
it is directly unit-testable — a focused test could cheaply lock its
release/metrics contract (null `current` -> no close; non-null ->
`closeIfFreeable`; exhausted iterator -> returns null with no metric bump;
non-exhausted -> `numInputBatches`+1 / `numOutputRows`+numRows). Optional:
`WholeStageCodegenSuite` / `ParquetV1QuerySuite` / `ParquetV2QuerySuite`
already cover every reachable path, so this is a robustness nicety rather than
a coverage gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
