Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19656#discussion_r148917096
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -213,19 +213,32 @@ trait CodegenSupport extends SparkPlan {
}
/**
- * For optimization to suppress shouldStop() in a loop of
WholeStageCodegen.
- * Returning true means we need to insert shouldStop() into the loop
producing rows, if any.
+ * Whether or not the result rows of this operator should be copied
before putting into a buffer.
+ *
+ * If any operator inside WholeStageCodegen generate multiple rows from
a single row (for
+ * example, Join), this should be true.
+ *
+ * If an operator starts a new pipeline, this should be false.
*/
- def isShouldStopRequired: Boolean = {
- return shouldStopRequired && (this.parent == null ||
this.parent.isShouldStopRequired)
+ protected def needCopyResult: Boolean = {
+ if (children.isEmpty) {
+ false
+ } else if (children.length == 1) {
+ children.head.asInstanceOf[CodegenSupport].needStopCheck
+ } else {
+ throw new UnsupportedOperationException
+ }
}
/**
- * Set to false if this plan consumes all rows produced by children but
doesn't output row
- * to buffer by calling append(), so the children don't require
shouldStop()
- * in the loop of producing rows.
+ * Whether or not the children of this operator should generate a stop
check when consuming input
+ * rows. This is used to suppress shouldStop() in a loop of
WholeStageCodegen.
+ *
+ * This should be false if an operator starts a new pipeline, which
means it consumes all rows
+ * produced by children but doesn't output row to buffer by calling
append(), so the children
+ * don't require shouldStop() in the loop of producing rows.
*/
- protected def shouldStopRequired: Boolean = true
+ protected def needStopCheck: Boolean = parent.needStopCheck
--- End diff --
it's unrelated but a simple clean up: we only need one method instead of 2.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
