Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19656#discussion_r148933391
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -213,19 +213,32 @@ trait CodegenSupport extends SparkPlan {
}
/**
- * For optimization to suppress shouldStop() in a loop of
WholeStageCodegen.
- * Returning true means we need to insert shouldStop() into the loop
producing rows, if any.
+ * Whether or not the result rows of this operator should be copied
before putting into a buffer.
+ *
+ * If any operator inside WholeStageCodegen generate multiple rows from
a single row (for
+ * example, Join), this should be true.
+ *
+ * If an operator starts a new pipeline, this should be false.
*/
- def isShouldStopRequired: Boolean = {
- return shouldStopRequired && (this.parent == null ||
this.parent.isShouldStopRequired)
+ def needCopyResult: Boolean = {
+ if (children.isEmpty) {
+ false
+ } else if (children.length == 1) {
+ children.head.asInstanceOf[CodegenSupport].needStopCheck
--- End diff --
`needCopyResult` here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
