cloud-fan commented on code in PR #52557:
URL: https://github.com/apache/spark/pull/52557#discussion_r2443585700
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala:
##########
@@ -1655,3 +1725,25 @@ class
TaskCommitFailureParquetOutputCommitter(outputPath: Path, context: TaskAtt
sys.error("Intentional exception for testing purposes")
}
}
+
+case class VerifyNoAdditionalScanOutputExec(override val child: SparkPlan)
extends UnaryExecNode {
+ override def doExecute(): RDD[InternalRow] = {
+ val childOutputTypes = child.output.map(_.dataType)
+ child.executeColumnar().mapPartitionsInternal { batches =>
+ batches.flatMap { input =>
+ input.rowIterator().asScala
Review Comment:
if we decide to keep this custom physical plan, we can simplify it by
checking columnar batches directly:
```
0.until(input.numCols).foreach { index =>
assert(childOutputTypes(index).dataType == input.column(index).dataType)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
