Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19440#discussion_r143091744
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
---
@@ -185,4 +185,22 @@ class WholeStageCodegenSuite extends SparkPlanTest
with SharedSQLContext {
val (_, maxCodeSize2) = CodeGenerator.compile(codeWithLongFunctions)
assert(maxCodeSize2 >
SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.defaultValue.get)
}
+
+ test("returning batch for wide table") {
+ import testImplicits._
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ val df = spark.range(10).select(Seq.tabulate(201) {i => ('id +
i).as(s"c$i")} : _*)
+ df.write.mode(SaveMode.Overwrite).parquet(path)
+
+ withSQLConf(SQLConf.WHOLESTAGE_MAX_NUM_FIELDS.key -> "202",
+ SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key -> "8000") {
+ // donot return batch, because whole stage codegen is disabled for
wide table (>202 columns)
--- End diff --
Is this comment wrong or I misunderstand it? Looks like it returns batch as
it asserts `supportsBatch`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
