maropu commented on a change in pull request #29950:
URL: https://github.com/apache/spark/pull/29950#discussion_r502206637
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CollapseProjectSuite.scala
##########
@@ -170,4 +171,34 @@ class CollapseProjectSuite extends PlanTest {
val expected = Sample(0.0, 0.6, false, 11L, relation.select('a as
'c)).analyze
comparePlans(optimized, expected)
}
+
+ test("SPARK-32945: avoid collapsing projects if reaching max allowed common
exprs") {
+ val options = Map.empty[String, String]
+ val schema = StructType.fromDDL("a int, b int, c string, d long")
+
+ Seq("1", "2", "3", "4").foreach { maxCommonExprs =>
+ withSQLConf(SQLConf.MAX_COMMON_EXPRS_IN_COLLAPSE_PROJECT.key ->
maxCommonExprs) {
+ // If we collapse two Projects, `JsonToStructs` will be repeated three
times.
+ val relation = LocalRelation('json.string)
+ val query = relation.select(
+ JsonToStructs(schema, options, 'json).as("struct"))
+ .select(
+ GetStructField('struct, 0).as("a"),
+ GetStructField('struct, 1).as("b"),
+ GetStructField('struct, 2).as("c")).analyze
Review comment:
hm, it sounds fine, too. Rather, it seems a top-down transformation can
collapse projects in one shot just like `RemoveRedundantProjects`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
