viirya commented on a change in pull request #29950:
URL: https://github.com/apache/spark/pull/29950#discussion_r502211597
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CollapseProjectSuite.scala
##########
@@ -170,4 +171,34 @@ class CollapseProjectSuite extends PlanTest {
val expected = Sample(0.0, 0.6, false, 11L, relation.select('a as
'c)).analyze
comparePlans(optimized, expected)
}
+
+ test("SPARK-32945: avoid collapsing projects if reaching max allowed common
exprs") {
+ val options = Map.empty[String, String]
+ val schema = StructType.fromDDL("a int, b int, c string, d long")
+
+ Seq("1", "2", "3", "4").foreach { maxCommonExprs =>
+ withSQLConf(SQLConf.MAX_COMMON_EXPRS_IN_COLLAPSE_PROJECT.key ->
maxCommonExprs) {
+ // If we collapse two Projects, `JsonToStructs` will be repeated three
times.
+ val relation = LocalRelation('json.string)
+ val query = relation.select(
+ JsonToStructs(schema, options, 'json).as("struct"))
+ .select(
+ GetStructField('struct, 0).as("a"),
+ GetStructField('struct, 1).as("b"),
+ GetStructField('struct, 2).as("c")).analyze
Review comment:
Seems like we need to change to `transformDown` and take a recursive
approach like `RemoveRedundantProjects` and `recursiveRemoveSort` for
collapsing `Project`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
