viirya commented on a change in pull request #29950:
URL: https://github.com/apache/spark/pull/29950#discussion_r509035110
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
##########
@@ -124,14 +126,34 @@ object ScanOperation extends OperationHelper with
PredicateHelper {
}.exists(!_.deterministic))
}
+ def moreThanMaxAllowedCommonOutput(
+ expr: Seq[NamedExpression],
+ aliases: AttributeMap[Expression]): Boolean = {
+ val exprMap = mutable.HashMap.empty[Attribute, Int]
+
+ expr.foreach(_.collect {
+ case a: Attribute if aliases.contains(a) => exprMap.update(a,
exprMap.getOrElse(a, 0) + 1)
+ })
+
+ val commonOutputs = if (exprMap.size > 0) {
+ exprMap.maxBy(_._2)._2
+ } else {
+ 0
+ }
+
+ commonOutputs > SQLConf.get.maxCommonExprsInCollapseProject
+ }
+
private def collectProjectsAndFilters(plan: LogicalPlan): ScanReturnType = {
plan match {
case Project(fields, child) =>
collectProjectsAndFilters(child) match {
case Some((_, filters, other, aliases)) =>
// Follow CollapseProject and only keep going if the collected
Projects
- // do not have common non-deterministic expressions.
- if (!hasCommonNonDeterministic(fields, aliases)) {
+ // do not have common non-deterministic expressions, and do not
have more than
+ // maximum allowed common outputs.
+ if (!hasCommonNonDeterministic(fields, aliases) &&
+ !moreThanMaxAllowedCommonOutput(fields, aliases)) {
Review comment:
I'm wondering how we decide to use `PhysicalOperation` or
`ScanOperation`?
I can see `ScanOperation` is used for re-construct Project/Filter around
scan node. `PhysicalOperation` has mixed use cases, including re-construct
Project/Filter around scan like `ScanOperation` (so can we replace
`PhysicalOperation` with `ScanOperation` in these cases?), and others e.g.
`OptimizeMetadataOnlyQuery`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
