cloud-fan commented on issue #26441: [SPARK-29682][SQL] Resolve conflicting references in aggregate expressions URL: https://github.com/apache/spark/pull/26441#issuecomment-552736120 @imback82 thanks for updating the PR description! I see the problem now. The `nums#38` attribute is conflicting and it comes from `Expand`. Ideally we should let `Expand` to re-generate its output attributes, but the `Expand` doesn't clearly distinguish the output attributes from its child and the output attributes of itself. I think we should change `Expand` to follow Generate ``` case class Expand(..., additionalOutput: Seq[Attribute]) { override def producedAttributes: AttributeSet = AttributeSet(additionalOutput) def output = child.output ++ additionalOutput } ``` And also follow how we dedup Generate: ``` case oldVersion: Generate if oldVersion.producedAttributes.intersect(conflictingAttributes).nonEmpty => val newOutput = oldVersion.generatorOutput.map(_.newInstance()) (oldVersion, oldVersion.copy(generatorOutput = newOutput)) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
