i7xh commented on a change in pull request #1129:
URL: https://github.com/apache/incubator-kyuubi/pull/1129#discussion_r723936241
##########
File path:
dev/kyuubi-extension-spark-3-1/src/main/scala/org/apache/kyuubi/sql/watchdog/ForcedMaxOutputRowsRule.scala
##########
@@ -49,10 +56,19 @@ case class ForcedMaxOutputRowsRule(session: SparkSession)
extends Rule[LogicalPl
maxOutputRowsOpt match {
case Some(forcedMaxOutputRows) => val supported = p match {
- case _: Project => true
- case _: Aggregate => true
- case Limit(_, _) => true
- case _ => false
+ case agg: Aggregate => agg match {
+ case Aggregate(_, Alias(_, "havingCondition")::Nil, _) => false
Review comment:
> I see what you are trying to fix, but it seems too hack.
>
> Since this issue is from Spark and fix after 3.2.0, I prefer don't add
Limit for `Aggregate` now. what do you think @i7xh ?
It's able to solution the problem, and it rarely upgrade the spark version
special for the issue for us
The solution of the issue: the basic principle is that can't add limit node
to the non-top aggregate node, so I pre-process the non-top aggregate node via
mark tag value the aggregate node's children of `aggregateExpressions`, it
seems hacker but it can't mark the aggregate node self, because the nested call
will cover the pre-tagged value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
