ulysses-you commented on PR #57742:
URL: https://github.com/apache/spark/pull/57742#issuecomment-5236196521
thank you @peter-toth the thorough review. I think I addressed the main
comments. Here is the un-addressed comment, the Finding 2, for such case cause
5x shuffle rows:
```scala
spark.range(0, 500000, 1, 1)
.select(when($"id" < 100000, $"id").otherwise($"id" % 100) as "k", $"id"
as "v")
.groupBy($"k").agg(sum($"v"))
```
Due to the pass through flip is one-way and the first check fires at minRows
regardless of task size, so a distinct-heavy prefix can commit a large task.
IMO, the 100 thousand rows is not a small data size for patial-agg per task,
not sure if it can be happened in real world, but if it happens it will be a
regression. I have no better idea for that so leave it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
