Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
While this saves a shuffle, but the algorithm becomes different. Previously
we use the shuffe aggregator, which stores data in a `ExternalAppendOnlyMap`.
Now we use a scala set, which may OOM.--- --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
