Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
I think this works, can we post some Spark web UI screenshots to confirm
the shuffle is indeed eliminated?
BTW one idea to simplify the implementation:
```
def distinct(numPartitions) = {
...
mapPartitions(iter => {
val map = new ExternalAppendOnlyMap[T, Null, Null](
createCombiner = identity,
mergeValue = (a, b) => a,
mergeCombiners = (a, b) => a)
map.insertAll(iter.map(_ -> null))
map.iterator.map(_._1)
}, preservesPartitioning = true)
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
