Github user sddyljsx commented on the issue:
https://github.com/apache/spark/pull/21859
'The ShuffleWriter should treat RangePartitioner specially and consume the
sampled data in RangePartitioner instead of the input iterator.' This idea is
good, maybe we can cache both the K and V when doing sample.
I will have a try on this idea.--- --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
