Hi Bijay,
The Shuffle Spill (Disk) is the total number of bytes written to disk by
records spilled during the shuffle. The Shuffle Spill (Memory) is the
amount of space the spilled records occupied in memory before they were
spilled. These differ because the serialized format is more compact, and
the on-disk version can be compressed as well.
-Sandy
On Mon, Mar 23, 2015 at 5:29 PM, Bijay Pathak bijay.pat...@cloudwick.com
wrote:
Hello,
I am running TeraSort https://github.com/ehiggs/spark-terasort on
100GB of data. The final metrics I am getting on Shuffle Spill are:
Shuffle Spill(Memory): 122.5 GB
Shuffle Spill(Disk): 3.4 GB
What's the difference and relation between these two metrics? Does these
mean 122.5 GB was spill from memory during the shuffle?
thank you,
bijay
