Afternoon all,

Really loving this project and the community behind it. Thank you all for
your hard work.


This past week, though, I've been having a hard time getting my first
deployed job to run without failing at the same point every time: Right
after a leftOuterJoin, most partitions (600 total) are small (1-100MB),
while some others are large (3-6GB). The large ones consistently spill
20-60GB into memory, and eventually fail.


If I could only get the partitions to be smaller, right out of the
leftOuterJoin, it seems like the job would run fine.


I've tried trawling through the logs, but it hasn't been very fruitful in
finding out what, specifically, is the issue.


Cluster setup:

* 6 worker nodes (16 cores, 104GB Memory, 500GB storage)

* 1 master (same config as above)


Running Spark on YARN, with:


storage.memoryFraction = .3

--executors = 6

--executor-cores = 12

--executor-memory = kind of confusing due to YARN, but basically in the
Spark monitor site's Executors page, it shows each as running with 18.8GB
memory, though I know usage is much larger due to YARN managing various
pieces. (Total memory available to yarn shows 480GB, with 270GB currently
used).


Screenshot of the task page: http://i.imgur.com/xG3KdEl.png

Code:
https://gist.github.com/momer/8bc03c60a639e5c04eda#file-spark-scala-L60 (see
line 60 for the relevant area)



Any pointers in the right direction, or advice on articles to read, or even
debugging / settings advice or recommendations would be extremely helpful.
I'll put a bounty on this of $50 donation to the ASF! :D


Thank you all for reading (and hopefully replying!),


Mo Omer

Reply via email to