Hi all,
In my occasion, I have a huge HashMap[(Int, Long), (Double, Double,
Double)], say several GB to tens of GB, after each iteration, I need to
collect() this HashMap and perform some calculation, and then broadcast()
it to every node. Now I have 20GB for each executor and after it
Hi Will,
When you call collect() the item you are collecting needs to fit in
memory on the driver. Is it possible your driver program does not have
enough memory?
- Patrick
On Wed, Dec 24, 2014 at 9:34 PM, Will Yang era.ye...@gmail.com wrote:
Hi all,
In my occasion, I have a huge
