Re: Executor Lost with StorageLevel.MEMORY_AND_DISK_SER

2015-02-10 Thread Marius Soutier
-Original Message- From: Marius Soutier [mailto:mps@gmail.com] Sent: Monday, February 09, 2015 2:19 AM To: user Subject: Executor Lost with StorageLevel.MEMORY_AND_DISK_SER Hi there, I'm trying to improve performance on a job that has GC troubles and takes longer

Executor Lost with StorageLevel.MEMORY_AND_DISK_SER

2015-02-09 Thread Marius Soutier
Hi there, I’m trying to improve performance on a job that has GC troubles and takes longer to compute simply because it has to recompute failed tasks. After deferring object creation as much as possible, I’m now trying to improve memory usage with StorageLevel.MEMORY_AND_DISK_SER and a custom