I just want to pitch in and say that I ran into the same problem with
running with 64GB executors. For example, some of the tasks take 5 minutes
to execute, out of which 4 minutes are spent in GC. I'll try out smaller
executors.
On Mon, Oct 6, 2014 at 6:35 PM, Otis Gospodnetic otis.gospodne...@gmail.com
wrote:
Hi,
The other option to consider is using G1 GC, which should behave better
with large heaps. But pointers are not compressed in heaps 32 GB in
size, so you may be better off staying under 32 GB.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr Elasticsearch Support * http://sematext.com/
On Mon, Oct 6, 2014 at 8:08 PM, Mingyu Kim m...@palantir.com wrote:
Ok, cool. This seems to be general issues in JVM with very large heaps. I
agree that the best workaround would be to keep the heap size below 32GB.
Thanks guys!
Mingyu
From: Arun Ahuja aahuj...@gmail.com
Date: Monday, October 6, 2014 at 7:50 AM
To: Andrew Ash and...@andrewash.com
Cc: Mingyu Kim m...@palantir.com, user@spark.apache.org
user@spark.apache.org, Dennis Lawler dlaw...@palantir.com
Subject: Re: Larger heap leads to perf degradation due to GC
We have used the strategy that you suggested, Andrew - using many workers
per machine and keeping the heaps small ( 20gb).
Using a large heap resulted in workers hanging or not responding (leading
to timeouts). The same dataset/job for us will fail (most often due to
akka disassociated or fetch failures errors) with 10 cores / 100 executors,
60 gb per executor while succceed with 1 core / 1000 executors / 6gb per
executor.
When the job does succceed with more cores per executor and larger heap
it is usually much slower than the smaller executors (the same 8-10 min job
taking 15-20 min to complete)
The unfortunate downside of this has been, we have had some large
broadcast variables which may not fit into memory (and unnecessarily
duplicated) when using the smaller executors.
Most of this is anecdotal but for the most part we have had more success
and consistency with more executors with smaller memory requirements.
On Sun, Oct 5, 2014 at 7:20 PM, Andrew Ash and...@andrewash.com wrote:
Hi Mingyu,
Maybe we should be limiting our heaps to 32GB max and running multiple
workers per machine to avoid large GC issues.
For a 128GB memory, 32 core machine, this could look like:
SPARK_WORKER_INSTANCES=4
SPARK_WORKER_MEMORY=32
SPARK_WORKER_CORES=8
Are people running with large (32GB+) executor heaps in production? I'd
be curious to hear if so.
Cheers!
Andrew
On Thu, Oct 2, 2014 at 1:30 PM, Mingyu Kim m...@palantir.com wrote:
This issue definitely needs more investigation, but I just wanted to
quickly check if anyone has run into this problem or has general guidance
around it. We’ve seen a performance degradation with a large heap on a
simple map task (I.e. No shuffle). We’ve seen the slowness starting around
from 50GB heap. (I.e. spark.executor.memoty=50g) And, when we checked the
CPU usage, there were just a lot of GCs going on.
Has anyone seen a similar problem?
Thanks,
Mingyu
