This is a different kind of error. Thomas' OOM error was specific to the
kernel refusing to create another thread/process for his application.
Matthew
On Wed, Mar 25, 2015 at 10:51 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote:
I have a YARN cluster where the max memory allowed is 16GB. I set 12G for
my driver, however i see OutOFMemory error even for this program
http://spark.apache.org/docs/1.3.0/sql-programming-guide.html#hive-tables
. What do you suggest ?
On Wed, Mar 25, 2015 at 8:23 AM, Thomas Gerber thomas.ger...@radius.com
wrote:
So,
1. I reduced my -XX:ThreadStackSize to 5m (instead of 10m - default is
1m), which is still OK for my need.
2. I reduced the executor memory to 44GB for a 60GB machine (instead of
49GB).
This seems to have helped. Thanks to Matthew and Sean.
Thomas
On Tue, Mar 24, 2015 at 3:49 PM, Matt Silvey matt.sil...@videoamp.com
wrote:
My memory is hazy on this but aren't there hidden limitations to
Linux-based threads? I ran into some issues a couple of years ago where,
and here is the fuzzy part, the kernel wants to reserve virtual memory per
thread equal to the stack size. When the total amount of reserved memory
(not necessarily resident memory) exceeds the memory of the system it
throws an OOM. I'm looking for material to back this up. Sorry for the
initial vague response.
Matthew
On Tue, Mar 24, 2015 at 12:53 PM, Thomas Gerber
thomas.ger...@radius.com wrote:
Additional notes:
I did not find anything wrong with the number of threads (ps -u USER -L
| wc -l): around 780 on the master and 400 on executors. I am running on
100 r3.2xlarge.
On Tue, Mar 24, 2015 at 12:38 PM, Thomas Gerber
thomas.ger...@radius.com wrote:
Hello,
I am seeing various crashes in spark on large jobs which all share a
similar exception:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
I increased nproc (i.e. ulimit -u) 10 fold, but it doesn't help.
Does anyone know how to avoid those kinds of errors?
Noteworthy: I added -XX:ThreadStackSize=10m on both driver and
executor extra java options, which might have amplified the problem.
Thanks for you help,
Thomas
--
Deepak
