Re: Troubleshooting JVM OOM during Spark Unit Tests

2014-11-22 Thread Reynold Xin
What does /tmp/jvm-21940/hs_error.log tell you? It might give hints to what threads are allocating the extra off-heap memory. On Fri, Nov 21, 2014 at 1:50 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Howdy folks, I’m trying to understand why I’m getting “insufficient memory”

Re: Troubleshooting JVM OOM during Spark Unit Tests

2014-11-22 Thread Nicholas Chammas
Here’s that log file https://gist.github.com/nchammas/08d3a3a02486cf602ceb from a different run of the unit tests that also failed. I’m not sure what to look for. If it matters any, I also changed JAVA_OPTS as follows for this run: export JAVA_OPTS=-Xms512m -Xmx1024m -XX:PermSize=64m

Troubleshooting JVM OOM during Spark Unit Tests

2014-11-21 Thread Nicholas Chammas
Howdy folks, I’m trying to understand why I’m getting “insufficient memory” errors when trying to run Spark Units tests within a CentOS Docker container. I’m building Spark and running the tests as follows: # build sbt/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive