Yes, just reran with the following (spark_build)root@ip-10-45-130-206:~/spark# export MAVEN_OPTS="-Xmx4096mb > -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m" > (spark_build)root@ip-10-45-130-206:~/spark# build/mvn -Pyarn -Phadoop-2.4 > -Dhadoop.version=2.4.0 -DskipTests clean package
and grepping for java root 641 9.9 0.3 4411732 49040 pts/4 Sl+ 17:35 0:01 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server -Xmx2g > -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m > -Dzinc.home=/root/spark/build/zinc-0.3.5.3 -classpath > /root/spark/build/zinc-0.3.5.3/lib/compiler-interface-sources.jar:/root/spark/build/zinc-0.3.5.3/lib/incremental-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/nailgun-server.jar:/root/spark/build/zinc-0.3.5.3/lib/sbt-interface.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-library.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-reflect.jar:/root/spark/build/zinc-0.3.5.3/lib/zinc.jar > com.typesafe.zinc.Nailgun 3030 0 > root 687 226 2.0 1803664 312876 pts/4 Sl+ 17:36 0:22 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms256m -Xmx512m -classpath > /opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar > -Dclassworlds.conf=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/bin/m2.conf > -Dmaven.home=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3 > -Dmaven.multiModuleProjectDirectory=/root/spark > org.codehaus.plexus.classworlds.launcher.Launcher -DzincPort=3030 -Pyarn > -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package On Tue, Sep 8, 2015 at 1:14 PM, Sean Owen <so...@cloudera.com> wrote: > MAVEN_OPTS shouldn't affect zinc as it's an unrelated application. You > can run "zinc -J-Xmx4g..." in general, but in the provided script, > ZINC_OPTS seems to be the equivalent, yes. It kind of looks like your > mvn process isn't getting any special memory args there. Is MAVEN_OPTS > really exported? > > FWIW I use my own local mvn and zinc and it works fine. > > On Tue, Sep 8, 2015 at 6:05 PM, Benjamin Zaitlen <quasi...@gmail.com> > wrote: > > I'm running zinv while compiling. It seems that MAVEN_OPTS doesn't > really > > change much? Or perhaps I'm misunderstanding something -- grepping for > java > > i see > > > >> root 24355 102 8.8 4687376 1350724 pts/4 Sl 16:51 11:08 > >> /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server -Xmx2g > >> -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m > >> -Dzinc.home=/root/spark/build/zinc-0.3.5.3 -classpath > >> > /root/spark/build/zinc-0.3.5.3/lib/compiler-interface-sources.jar:/root/spark/build/zinc-0.3.5.3/lib/incremental-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/nailgun-server.jar:/root/spark/build/zinc-0.3.5.3/lib/sbt-interface.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-library.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-reflect.jar:/root/spark/build/zinc-0.3.5.3/lib/zinc.jar > >> com.typesafe.zinc.Nailgun 3030 0 > >> root 25151 22.0 3.2 2269092 495276 pts/4 Sl+ 16:53 1:56 > >> /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms256m -Xmx512m -classpath > >> > /opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar > >> > -Dclassworlds.conf=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/bin/m2.conf > >> -Dmaven.home=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3 > >> -Dmaven.multiModuleProjectDirectory=/root/spark > >> org.codehaus.plexus.classworlds.launcher.Launcher -DzincPort=3030 clean > >> package -DskipTests -Pyarn -Phive -Phive-thriftserver -Phadoop-2.4 > >> -Dhadoop.version=2.4.0 > > > > > > So the heap size is still 2g even with MAVEN_OPTS set with 4g. I noticed > > that within build/mvn _COMPILE_JVM_OPTS is set to 2g and this is what > > ZINC_OPTS is set to. > > > > --Ben > > > > > > On Tue, Sep 8, 2015 at 11:06 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> > >> Do you run Zinc while compiling ? > >> > >> Cheers > >> > >> On Tue, Sep 8, 2015 at 7:56 AM, Benjamin Zaitlen <quasi...@gmail.com> > >> wrote: > >>> > >>> I'm still getting errors with 3g. I've increase to 4g and I'll report > >>> back > >>> > >>> To be clear: > >>> > >>> export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M > >>> -XX:ReservedCodeCacheSize=1024m" > >>> > >>>> [ERROR] GC overhead limit exceeded -> [Help 1] > >>>> [ERROR] > >>>> [ERROR] To see the full stack trace of the errors, re-run Maven with > the > >>>> -e switch. > >>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging. > >>>> [ERROR] > >>>> [ERROR] For more information about the errors and possible solutions, > >>>> please read the following articles: > >>>> [ERROR] [Help 1] > >>>> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError > >>>> + return 1 > >>>> + exit 1 > >>> > >>> > >>> On Tue, Sep 8, 2015 at 10:03 AM, Sean Owen <so...@cloudera.com> wrote: > >>>> > >>>> It might need more memory in certain situations / running certain > >>>> tests. If 3gb works for your relatively full build, yes you can open a > >>>> PR to change any occurrences of lower recommendations to 3gb. > >>>> > >>>> On Tue, Sep 8, 2015 at 3:02 PM, Benjamin Zaitlen <quasi...@gmail.com> > >>>> wrote: > >>>> > Ah, right. Should've caught that. > >>>> > > >>>> > The docs seem to recommend 2gb. Should that be increased as well? > >>>> > > >>>> > --Ben > >>>> > > >>>> > On Tue, Sep 8, 2015 at 9:33 AM, Sean Owen <so...@cloudera.com> > wrote: > >>>> >> > >>>> >> It shows you there that Maven is out of memory. Give it more heap. > I > >>>> >> use > >>>> >> 3gb. > >>>> >> > >>>> >> On Tue, Sep 8, 2015 at 1:53 PM, Benjamin Zaitlen < > quasi...@gmail.com> > >>>> >> wrote: > >>>> >> > Hi All, > >>>> >> > > >>>> >> > I'm trying to build a distribution off of the latest in master > and > >>>> >> > I > >>>> >> > keep > >>>> >> > getting errors on MQTT and the build fails. I'm running the > build > >>>> >> > on a > >>>> >> > m1.large which has 7.5 GB of RAM and no other major processes are > >>>> >> > running. > >>>> >> > > >>>> >> >> MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M > >>>> >> >> -XX:ReservedCodeCacheSize=512m" > >>>> >> >> ./make-distribution.sh --name continuum-custom-spark-1.5 --tgz > >>>> >> >> -Pyarn > >>>> >> >> -Phive -Phive-thriftserver -Phadoop-2.4 -Dhadoop.version=2.4.0 > >>>> >> > > >>>> >> > > >>>> >> > > >>>> >> >> INFO] Spark Project GraphX ............................... > SUCCESS > >>>> >> >> [ > >>>> >> >> 33.345 s] > >>>> >> >> [INFO] Spark Project Streaming ............................ > >>>> >> >> SUCCESS > >>>> >> >> [01:08 > >>>> >> >> min] > >>>> >> >> [INFO] Spark Project Catalyst ............................. > >>>> >> >> SUCCESS > >>>> >> >> [01:39 > >>>> >> >> min] > >>>> >> >> [INFO] Spark Project SQL .................................. > >>>> >> >> SUCCESS > >>>> >> >> [02:06 > >>>> >> >> min] > >>>> >> >> [INFO] Spark Project ML Library ........................... > >>>> >> >> SUCCESS > >>>> >> >> [02:16 > >>>> >> >> min] > >>>> >> >> [INFO] Spark Project Tools ................................ > >>>> >> >> SUCCESS [ > >>>> >> >> 4.087 s] > >>>> >> >> [INFO] Spark Project Hive ................................. > >>>> >> >> SUCCESS > >>>> >> >> [01:28 > >>>> >> >> min] > >>>> >> >> [INFO] Spark Project REPL ................................. > >>>> >> >> SUCCESS [ > >>>> >> >> 16.291 s] > >>>> >> >> [INFO] Spark Project YARN Shuffle Service ................. > >>>> >> >> SUCCESS [ > >>>> >> >> 13.671 s] > >>>> >> >> [INFO] Spark Project YARN ................................. > >>>> >> >> SUCCESS [ > >>>> >> >> 20.554 s] > >>>> >> >> [INFO] Spark Project Hive Thrift Server ................... > >>>> >> >> SUCCESS [ > >>>> >> >> 14.332 s] > >>>> >> >> [INFO] Spark Project Assembly ............................. > >>>> >> >> SUCCESS > >>>> >> >> [03:33 > >>>> >> >> min] > >>>> >> >> [INFO] Spark Project External Twitter ..................... > >>>> >> >> SUCCESS [ > >>>> >> >> 14.208 s] > >>>> >> >> [INFO] Spark Project External Flume Sink .................. > >>>> >> >> SUCCESS [ > >>>> >> >> 11.535 s] > >>>> >> >> [INFO] Spark Project External Flume ....................... > >>>> >> >> SUCCESS [ > >>>> >> >> 19.010 s] > >>>> >> >> [INFO] Spark Project External Flume Assembly .............. > >>>> >> >> SUCCESS [ > >>>> >> >> 5.210 s] > >>>> >> >> [INFO] Spark Project External MQTT ........................ > >>>> >> >> FAILURE > >>>> >> >> [01:10 > >>>> >> >> min] > >>>> >> >> [INFO] Spark Project External MQTT Assembly ............... > >>>> >> >> SKIPPED > >>>> >> >> [INFO] Spark Project External ZeroMQ ...................... > >>>> >> >> SKIPPED > >>>> >> >> [INFO] Spark Project External Kafka ....................... > >>>> >> >> SKIPPED > >>>> >> >> [INFO] Spark Project Examples ............................. > >>>> >> >> SKIPPED > >>>> >> >> [INFO] Spark Project External Kafka Assembly .............. > >>>> >> >> SKIPPED > >>>> >> >> [INFO] > >>>> >> >> > >>>> >> >> > >>>> >> >> > ------------------------------------------------------------------------ > >>>> >> >> [INFO] BUILD FAILURE > >>>> >> >> [INFO] > >>>> >> >> > >>>> >> >> > >>>> >> >> > ------------------------------------------------------------------------ > >>>> >> >> [INFO] Total time: 22:55 min > >>>> >> >> [INFO] Finished at: 2015-09-07T22:42:57+00:00 > >>>> >> >> [INFO] Final Memory: 240M/455M > >>>> >> >> [INFO] > >>>> >> >> > >>>> >> >> > >>>> >> >> > ------------------------------------------------------------------------ > >>>> >> >> [ERROR] GC overhead limit exceeded -> [Help 1] > >>>> >> >> [ERROR] > >>>> >> >> [ERROR] To see the full stack trace of the errors, re-run Maven > >>>> >> >> with > >>>> >> >> the > >>>> >> >> -e switch. > >>>> >> >> [ERROR] Re-run Maven using the -X switch to enable full debug > >>>> >> >> logging. > >>>> >> >> [ERROR] > >>>> >> >> [ERROR] For more information about the errors and possible > >>>> >> >> solutions, > >>>> >> >> please read the following articles: > >>>> >> >> [ERROR] [Help 1] > >>>> >> >> > http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError > >>>> >> >> + return 1 > >>>> >> >> + exit 1 > >>>> >> > > >>>> >> > > >>>> >> > Any thoughts would be extremely helpful. > >>>> >> > > >>>> >> > --Ben > >>>> > > >>>> > > >>> > >>> > >> > > >
