bottom line increase executor's non-mem-block memory and reduce indivdiual starting task size until it all fits.
On Tue, Feb 16, 2016 at 4:09 PM, Dmitriy Lyubimov <dlie...@gmail.com> wrote: > the original exception definitely happens in the task when mahout tries to > build an entire matrix block out of a partition. Use more tasks, smaller in > size initially. using par(min=??) will help to repartition to at least ?? > tasks. off-hdfs defaults are just too big for matrix processing. Not sure > how to do that with command line utility, Pat may help. > > On Tue, Feb 16, 2016 at 9:59 AM, Jaume Galí <jg...@konodrac.com> wrote: > >> Hi, >> >> I did all you suggest but i couldn’t solve the problem yet and i don’t >> know what else to do. >> >> Now I have a machine with 64Gb of Memory Ram, so physical memory should >> not be a problem any more. >> I attach input matrix if anybody could try to execute the command it >> would be great. >> >> This is what I tried: >> >> - I used this command as Angelo suggested: >> >> /opt/mahout/bin/mahout spark-rowsimilarity -i matrix_country_115k.dat -o >> test_country_115k_output.tmp --maxObservations 500 --maxSimilaritiesPerRow >> 100 --omitStrength --master local --sparkExecutorMem 10g >> -D:spark.dynamicAllocation.enabled=true >> -D:spark.shuffle.service.enabled=true >> >> - I increased *MAHOUT_HEAPSIZE *up to 32Gb in two ways: >> >> >> + Mahout script (MAHOUT_HOME/bin/mahout): >> >> JAVA=$JAVA_HOME/bin/java >> >> JAVA_HEAP_MAX=-Xmx4g >> >> MAHOUT_HEAPSIZE=32768 >> >> >> + ~/.profile setting environment variables: >> >> #Global conf JAVA >> export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 >> export JAVA_OPTS=-Xmx32g >> export _JAVA_OPTIONS=-Xmx32g >> export HADOOP_PREFIX=/opt/hadoop >> export SPARK_HOME=/opt/spark >> export MAHOUT_HOME=/opt/mahout >> export MAHOUT_HEAPSIZE=32g >> >> >> I printed trace of memoery following mahout script and this is output: >> >> run with heapsize 32768 >> >> -Xmx32768m >> >> So mahout is reading memory parameters fine. >> >> >> I’m glad if you people could guide me about what parameters I have to >> tune or check in order to solved this issue because I don’t know to do >> >> Thank you for advance. >> Jaume. >> >> >> >> El 13/2/2016, a las 22:56, Pat Ferrel <p...@occamsmachete.com> escribió: >> >> OK, this makes sense. When people see Out of Memory problems they >> naturally try to give more to the process throwing the exception but what >> is often happening is that you have given too much to the collection of >> other processes on the machine so there is not enough to go around and the >> allocation fails on Spark. In which case you need to allocate less to Spark >> so you can guarantee it will always be able to get that much. >> >> >> On Feb 13, 2016, at 9:30 AM, Angelo Leto <angl...@gmail.com> wrote: >> >> I was able to make it working by setting the executor memory to 10g >> and with -D:spark.dynamicAllocation.enabled=true : >> >> mahout spark-rowsimilarity --input hdfs:/indata/row-similarity.tsv >> --output rowsim-out --omitStrength --sparkExecutorMem 10g --master >> yarn-client -D:spark.dynamicAllocation.enabled=true >> -D:spark.shuffle.service.enabled=true >> >> >> On Sat, Feb 13, 2016 at 2:42 PM, Angelo Leto <angl...@gmail.com> wrote: >> >> Hello, >> I have the same problem described above using spark-rowsimilarity. >> I have a ~65k lines input file (each row with less than 300 items), >> and I run the job on a small cluster with 1 master and 2 workers, each >> machine has 15GB of RAM. >> I tried to increase executor and driver memory: >> --sparkExecutorMem 15g >> -D:spark.driver.memory=15g >> >> but I get the OutOfMemoryError exception: >> >> 16/02/13 13:00:36 ERROR Executor: Exception in task 0.0 in stage 12.0 >> (TID 12) >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> at >> org.apache.mahout.math.OrderedIntDoubleMapping.growTo(OrderedIntDoubleMapping.java:86) >> at >> org.apache.mahout.math.OrderedIntDoubleMapping.set(OrderedIntDoubleMapping.java:118) >> [...] >> >> Thanks for any hint. >> Angelo >> >> On Fri, Feb 12, 2016 at 10:15 PM, Pat Ferrel <p...@occamsmachete.com> >> wrote: >> >> You have to set the executor memory. BTW you have given the driver all >> memory on the machine. >> >> On Feb 10, 2016, at 9:30 AM, Jaume Galí <jg...@konodrac.com> wrote: >> >> Hi again, >> (Sorry for my delay but we didn’t have machine to test your thoughts >> about memory issue.) >> >> The problem still happening testing with an input matrix of 100k rows by >> 300 items, I increase memory as you suggest but nothing changed. I attached >> spark_env.sh and new specs of machine >> >> Machine specs: >> >> m3.xlarge AWS (Ivy Bridge, 15Gb ram, 2x40gb HD) >> >> This is my spark-env.sh: >> >> #!/usr/bin/env bash >> # Licensed to ... >> >> export SPARK_HOME=${SPARK_HOME:-/usr/lib/spark} >> export SPARK_LOG_DIR=${SPARK_LOG_DIR:-/var/log/spark} >> export HADOOP_HOME=${HADOOP_HOME:-/usr/lib/hadoop} >> export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf} >> export HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf} >> >> export STANDALONE_SPARK_MASTER_HOST=ip-10-12-17-235.eu < >> http://ip-10-12-17-235.eu/>-west-1.compute.internal >> export SPARK_MASTER_PORT=7077 >> export SPARK_MASTER_IP=$STANDALONE_SPARK_MASTER_HOST >> export SPARK_MASTER_WEBUI_PORT=8080 >> >> export SPARK_WORKER_DIR=${SPARK_WORKER_DIR:-/var/run/spark/work} >> export SPARK_WORKER_PORT=7078 >> export SPARK_WORKER_WEBUI_PORT=8081 >> >> export HIVE_SERVER2_THRIFT_BIND_HOST=0.0.0.0 >> export HIVE_SERVER2_THRIFT_PORT=10001 >> >> export SPARK_DRIVER_MEMORY=15G >> export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS >> -XX:OnOutOfMemoryError='kill -9 %p’” >> >> Log: >> >> Exception in thread "main" org.apache.spark.SparkException: Job aborted >> due to stage failure: Task 0 in stage 12.0 failed 1 times, most recent >> failure: Lost task 0.0 in stage 12.0 (TID 24, localhost): >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> ……. >> ….. >> .. >> . >> >> Driver stacktrace: >> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >> ……. >> ….. >> ... >> .. >> . >> >> >> Thanks for advance >> >> El 2/2/2016, a las 7:48, Pat Ferrel <p...@occamsmachete.com < >> mailto:p...@occamsmachete.com <p...@occamsmachete.com>>> escribió: >> >> You probably need to increase your driver memory and 8g will not work. >> 16g is probably the smallest stand alone machine that will work since the >> driver and executors run on it. >> >> On Feb 1, 2016, at 1:24 AM, jg...@konodrac.com <mailto:jg...@konodrac.com >> <jg...@konodrac.com>> wrote: >> >> Hello everybody, >> >> We are experimenting problems when we use "mahout spark-rowsimilarity” >> operation. We have an input matrix with 100k rows and 100 items and process >> throws an exception about “Exception in task 0.0 in stage 13.0 (TID 13) >> java.lang.OutOfMemoryError: Java heap space” and we try to increase JAVA >> HEAP MEMORY, MAHOUT HEAP MEMORY and spark.driver.memory. >> >> Environment versions: >> Mahout: 0.11.1 >> Spark: 1.6.0. >> >> Mahout command line: >> /opt/mahout/bin/mahout spark-rowsimilarity -i 50k_rows__50items.dat -o >> test_output.tmp --maxObservations 500 --maxSimilaritiesPerRow 100 >> --omitStrength --master local --sparkExecutorMem 8g >> >> This process is running on a machine with following specifications: >> Mem RAM: 8gb >> CPU with 8 cores >> >> .profile file: >> export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 >> export HADOOP_HOME=/opt/hadoop-2.6.0 >> export SPARK_HOME=/opt/spark >> export MAHOUT_HOME=/opt/mahout >> >> export MAHOUT_HEAPSIZE=8192 >> >> Throws exception: >> >> 16/01/22 11:45:06 ERROR Executor: Exception in task 0.0 in stage 13.0 >> (TID 13) >> java.lang.OutOfMemoryError: Java heap space >> at org.apache.mahout.math.DenseMatrix.<init>(DenseMatrix.java:66) >> at >> org.apache.mahout.sparkbindings.drm.package$$anonfun$blockify$1.apply(package.scala:70) >> at >> org.apache.mahout.sparkbindings.drm.package$$anonfun$blockify$1.apply(package.scala:59) >> at >> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) >> at >> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >> at org.apache.spark.scheduler.Task.run(Task.scala:89) >> at >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> 16/01/22 11:45:06 WARN NettyRpcEndpointRef: Error sending message >> [message = Heartbeat(driver,[Lscala.Tuple2;@12498227,BlockManagerId(driver, >> localhost, 42107))] in 1 attempts >> org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 >> seconds]. This timeout is controlled by spark.rpc.askTimeout >> at org.apache.spark.rpc.RpcTimeout.org >> $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) >> at >> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) >> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) >> at >> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101) >> at >> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77) >> at org.apache.spark.executor.Executor.org >> $apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:448) >> at >> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:468) >> at >> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:468) >> at >> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:468) >> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1741) >> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:468) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> 16/01/22 11:45:06 WARN NettyRpcEndpointRef: Error sending message >> [message = Heartbeat(driver,[Lscala.Tuple2;@12498227,BlockManagerId(driver, >> localhost, 42107))] in 1 attempts >> org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 >> seconds]. This timeout is controlled by spark.rpc.askTimeout >> at org.apache.spark.rpc.RpcTimeout.org >> $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) >> at >> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) >> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) >> at >> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101) >> at >> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77) >> at org.apache.spark.executor.Executor.org >> $apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:448) >> at >> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:468) >> at >> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:468) >> at >> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:468) >> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1741) >> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:468) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.util.concurrent.TimeoutException: Futures timed out after >> [120 seconds] >> at >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> at >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> at >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> at scala.concurrent.Await$.result(package.scala:107) >> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) >> ... >> >> Can you please advise? >> >> >> Thanks for advance. >> Cheers. >> >> >> >> >> >> >> >> >