Any help?please. Help me do a right configure.
李铖 <lidali...@gmail.com>于2015年4月7日星期二写道： > In my dev-test env .I have 3 virtual machines ,every machine have 12G > memory,8 cpu core. > > Here is spark-defaults.conf,and spark-env.sh.Maybe some config is not > right. > > I run this command :*spark-submit --master yarn-client --driver-memory 7g > --executor-memory 6g /home/hadoop/spark/main.py* > exception rised. > > *spark-defaults.conf* > > spark.master spark://cloud1:7077 > spark.default.parallelism 100 > spark.eventLog.enabled true > spark.serializer org.apache.spark.serializer.KryoSerializer > spark.driver.memory 5g > spark.driver.maxResultSize 6g > spark.kryoserializer.buffer.mb 256 > spark.kryoserializer.buffer.max.mb 512 > spark.executor.memory 4g > spark.rdd.compress true > spark.storage.memoryFraction 0 > spark.akka.frameSize 50 > spark.shuffle.compress true > spark.shuffle.spill.compress false > spark.local.dir /home/hadoop/tmp > > * spark-evn.sh* > > export SCALA=/home/hadoop/softsetup/scala > export JAVA_HOME=/home/hadoop/softsetup/jdk1.7.0_71 > export SPARK_WORKER_CORES=1 > export SPARK_WORKER_MEMORY=4g > export HADOOP_CONF_DIR=/opt/cloud/hadoop/etc/hadoop > export SPARK_EXECUTOR_MEMORY=4g > export SPARK_DRIVER_MEMORY=4g > > *Exception:* > > 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_28 on disk on > cloud3:38109 (size: 162.7 MB) > 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_28 on disk on > cloud3:38109 (size: 162.7 MB) > 15/04/07 18:11:03 INFO TaskSetManager: Starting task 31.0 in stage 1.0 > (TID 31, cloud3, NODE_LOCAL, 1296 bytes) > 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_29 on disk on > cloud2:49451 (size: 163.7 MB) > 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_29 on disk on > cloud2:49451 (size: 163.7 MB) > 15/04/07 18:11:03 INFO TaskSetManager: Starting task 30.0 in stage 1.0 > (TID 32, cloud2, NODE_LOCAL, 1296 bytes) > 15/04/07 18:11:03 ERROR Utils: Uncaught exception in thread > task-result-getter-0 > java.lang.OutOfMemoryError: Java heap space > at > org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:61) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985) > at > org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:58) > at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:81) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:73) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1460) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:48) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Exception in thread "task-result-getter-0" java.lang.OutOfMemoryError: > Java heap space > at > org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:61) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985) > at > org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:58) > at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:81) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:73) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1460) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:48) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_28 on disk on > cloud3:38109 (size: 162.7 MB) > 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_29 on disk on > cloud2:49451 (size: 163.7 MB) > 15/04/07 18:11:05 ERROR Utils: Uncaught exception in thread > task-result-getter-4 > java.lang.OutOfMemoryError: Java heap space > Exception in thread "task-result-getter-4" java.lang.OutOfMemoryError: > Java heap space > 15/04/07 18:11:07 INFO BlockManagerInfo: Added taskresult_31 on disk on > cloud3:38109 (size: 87.9 MB) > 15/04/07 18:11:07 INFO BlockManagerInfo: Added taskresult_31 on disk on > cloud3:38109 (size: 87.9 MB) > 15/04/07 18:11:08 WARN TransportChannelHandler: Exception in connection > from cloud3/192.168.0.95:38109 > java.lang.OutOfMemoryError: Java heap space > 15/04/07 18:11:08 ERROR TransportResponseHandler: Still have 1 requests > outstanding when connection from cloud3/192.168.0.95:38109 is closed > 15/04/07 18:11:08 ERROR OneForOneBlockFetcher: Failed while starting block > fetches > java.lang.OutOfMemoryError: Java heap space > 15/04/07 18:11:08 ERROR RetryingBlockFetcher: Failed to fetch block > taskresult_31, and will not retry (0 retries) > java.lang.OutOfMemoryError: Java heap space > 15/04/07 18:11:08 ERROR TransportClient: Failed to send RPC > 7722440433247749491 to cloud3/192.168.0.95:38109: > java.nio.channels.ClosedChannelException > java.nio.channels.ClosedChannelException > 15/04/07 18:11:08 ERROR OneForOneBlockFetcher: Failed while starting block > fetches > java.io.IOException: Failed to send RPC 7722440433247749491 to cloud3/ > 192.168.0.95:38109: java.nio.channels.ClosedChannelException > at > org.apache.spark.network.client.TransportClient$2.operationComplete(TransportClient.java:158) > at > org.apache.spark.network.client.TransportClient$2.operationComplete(TransportClient.java:145) > at > io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) > at > io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) > at > io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:745) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:646) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1054) > at > io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658) > at > io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:716) > at > io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:651) > at > io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:112) > at > io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658) > at > io.netty.channel.AbstractChannelHandlerContext.access$2000(AbstractChannelHandlerContext.java:32) > at > io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:939) > at > io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:991) > at > io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:924) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.nio.channels.ClosedChannelException > 15/04/07 18:11:08 INFO BlockManagerInfo: Added taskresult_30 on disk on > cloud1:44029 (size: 163.5 MB) > 15/04/07 18:11:08 INFO BlockManagerInfo: Added taskresult_30 on disk on > cloud1:44029 (size: 163.5 MB) > 15/04/07 18:11:08 ERROR Utils: Uncaught exception in thread > task-result-getter-6 > java.lang.OutOfMemoryError: Java heap space > Exception in thread "task-result-getter-6" java.lang.OutOfMemoryError: > Java heap space > 15/04/07 18:11:08 ERROR TaskResultGetter: Exception while getting task > result > java.util.concurrent.ExecutionException: Boxed Error > at scala.concurrent.impl.Promise$.resolver(Promise.scala:55) > at > scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244) > at scala.concurrent.Promise$class.complete(Promise.scala:55) > at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) > at scala.concurrent.Promise$class.failure(Promise.scala:107) > at scala.concurrent.impl.Promise$DefaultPromise.failure(Promise.scala:153) > at > org.apache.spark.network.BlockTransferService$$anon$1.onBlockFetchFailure(BlockTransferService.scala:92) > at > org.apache.spark.network.shuffle.RetryingBlockFetcher$RetryingBlockFetchListener.onBlockFetchFailure(RetryingBlockFetcher.java:230) > at > org.apache.spark.network.shuffle.OneForOneBlockFetcher.failRemainingBlocks(OneForOneBlockFetcher.java:123) > at > org.apache.spark.network.shuffle.OneForOneBlockFetcher.access$300(OneForOneBlockFetcher.java:43) > at > org.apache.spark.network.shuffle.OneForOneBlockFetcher$1.onFailure(OneForOneBlockFetcher.java:114) > at > org.apache.spark.network.client.TransportResponseHandler.failOutstandingRequests(TransportResponseHandler.java:84) > at > org.apache.spark.network.client.TransportResponseHandler.exceptionCaught(TransportResponseHandler.java:108) > at > org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:69) > at > io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:271) > at > io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:768) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.OutOfMemoryError: Java heap space >
