Did you set `--driver-memory` with spark-submit? -Xiangrui
On Mon, May 4, 2015 at 5:16 PM, Vinay Muttineni <vmuttin...@ebay.com> wrote:
> Hi, I am training a GMM with 10 gaussians on a 4 GB dataset(720,000 * 760).
> The spark (1.3.1) job is allocated 120 executors with 6GB each and the
> driver also has 6GB.
> Spark Config Params:
>
> .set("spark.hadoop.validateOutputSpecs",
> "false").set("spark.dynamicAllocation.enabled",
> "false").set("spark.driver.maxResultSize",
> "4g").set("spark.default.parallelism", "300").set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buffer.mb",
> "500").set("spark.akka.frameSize", "256").set("spark.akka.timeout", "300")
>
> However, at the aggregate step (Line 168)
> val sums = breezeData.aggregate(ExpectationSum.zero(k, d))(compute.value, _
> += _)
>
> I get OOM error and the application hangs indefinitely. Is this an issue or
> am I missing something?
> java.lang.OutOfMemoryError: Java heap space
> at akka.util.CompactByteString$.apply(ByteString.scala:410)
> at akka.util.ByteString$.apply(ByteString.scala:22)
> at
> akka.remote.transport.netty.TcpHandlers$class.onMessage(TcpSupport.scala:45)
> at
> akka.remote.transport.netty.TcpServerHandler.onMessage(TcpSupport.scala:57)
> at
> akka.remote.transport.netty.NettyServerHelpers$class.messageReceived(NettyHelpers.scala:43)
> at
> akka.remote.transport.netty.ServerHandler.messageReceived(NettyTransport.scala:180)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> 15/05/04 16:23:38 ERROR util.Utils: Uncaught exception in thread
> task-result-getter-2
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "task-result-getter-2" java.lang.OutOfMemoryError: Java
> heap space
> 15/05/04 16:23:45 INFO scheduler.TaskSetManager: Finished task 1070.0 in
> stage 6.0 (TID 8276) in 382069 ms on [] (160/3600)
> 15/05/04 16:23:54 WARN channel.DefaultChannelPipeline: An exception was
> thrown by a user handler while handling an exception event ([id: 0xc57da871,
> ] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
> java.lang.OutOfMemoryError: Java heap space
> 15/05/04 16:23:55 WARN channel.DefaultChannelPipeline: An exception was
> thrown by a user handler while handling an exception event ([id: 0x3c3dbb0c,
> ] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
> 15/05/04 16:24:45 ERROR actor.ActorSystemImpl: Uncaught fatal error from
> thread [sparkDriver-akka.remote.default-remote-dispatcher-6] shutting down
> ActorSystem [sparkDriver]
>
>
>
> Thanks!
> Vinay
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
