Hey Reynold. I did look at the worker log, which I've attached in my original email btw.
I can't figure out what the logs are telling me though... Why is the connection getting reset? Something died? Out of memory? ?? 13/09/21 00:41:46 ERROR actor.ActorSystemImpl: RemoteServerError @akka://[email protected]:54538] Error[java.io.IOException:Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:193) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:63) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:385) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:256) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) ] 13/09/21 00:41:46 INFO worker.Worker: Executor app-20130921004115-0000/0 finished with state KILLED 13/09/21 00:41:46 WARN actor.ActorSystemImpl: RemoteClientWriteFailed @akka://[email protected]:45314: MessageClass[scala.Tuple3] Error[java.nio.channels.ClosedChannelException:null at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:698) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:421) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:127) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:66) On Mon, Sep 23, 2013 at 12:03 PM, Reynold Xin <[email protected]> wrote: > You will need to look into the worker's log. > > You can ssh to the worker machine, and look at the work folder in Spark. > > > -- > Reynold Xin, AMPLab, UC Berkeley > http://rxin.org > > > > On Sat, Sep 21, 2013 at 12:30 PM, Shay Seng <[email protected]> wrote: > >> Hey, >> I've been struggling to set up a work flow with spark. I'm basically >> using the AMI for the amplab3 tutorials, but added a couple of packages for >> R, rJava and some of my own jars. Basically Spark 0.7.3 standalone. (can't >> get Mesos running but that's a question for a different time) >> >> I read data from S3, and do a cascade of filters, maps, joins and reduce >> on them. If I perform the task with a smallish data set (<1000) it >> succeeds, but if I use a data set of > 1.5M rows, I keep getting the follow >> error when I do a collect on the RDD >> >> 13/09/21 00:41:45 INFO master.Master: Removing app app-20130921004115-0000 >> 13/09/21 00:41:45 ERROR actor.ActorSystemImpl: RemoteClientError@akka:// >> [email protected]:44283: Error[java.net.ConnectException:Connection >> refused >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708) >> at >> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404) >> at >> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366) >> at >> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:724) >> ] >> >> I'm at loss on where to start debugging -- is it some configuration >> issue on my part, or some scala error, or some spark error? I've attached >> the log file from the master and worker.... If anyone has any ideas on how >> to start debugging, please.. I'll be very appreciative. >> >> tks,shay >> > >
