I defer to the experts but that is what i do and it works. Weird rpc errors on spark are often due to version mismatch. I wish there was a simple version check but afaik there isnt
Sent from my iPhone > On Aug 4, 2016, at 11:51 AM, Brian Liao <brianbl...@gmail.com> wrote: > > So you are saying that I need to install spark locally in order to get this > to work? > >> On Aug 4, 2016, at 11:12 AM, Mohit Jaggi <mohitja...@gmail.com> wrote: >> >> One of the Z config variables points to your local spark installation. Make >> sure it is the same version as the one on the cluster. >> >>> On Aug 4, 2016, at 10:49 AM, Brian Liao <brianbl...@gmail.com> wrote: >>> >>> Hi, >>> >>> Thank you. >>> >>> I used the prebuilt binary package version 0.6.0 from Zeppelin and my spark >>> cluster is 1.5. >>> >>> Is there a way to force the prebuilt zeppelin to use 1.5? or the only way >>> to do this is to compile from source? >>> >>> Also I don’t need to install Spark locally on where I host Zeppelin? >>> >>> >>>> On Aug 4, 2016, at 10:25 AM, Mohit Jaggi <mohitja...@gmail.com> wrote: >>>> >>>> mismatched spark version? >>>> >>>> >>>>> On Aug 4, 2016, at 8:11 AM, Brian Liao <brianbl...@gmail.com> wrote: >>>>> >>>>> Hi >>>>> >>>>> I am following this guide (http://zeppelin.apache.org/download.html) to >>>>> install Zeppelin, but wasn't able to configure it and get it to work. I >>>>> used the binary package (the one that I don't need to compile anymore). >>>>> >>>>> I would like to have Zeppelin setup as a separate server but really have >>>>> no idea what the requirement of this would be. >>>>> >>>>> I have a Spark (1.5) standalone cluster setup. >>>>> >>>>> The error I get from my Zeppelin interpreter log when trying to run a >>>>> simple %md command is the following: >>>>> >>>>> INFO [2016-08-04 00:24:51,645] ({Thread-0} >>>>> RemoteInterpreterServer.java[run]:81) - Starting remote interpreter >>>>> server on port 40408 >>>>> INFO [2016-08-04 00:24:51,980] ({pool-1-thread-2} >>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>> interpreter org.apache.zeppelin.spark.SparkInterpreter >>>>> INFO [2016-08-04 00:24:52,019] ({pool-1-thread-2} >>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>> interpreter org.apache.zeppelin.spark.PySparkInterpreter >>>>> INFO [2016-08-04 00:24:52,023] ({pool-1-thread-2} >>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>> interpreter org.apache.zeppelin.spark.SparkRInterpreter >>>>> INFO [2016-08-04 00:24:52,024] ({pool-1-thread-2} >>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>> interpreter org.apache.zeppelin.spark.SparkSqlInterpreter >>>>> INFO [2016-08-04 00:24:52,027] ({pool-1-thread-2} >>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>> interpreter org.apache.zeppelin.spark.DepInterpreter >>>>> INFO [2016-08-04 00:24:52,056] ({pool-2-thread-2} >>>>> SchedulerFactory.java[jobStarted]:131) - Job >>>>> remoteInterpretJob_1470270292054 started by scheduler >>>>> org.apache.zeppelin.spark.SparkInterpreter1041596993 >>>>> WARN [2016-08-04 00:24:52,897] ({pool-2-thread-2} >>>>> NativeCodeLoader.java[<clinit>]:62) - Unable to load native-hadoop >>>>> library for your platform... using builtin-java classes where applicable >>>>> INFO [2016-08-04 00:24:53,046] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Changing view acls to: root >>>>> INFO [2016-08-04 00:24:53,047] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Changing modify acls to: root >>>>> INFO [2016-08-04 00:24:53,047] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui >>>>> acls disabled; users with view permissions: Set(root); users with modify >>>>> permissions: Set(root) >>>>> INFO [2016-08-04 00:24:53,279] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Starting HTTP Server >>>>> INFO [2016-08-04 00:24:53,316] ({pool-2-thread-2} >>>>> Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT >>>>> INFO [2016-08-04 00:24:53,329] ({pool-2-thread-2} >>>>> AbstractConnector.java[doStart]:338) - Started >>>>> SocketConnector@0.0.0.0:42231 >>>>> INFO [2016-08-04 00:24:53,330] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Successfully started service 'HTTP class >>>>> server' on port 42231. >>>>> INFO [2016-08-04 00:24:55,298] ({pool-2-thread-2} >>>>> SparkInterpreter.java[createSparkContext]:233) - ------ Create new >>>>> SparkContext spark://10.1.4.190:7077 ------- >>>>> INFO [2016-08-04 00:24:55,313] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Running Spark version 1.6.1 >>>>> WARN [2016-08-04 00:24:55,326] ({pool-2-thread-2} >>>>> Logging.scala[logWarning]:70) - >>>>> SPARK_CLASSPATH was detected (set to >>>>> ':/home/ubuntu/zeppelin/interpreter/spark/dep/*:/home/ubuntu/zeppelin/interpreter/spark/*::/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/lib/zeppelin-interpreter-0.6.0.jar:/etc/hadoop/conf/'). >>>>> This is deprecated in Spark 1.0+. >>>>> >>>>> Please instead use: >>>>> - ./spark-submit with --driver-class-path to augment the driver classpath >>>>> - spark.executor.extraClassPath to augment the executor classpath >>>>> >>>>> WARN [2016-08-04 00:24:55,327] ({pool-2-thread-2} >>>>> Logging.scala[logWarning]:70) - Setting 'spark.executor.extraClassPath' >>>>> to >>>>> ':/home/ubuntu/zeppelin/interpreter/spark/dep/*:/home/ubuntu/zeppelin/interpreter/spark/*::/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/lib/zeppelin-interpreter-0.6.0.jar:/etc/hadoop/conf/' >>>>> as a work-around. >>>>> WARN [2016-08-04 00:24:55,327] ({pool-2-thread-2} >>>>> Logging.scala[logWarning]:70) - Setting 'spark.driver.extraClassPath' to >>>>> ':/home/ubuntu/zeppelin/interpreter/spark/dep/*:/home/ubuntu/zeppelin/interpreter/spark/*::/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/lib/zeppelin-interpreter-0.6.0.jar:/etc/hadoop/conf/' >>>>> as a work-around. >>>>> INFO [2016-08-04 00:24:55,338] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Changing view acls to: root >>>>> INFO [2016-08-04 00:24:55,339] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Changing modify acls to: root >>>>> INFO [2016-08-04 00:24:55,339] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui >>>>> acls disabled; users with view permissions: Set(root); users with modify >>>>> permissions: Set(root) >>>>> INFO [2016-08-04 00:24:55,483] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Successfully started service 'sparkDriver' >>>>> on port 56365. >>>>> INFO [2016-08-04 00:24:55,723] >>>>> ({sparkDriverActorSystem-akka.actor.default-dispatcher-4} >>>>> Slf4jLogger.scala[applyOrElse]:80) - Slf4jLogger started >>>>> INFO [2016-08-04 00:24:55,749] >>>>> ({sparkDriverActorSystem-akka.actor.default-dispatcher-4} >>>>> Slf4jLogger.scala[apply$mcV$sp]:74) - Starting remoting >>>>> INFO [2016-08-04 00:24:55,869] >>>>> ({sparkDriverActorSystem-akka.actor.default-dispatcher-4} >>>>> Slf4jLogger.scala[apply$mcV$sp]:74) - Remoting started; listening on >>>>> addresses :[akka.tcp://sparkDriverActorSystem@10.1.4.253:57523] >>>>> INFO [2016-08-04 00:24:55,869] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Successfully started service >>>>> 'sparkDriverActorSystem' on port 57523. >>>>> INFO [2016-08-04 00:24:55,878] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Registering MapOutputTracker >>>>> INFO [2016-08-04 00:24:55,894] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Registering BlockManagerMaster >>>>> INFO [2016-08-04 00:24:55,904] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Created local directory at >>>>> /tmp/blockmgr-947bf6b6-5c70-4d29-b4b0-975692e0c08d >>>>> INFO [2016-08-04 00:24:55,908] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - MemoryStore started with capacity 511.1 MB >>>>> INFO [2016-08-04 00:24:55,981] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Registering OutputCommitCoordinator >>>>> INFO [2016-08-04 00:24:56,081] ({pool-2-thread-2} >>>>> Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT >>>>> INFO [2016-08-04 00:24:56,100] ({pool-2-thread-2} >>>>> AbstractConnector.java[doStart]:338) - Started >>>>> SelectChannelConnector@0.0.0.0:4040 >>>>> INFO [2016-08-04 00:24:56,100] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Successfully started service 'SparkUI' on >>>>> port 4040. >>>>> INFO [2016-08-04 00:24:56,103] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Started SparkUI at http://10.1.4.253:4040 >>>>> INFO [2016-08-04 00:24:56,215] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - HTTP File server directory is >>>>> /tmp/spark-96766055-2740-40cc-b077-475152c38b03/httpd-e1f81450-2a99-413e-af3f-b21fb5ece333 >>>>> INFO [2016-08-04 00:24:56,215] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Starting HTTP Server >>>>> INFO [2016-08-04 00:24:56,216] ({pool-2-thread-2} >>>>> Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT >>>>> INFO [2016-08-04 00:24:56,218] ({pool-2-thread-2} >>>>> AbstractConnector.java[doStart]:338) - Started >>>>> SocketConnector@0.0.0.0:42734 >>>>> INFO [2016-08-04 00:24:56,218] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Successfully started service 'HTTP file >>>>> server' on port 42734. >>>>> INFO [2016-08-04 00:24:56,233] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Copying >>>>> /home/ubuntu/zeppelin/interpreter/spark/pyspark/pyspark.zip to >>>>> /tmp/spark-96766055-2740-40cc-b077-475152c38b03/userFiles-9f602668-e282-4214-be77-c68e36e9e110/pyspark.zip >>>>> INFO [2016-08-04 00:24:56,242] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Added file >>>>> file:/home/ubuntu/zeppelin/interpreter/spark/pyspark/pyspark.zip at >>>>> http://10.1.4.253:42734/files/pyspark.zip with timestamp 1470270296233 >>>>> INFO [2016-08-04 00:24:56,243] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Copying >>>>> /home/ubuntu/zeppelin/interpreter/spark/pyspark/py4j-0.9-src.zip to >>>>> /tmp/spark-96766055-2740-40cc-b077-475152c38b03/userFiles-9f602668-e282-4214-be77-c68e36e9e110/py4j-0.9-src.zip >>>>> INFO [2016-08-04 00:24:56,251] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Added file >>>>> file:/home/ubuntu/zeppelin/interpreter/spark/pyspark/py4j-0.9-src.zip at >>>>> http://10.1.4.253:42734/files/py4j-0.9-src.zip with timestamp >>>>> 1470270296243 >>>>> INFO [2016-08-04 00:24:56,285] ({pool-2-thread-2} >>>>> Logging.scala[logInfo]:58) - Created default pool default, >>>>> schedulingMode: FIFO, minShare: 0, weight: 1 >>>>> INFO [2016-08-04 00:24:56,330] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logInfo]:58) - Connecting to master >>>>> spark://10.1.4.190:7077... >>>>> INFO [2016-08-04 00:25:16,331] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logInfo]:58) - Connecting to master >>>>> spark://10.1.4.190:7077... >>>>> ERROR [2016-08-04 00:25:16,341] ({shuffle-client-0} >>>>> TransportResponseHandler.java[channelUnregistered]:122) - Still have 2 >>>>> requests outstanding when connection from ip-10-1-4-190/10.1.4.190:7077 >>>>> is closed >>>>> WARN [2016-08-04 00:25:16,343] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logWarning]:91) - Failed to connect to master >>>>> 10.1.4.190:7077 >>>>> java.io.IOException: Connection from ip-10-1-4-190/10.1.4.190:7077 closed >>>>> at >>>>> org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124) >>>>> at >>>>> org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>> at >>>>> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>> at >>>>> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>> at >>>>> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>> at >>>>> io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739) >>>>> at >>>>> io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659) >>>>> at >>>>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>>>> at >>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> INFO [2016-08-04 00:25:36,330] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logInfo]:58) - Connecting to master >>>>> spark://10.1.4.190:7077... >>>>> INFO [2016-08-04 00:25:36,331] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logInfo]:58) - Connecting to master >>>>> spark://10.1.4.190:7077... >>>>> ERROR [2016-08-04 00:25:36,332] ({shuffle-client-0} >>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>> 5850386894071965768 to ip-10-1-4-190/10.1.4.190:7077: >>>>> java.nio.channels.ClosedChannelException >>>>> java.nio.channels.ClosedChannelException >>>>> ERROR [2016-08-04 00:25:36,333] ({shuffle-client-0} >>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>> 4955841159714871653 to ip-10-1-4-190/10.1.4.190:7077: >>>>> java.nio.channels.ClosedChannelException >>>>> java.nio.channels.ClosedChannelException >>>>> WARN [2016-08-04 00:25:36,334] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logWarning]:91) - Failed to connect to master >>>>> 10.1.4.190:7077 >>>>> java.io.IOException: Failed to send RPC 4955841159714871653 to >>>>> ip-10-1-4-190/10.1.4.190:7077: java.nio.channels.ClosedChannelException >>>>> at >>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239) >>>>> at >>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226) >>>>> at >>>>> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) >>>>> at >>>>> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) >>>>> at >>>>> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) >>>>> at >>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801) >>>>> at >>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699) >>>>> at >>>>> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893) >>>>> at >>>>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>>>> at >>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: java.nio.channels.ClosedChannelException >>>>> INFO [2016-08-04 00:25:56,330] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logInfo]:58) - Connecting to master >>>>> spark://10.1.4.190:7077... >>>>> INFO [2016-08-04 00:25:56,331] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logInfo]:58) - Connecting to master >>>>> spark://10.1.4.190:7077... >>>>> ERROR [2016-08-04 00:25:56,332] ({appclient-registration-retry-thread} >>>>> Logging.scala[logError]:74) - Application has been killed. Reason: All >>>>> masters are unresponsive! Giving up. >>>>> ERROR [2016-08-04 00:25:56,334] ({shuffle-client-0} >>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>> 6244878284399143650 to ip-10-1-4-190/10.1.4.190:7077: >>>>> java.nio.channels.ClosedChannelException >>>>> java.nio.channels.ClosedChannelException >>>>> WARN [2016-08-04 00:25:56,335] ({pool-2-thread-2} >>>>> Logging.scala[logWarning]:70) - Application ID is not initialized yet. >>>>> ERROR [2016-08-04 00:25:56,339] ({shuffle-client-0} >>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>> 4693556837279618360 to ip-10-1-4-190/10.1.4.190:7077: >>>>> java.nio.channels.ClosedChannelException >>>>> java.nio.channels.ClosedChannelException >>>>> WARN [2016-08-04 00:25:56,340] ({appclient-register-master-threadpool-0} >>>>> Logging.scala[logWarning]:91) - Failed to connect to master >>>>> 10.1.4.190:7077 >>>>> java.io.IOException: Failed to send RPC 4693556837279618360 to >>>>> ip-10-1-4-190/10.1.4.190:7077: java.nio.channels.ClosedChannelException >>>>> at >>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239) >>>>> at >>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226) >>>>> at >>>>> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) >>>>> at >>>>> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) >>>>> at >>>>> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) >>>>> at >>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801) >>>>> at >>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699) >>>>> at >>>>> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960) >>>>> at >>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893) >>>>> at >>>>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>>>> at >>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: java.nio.channels.ClosedChannelException >>>>> >>>>> >>>>> The error log from my spark master is: >>>>> >>>>> 16/08/04 00:25:15 ERROR actor.OneForOneStrategy: Error while decoding >>>>> incoming Akka PDU of length: 1305 >>>>> akka.remote.transport.AkkaProtocolException: Error while decoding >>>>> incoming Akka PDU of length: 1305 >>>>> Caused by: akka.remote.transport.PduCodecException: Decoding PDU failed. >>>>> at >>>>> akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:167) >>>>> at >>>>> akka.remote.transport.ProtocolStateActor.akka$remote$transport$ProtocolStateActor$$decodePdu(AkkaProtocolTransport.scala:513) >>>>> at >>>>> akka.remote.transport.ProtocolStateActor$$anonfun$4.applyOrElse(AkkaProtocolTransport.scala:320) >>>>> at >>>>> akka.remote.transport.ProtocolStateActor$$anonfun$4.applyOrElse(AkkaProtocolTransport.scala:292) >>>>> at >>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) >>>>> at akka.actor.FSM$class.processEvent(FSM.scala:595) >>>>> at >>>>> akka.remote.transport.ProtocolStateActor.processEvent(AkkaProtocolTransport.scala:220) >>>>> at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:589) >>>>> at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:583) >>>>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) >>>>> at akka.actor.ActorCell.invoke(ActorCell.scala:456) >>>>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) >>>>> at akka.dispatch.Mailbox.run(Mailbox.scala:219) >>>>> at >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) >>>>> at >>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>>>> at >>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>>>> at >>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>>>> at >>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>>>> Caused by: com.google.protobuf_spark.InvalidProtocolBufferException: >>>>> Protocol message contained an invalid tag (zero). >>>>> at >>>>> com.google.protobuf_spark.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:68) >>>>> at >>>>> com.google.protobuf_spark.CodedInputStream.readTag(CodedInputStream.java:108) >>>>> at >>>>> akka.remote.WireFormats$AkkaProtocolMessage$Builder.mergeFrom(WireFormats.java:5410) >>>>> at >>>>> akka.remote.WireFormats$AkkaProtocolMessage$Builder.mergeFrom(WireFormats.java:5275) >>>>> at >>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:300) >>>>> at >>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) >>>>> at >>>>> com.google.protobuf_spark.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:162) >>>>> at >>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:716) >>>>> at >>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) >>>>> at >>>>> com.google.protobuf_spark.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:153) >>>>> at >>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:709) >>>>> at >>>>> akka.remote.WireFormats$AkkaProtocolMessage.parseFrom(WireFormats.java:5209) >>>>> at >>>>> akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:168) >>>>> ... 17 more >>>>> >>>>> Regards, >>>>> >>>>> Brian >
