Hi Thank you for the help. This is really great.
I have Zeppelin running now with a remote standalone Spark cluster. To summarize, you will need to install hadoop client, spark clients onto the same server you are running zeppelin. You will also need to ensure that you configure the .xml properly in /etc/hadoop/conf, /usr/lib/spark, /etc/hive/conf, each of these need to be configure to point to the right cluster you are using. I am able to run the examples that came with the binary. However I want to check to see if this a proper/normal behavior. I noticed that Zeppelin fires a job on spark and it never ends. See screenshot below: It seems the only way to get rid of it is to restart Zeppelin. Thanks again for all the help. > On Aug 4, 2016, at 8:46 PM, Jongyoul Lee <jongy...@gmail.com> wrote: > > If you have spark binary in your local, you can enforce the version by > setting SPARK_HOME in your zeppelin-env.sh > > On Fri, Aug 5, 2016 at 9:48 AM, Mohit Jaggi <mohitja...@gmail.com > <mailto:mohitja...@gmail.com>> wrote: > I defer to the experts but that is what i do and it works. Weird rpc errors > on spark are often due to version mismatch. I wish there was a simple version > check but afaik there isnt > > Sent from my iPhone > > On Aug 4, 2016, at 11:51 AM, Brian Liao <brianbl...@gmail.com > <mailto:brianbl...@gmail.com>> wrote: > >> So you are saying that I need to install spark locally in order to get this >> to work? >> >>> On Aug 4, 2016, at 11:12 AM, Mohit Jaggi <mohitja...@gmail.com >>> <mailto:mohitja...@gmail.com>> wrote: >>> >>> One of the Z config variables points to your local spark installation. Make >>> sure it is the same version as the one on the cluster. >>> >>>> On Aug 4, 2016, at 10:49 AM, Brian Liao <brianbl...@gmail.com >>>> <mailto:brianbl...@gmail.com>> wrote: >>>> >>>> Hi, >>>> >>>> Thank you. >>>> >>>> I used the prebuilt binary package version 0.6.0 from Zeppelin and my >>>> spark cluster is 1.5. >>>> >>>> Is there a way to force the prebuilt zeppelin to use 1.5? or the only way >>>> to do this is to compile from source? >>>> >>>> Also I don’t need to install Spark locally on where I host Zeppelin? >>>> >>>> >>>>> On Aug 4, 2016, at 10:25 AM, Mohit Jaggi <mohitja...@gmail.com >>>>> <mailto:mohitja...@gmail.com>> wrote: >>>>> >>>>> mismatched spark version? >>>>> >>>>> >>>>>> On Aug 4, 2016, at 8:11 AM, Brian Liao <brianbl...@gmail.com >>>>>> <mailto:brianbl...@gmail.com>> wrote: >>>>>> >>>>>> Hi >>>>>> >>>>>> I am following this guide (http://zeppelin.apache.org/download.html >>>>>> <http://zeppelin.apache.org/download.html>) to install Zeppelin, but >>>>>> wasn't able to configure it and get it to work. I used the binary >>>>>> package (the one that I don't need to compile anymore). >>>>>> >>>>>> I would like to have Zeppelin setup as a separate server but really have >>>>>> no idea what the requirement of this would be. >>>>>> >>>>>> I have a Spark (1.5) standalone cluster setup. >>>>>> >>>>>> The error I get from my Zeppelin interpreter log when trying to run a >>>>>> simple %md command is the following: >>>>>> >>>>>> INFO [2016-08-04 00:24:51,645] ({Thread-0} >>>>>> RemoteInterpreterServer.java[run]:81) - Starting remote interpreter >>>>>> server on port 40408 >>>>>> INFO [2016-08-04 00:24:51,980] ({pool-1-thread-2} >>>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>>> interpreter org.apache.zeppelin.spark.SparkInterpreter >>>>>> INFO [2016-08-04 00:24:52,019] ({pool-1-thread-2} >>>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>>> interpreter org.apache.zeppelin.spark.PySparkInterpreter >>>>>> INFO [2016-08-04 00:24:52,023] ({pool-1-thread-2} >>>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>>> interpreter org.apache.zeppelin.spark.SparkRInterpreter >>>>>> INFO [2016-08-04 00:24:52,024] ({pool-1-thread-2} >>>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>>> interpreter org.apache.zeppelin.spark.SparkSqlInterpreter >>>>>> INFO [2016-08-04 00:24:52,027] ({pool-1-thread-2} >>>>>> RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate >>>>>> interpreter org.apache.zeppelin.spark.DepInterpreter >>>>>> INFO [2016-08-04 00:24:52,056] ({pool-2-thread-2} >>>>>> SchedulerFactory.java[jobStarted]:131) - Job >>>>>> remoteInterpretJob_1470270292054 started by scheduler >>>>>> org.apache.zeppelin.spark.SparkInterpreter1041596993 >>>>>> WARN [2016-08-04 00:24:52,897] ({pool-2-thread-2} >>>>>> NativeCodeLoader.java[<clinit>]:62) - Unable to load native-hadoop >>>>>> library for your platform... using builtin-java classes where applicable >>>>>> INFO [2016-08-04 00:24:53,046] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Changing view acls to: root >>>>>> INFO [2016-08-04 00:24:53,047] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Changing modify acls to: root >>>>>> INFO [2016-08-04 00:24:53,047] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; >>>>>> ui acls disabled; users with view permissions: Set(root); users with >>>>>> modify permissions: Set(root) >>>>>> INFO [2016-08-04 00:24:53,279] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Starting HTTP Server >>>>>> INFO [2016-08-04 00:24:53,316] ({pool-2-thread-2} >>>>>> Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT >>>>>> INFO [2016-08-04 00:24:53,329] ({pool-2-thread-2} >>>>>> AbstractConnector.java[doStart]:338) - Started >>>>>> SocketConnector@0.0.0.0:42231 <http://SocketConnector@0.0.0.0:42231/> >>>>>> INFO [2016-08-04 00:24:53,330] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Successfully started service 'HTTP class >>>>>> server' on port 42231. >>>>>> INFO [2016-08-04 00:24:55,298] ({pool-2-thread-2} >>>>>> SparkInterpreter.java[createSparkContext]:233) - ------ Create new >>>>>> SparkContext spark://10.1.4.190:7077 <http://10.1.4.190:7077/> ------- >>>>>> INFO [2016-08-04 00:24:55,313] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Running Spark version 1.6.1 >>>>>> WARN [2016-08-04 00:24:55,326] ({pool-2-thread-2} >>>>>> Logging.scala[logWarning]:70) - >>>>>> SPARK_CLASSPATH was detected (set to >>>>>> ':/home/ubuntu/zeppelin/interpreter/spark/dep/*:/home/ubuntu/zeppelin/interpreter/spark/*::/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/lib/zeppelin-interpreter-0.6.0.jar:/etc/hadoop/conf/'). >>>>>> This is deprecated in Spark 1.0+. >>>>>> >>>>>> Please instead use: >>>>>> - ./spark-submit with --driver-class-path to augment the driver >>>>>> classpath >>>>>> - spark.executor.extraClassPath to augment the executor classpath >>>>>> >>>>>> WARN [2016-08-04 00:24:55,327] ({pool-2-thread-2} >>>>>> Logging.scala[logWarning]:70) - Setting 'spark.executor.extraClassPath' >>>>>> to >>>>>> ':/home/ubuntu/zeppelin/interpreter/spark/dep/*:/home/ubuntu/zeppelin/interpreter/spark/*::/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/lib/zeppelin-interpreter-0.6.0.jar:/etc/hadoop/conf/' >>>>>> as a work-around. >>>>>> WARN [2016-08-04 00:24:55,327] ({pool-2-thread-2} >>>>>> Logging.scala[logWarning]:70) - Setting 'spark.driver.extraClassPath' to >>>>>> ':/home/ubuntu/zeppelin/interpreter/spark/dep/*:/home/ubuntu/zeppelin/interpreter/spark/*::/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/conf:/home/ubuntu/zeppelin/lib/zeppelin-interpreter-0.6.0.jar:/etc/hadoop/conf/' >>>>>> as a work-around. >>>>>> INFO [2016-08-04 00:24:55,338] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Changing view acls to: root >>>>>> INFO [2016-08-04 00:24:55,339] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Changing modify acls to: root >>>>>> INFO [2016-08-04 00:24:55,339] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; >>>>>> ui acls disabled; users with view permissions: Set(root); users with >>>>>> modify permissions: Set(root) >>>>>> INFO [2016-08-04 00:24:55,483] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Successfully started service 'sparkDriver' >>>>>> on port 56365. >>>>>> INFO [2016-08-04 00:24:55,723] >>>>>> ({sparkDriverActorSystem-akka.actor.default-dispatcher-4} >>>>>> Slf4jLogger.scala[applyOrElse]:80) - Slf4jLogger started >>>>>> INFO [2016-08-04 00:24:55,749] >>>>>> ({sparkDriverActorSystem-akka.actor.default-dispatcher-4} >>>>>> Slf4jLogger.scala[apply$mcV$sp]:74) - Starting remoting >>>>>> INFO [2016-08-04 00:24:55,869] >>>>>> ({sparkDriverActorSystem-akka.actor.default-dispatcher-4} >>>>>> Slf4jLogger.scala[apply$mcV$sp]:74) - Remoting started; listening on >>>>>> addresses :[akka.tcp://sparkDriverActorSystem@10.1.4.253:57523 >>>>>> <http://sparkDriverActorSystem@10.1.4.253:57523/>] >>>>>> INFO [2016-08-04 00:24:55,869] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Successfully started service >>>>>> 'sparkDriverActorSystem' on port 57523. >>>>>> INFO [2016-08-04 00:24:55,878] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Registering MapOutputTracker >>>>>> INFO [2016-08-04 00:24:55,894] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Registering BlockManagerMaster >>>>>> INFO [2016-08-04 00:24:55,904] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Created local directory at >>>>>> /tmp/blockmgr-947bf6b6-5c70-4d29-b4b0-975692e0c08d >>>>>> INFO [2016-08-04 00:24:55,908] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - MemoryStore started with capacity 511.1 MB >>>>>> INFO [2016-08-04 00:24:55,981] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Registering OutputCommitCoordinator >>>>>> INFO [2016-08-04 00:24:56,081] ({pool-2-thread-2} >>>>>> Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT >>>>>> INFO [2016-08-04 00:24:56,100] ({pool-2-thread-2} >>>>>> AbstractConnector.java[doStart]:338) - Started >>>>>> SelectChannelConnector@0.0.0.0:4040 >>>>>> <http://SelectChannelConnector@0.0.0.0:4040/> >>>>>> INFO [2016-08-04 00:24:56,100] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Successfully started service 'SparkUI' on >>>>>> port 4040. >>>>>> INFO [2016-08-04 00:24:56,103] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Started SparkUI at http://10.1.4.253:4040 >>>>>> <http://10.1.4.253:4040/> >>>>>> INFO [2016-08-04 00:24:56,215] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - HTTP File server directory is >>>>>> /tmp/spark-96766055-2740-40cc-b077-475152c38b03/httpd-e1f81450-2a99-413e-af3f-b21fb5ece333 >>>>>> INFO [2016-08-04 00:24:56,215] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Starting HTTP Server >>>>>> INFO [2016-08-04 00:24:56,216] ({pool-2-thread-2} >>>>>> Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT >>>>>> INFO [2016-08-04 00:24:56,218] ({pool-2-thread-2} >>>>>> AbstractConnector.java[doStart]:338) - Started >>>>>> SocketConnector@0.0.0.0:42734 <http://SocketConnector@0.0.0.0:42734/> >>>>>> INFO [2016-08-04 00:24:56,218] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Successfully started service 'HTTP file >>>>>> server' on port 42734. >>>>>> INFO [2016-08-04 00:24:56,233] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Copying >>>>>> /home/ubuntu/zeppelin/interpreter/spark/pyspark/pyspark.zip to >>>>>> /tmp/spark-96766055-2740-40cc-b077-475152c38b03/userFiles-9f602668-e282-4214-be77-c68e36e9e110/pyspark.zip >>>>>> INFO [2016-08-04 00:24:56,242] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Added file >>>>>> file:/home/ubuntu/zeppelin/interpreter/spark/pyspark/pyspark.zip at >>>>>> http://10.1.4.253:42734/files/pyspark.zip >>>>>> <http://10.1.4.253:42734/files/pyspark.zip> with timestamp 1470270296233 >>>>>> INFO [2016-08-04 00:24:56,243] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Copying >>>>>> /home/ubuntu/zeppelin/interpreter/spark/pyspark/py4j-0.9-src.zip to >>>>>> /tmp/spark-96766055-2740-40cc-b077-475152c38b03/userFiles-9f602668-e282-4214-be77-c68e36e9e110/py4j-0.9-src.zip >>>>>> INFO [2016-08-04 00:24:56,251] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Added file >>>>>> file:/home/ubuntu/zeppelin/interpreter/spark/pyspark/py4j-0.9-src.zip at >>>>>> http://10.1.4.253:42734/files/py4j-0.9-src.zip >>>>>> <http://10.1.4.253:42734/files/py4j-0.9-src.zip> with timestamp >>>>>> 1470270296243 >>>>>> INFO [2016-08-04 00:24:56,285] ({pool-2-thread-2} >>>>>> Logging.scala[logInfo]:58) - Created default pool default, >>>>>> schedulingMode: FIFO, minShare: 0, weight: 1 >>>>>> INFO [2016-08-04 00:24:56,330] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logInfo]:58) - >>>>>> Connecting to master spark://10.1.4.190:7077 <>... >>>>>> INFO [2016-08-04 00:25:16,331] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logInfo]:58) - >>>>>> Connecting to master spark://10.1.4.190:7077 <>... >>>>>> ERROR [2016-08-04 00:25:16,341] ({shuffle-client-0} >>>>>> TransportResponseHandler.java[channelUnregistered]:122) - Still have 2 >>>>>> requests outstanding when connection from ip-10-1-4-190/10.1.4.190:7077 >>>>>> <http://10.1.4.190:7077/> is closed >>>>>> WARN [2016-08-04 00:25:16,343] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logWarning]:91) >>>>>> - Failed to connect to master 10.1.4.190:7077 <http://10.1.4.190:7077/> >>>>>> java.io.IOException: Connection from ip-10-1-4-190/10.1.4.190:7077 >>>>>> <http://10.1.4.190:7077/> closed >>>>>> at >>>>>> org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124) >>>>>> at >>>>>> org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>>> at >>>>>> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>>> at >>>>>> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>>> at >>>>>> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >>>>>> at >>>>>> io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739) >>>>>> at >>>>>> io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659) >>>>>> at >>>>>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) >>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>>>>> at >>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> INFO [2016-08-04 00:25:36,330] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logInfo]:58) - >>>>>> Connecting to master spark://10.1.4.190:7077 <>... >>>>>> INFO [2016-08-04 00:25:36,331] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logInfo]:58) - >>>>>> Connecting to master spark://10.1.4.190:7077 <>... >>>>>> ERROR [2016-08-04 00:25:36,332] ({shuffle-client-0} >>>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>>> 5850386894071965768 to ip-10-1-4-190/10.1.4.190:7077 >>>>>> <http://10.1.4.190:7077/>: java.nio.channels.ClosedChannelException >>>>>> java.nio.channels.ClosedChannelException >>>>>> ERROR [2016-08-04 00:25:36,333] ({shuffle-client-0} >>>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>>> 4955841159714871653 to ip-10-1-4-190/10.1.4.190:7077 >>>>>> <http://10.1.4.190:7077/>: java.nio.channels.ClosedChannelException >>>>>> java.nio.channels.ClosedChannelException >>>>>> WARN [2016-08-04 00:25:36,334] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logWarning]:91) >>>>>> - Failed to connect to master 10.1.4.190:7077 <http://10.1.4.190:7077/> >>>>>> java.io.IOException: Failed to send RPC 4955841159714871653 to >>>>>> ip-10-1-4-190/10.1.4.190:7077 <http://10.1.4.190:7077/>: >>>>>> java.nio.channels.ClosedChannelException >>>>>> at >>>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239) >>>>>> at >>>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226) >>>>>> at >>>>>> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) >>>>>> at >>>>>> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) >>>>>> at >>>>>> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) >>>>>> at >>>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801) >>>>>> at >>>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699) >>>>>> at >>>>>> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893) >>>>>> at >>>>>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) >>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>>>>> at >>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> Caused by: java.nio.channels.ClosedChannelException >>>>>> INFO [2016-08-04 00:25:56,330] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logInfo]:58) - >>>>>> Connecting to master spark://10.1.4.190:7077 <>... >>>>>> INFO [2016-08-04 00:25:56,331] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logInfo]:58) - >>>>>> Connecting to master spark://10.1.4.190:7077 <>... >>>>>> ERROR [2016-08-04 00:25:56,332] ({appclient-registration-retry-thread} >>>>>> Logging.scala[logError]:74) - Application has been killed. Reason: All >>>>>> masters are unresponsive! Giving up. >>>>>> ERROR [2016-08-04 00:25:56,334] ({shuffle-client-0} >>>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>>> 6244878284399143650 to ip-10-1-4-190/10.1.4.190:7077 >>>>>> <http://10.1.4.190:7077/>: java.nio.channels.ClosedChannelException >>>>>> java.nio.channels.ClosedChannelException >>>>>> WARN [2016-08-04 00:25:56,335] ({pool-2-thread-2} >>>>>> Logging.scala[logWarning]:70) - Application ID is not initialized yet. >>>>>> ERROR [2016-08-04 00:25:56,339] ({shuffle-client-0} >>>>>> TransportClient.java[operationComplete]:235) - Failed to send RPC >>>>>> 4693556837279618360 to ip-10-1-4-190/10.1.4.190:7077 >>>>>> <http://10.1.4.190:7077/>: java.nio.channels.ClosedChannelException >>>>>> java.nio.channels.ClosedChannelException >>>>>> WARN [2016-08-04 00:25:56,340] >>>>>> ({appclient-register-master-threadpool-0} Logging.scala[logWarning]:91) >>>>>> - Failed to connect to master 10.1.4.190:7077 <http://10.1.4.190:7077/> >>>>>> java.io.IOException: Failed to send RPC 4693556837279618360 to >>>>>> ip-10-1-4-190/10.1.4.190:7077 <http://10.1.4.190:7077/>: >>>>>> java.nio.channels.ClosedChannelException >>>>>> at >>>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239) >>>>>> at >>>>>> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226) >>>>>> at >>>>>> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) >>>>>> at >>>>>> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) >>>>>> at >>>>>> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) >>>>>> at >>>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801) >>>>>> at >>>>>> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699) >>>>>> at >>>>>> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960) >>>>>> at >>>>>> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893) >>>>>> at >>>>>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) >>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >>>>>> at >>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> Caused by: java.nio.channels.ClosedChannelException >>>>>> >>>>>> >>>>>> The error log from my spark master is: >>>>>> >>>>>> 16/08/04 00:25:15 ERROR actor.OneForOneStrategy: Error while decoding >>>>>> incoming Akka PDU of length: 1305 >>>>>> akka.remote.transport.AkkaProtocolException: Error while decoding >>>>>> incoming Akka PDU of length: 1305 >>>>>> Caused by: akka.remote.transport.PduCodecException: Decoding PDU failed. >>>>>> at >>>>>> akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:167) >>>>>> at >>>>>> akka.remote.transport.ProtocolStateActor.akka$remote$transport$ProtocolStateActor$$decodePdu(AkkaProtocolTransport.scala:513) >>>>>> at >>>>>> akka.remote.transport.ProtocolStateActor$$anonfun$4.applyOrElse(AkkaProtocolTransport.scala:320) >>>>>> at >>>>>> akka.remote.transport.ProtocolStateActor$$anonfun$4.applyOrElse(AkkaProtocolTransport.scala:292) >>>>>> at >>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) >>>>>> at akka.actor.FSM$class.processEvent(FSM.scala:595) >>>>>> at >>>>>> akka.remote.transport.ProtocolStateActor.processEvent(AkkaProtocolTransport.scala:220) >>>>>> at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:589) >>>>>> at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:583) >>>>>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) >>>>>> at akka.actor.ActorCell.invoke(ActorCell.scala:456) >>>>>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) >>>>>> at akka.dispatch.Mailbox.run(Mailbox.scala:219) >>>>>> at >>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>>>>> Caused by: com.google.protobuf_spark.InvalidProtocolBufferException: >>>>>> Protocol message contained an invalid tag (zero). >>>>>> at >>>>>> com.google.protobuf_spark.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:68) >>>>>> at >>>>>> com.google.protobuf_spark.CodedInputStream.readTag(CodedInputStream.java:108) >>>>>> at >>>>>> akka.remote.WireFormats$AkkaProtocolMessage$Builder.mergeFrom(WireFormats.java:5410) >>>>>> at >>>>>> akka.remote.WireFormats$AkkaProtocolMessage$Builder.mergeFrom(WireFormats.java:5275) >>>>>> at >>>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:300) >>>>>> at >>>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) >>>>>> at >>>>>> com.google.protobuf_spark.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:162) >>>>>> at >>>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:716) >>>>>> at >>>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) >>>>>> at >>>>>> com.google.protobuf_spark.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:153) >>>>>> at >>>>>> com.google.protobuf_spark.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:709) >>>>>> at >>>>>> akka.remote.WireFormats$AkkaProtocolMessage.parseFrom(WireFormats.java:5209) >>>>>> at >>>>>> akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:168) >>>>>> ... 17 more >>>>>> >>>>>> Regards, >>>>>> >>>>>> Brian >>>>> >>>> >>> >> > > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net <http://madeng.net/>
