Re: Windows - Spark 2 - Standalone - Worker not able to connect to Master
Your exception says, that you have connection trouble with Spark master. Check if it is available from your environment where you trying to run job. In Linux system for this can be suitable this commands: "telnet 127.0.0.1 7077" or "netstat -ntpl | grep 7077" or "nmap 127.0.0.1 | grep 7077". Try to use analog of this commands in Windows and check if is available spark master from your running environment? 2016-08-01 14:35 GMT+03:00 ayan guha : > No I confirmed master is running by spark ui at localhost:8080 > On 1 Aug 2016 18:22, "Nikolay Zhebet" wrote: > >> I think you haven't run spark master yet, or maybe port 7077 is not yours >> default port for spark master. >> >> 2016-08-01 4:24 GMT+03:00 ayan guha : >> >>> Hi >>> >>> I just downloaded Spark 2.0 on my windows 7 to check it out. However, >>> not able to set up a standalone cluster: >>> >>> >>> Step 1: master set up (Successful) >>> >>> bin/spark-class org.apache.spark.deploy.master.Master >>> >>> It did throw an error about not able to find winutils, but started >>> successfully. >>> >>> Step II: Set up Worker (Failed) >>> >>> bin/spark-class org.apache.spark.deploy.worker.Worker >>> spark://localhost:7077 >>> >>> This step fails with following error: >>> >>> 16/08/01 11:21:27 INFO Worker: Connecting to master localhost:7077... >>> 16/08/01 11:21:28 WARN Worker: Failed to connect to master localhost:7077 >>> org.apache.spark.SparkException: Exception thrown in awaitResult >>> at >>> org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca >>> la:77) >>> at >>> org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca >>> la:75) >>> at >>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.s >>> cala:36) >>> at >>> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO >>> rElse(RpcTimeout.scala:59) >>> at >>> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO >>> rElse(RpcTimeout.scala:59) >>> at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) >>> at >>> org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) >>> at >>> org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88) >>> at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96) >>> at >>> org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deplo >>> y$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:216) >>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) >>> at java.util.concurrent.FutureTask.run(Unknown Source) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>> Source) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> Source) >>> at java.lang.Thread.run(Unknown Source) >>> Caused by: java.io.IOException: Failed to connect to localhost/ >>> 127.0.0.1:7077 >>> at >>> org.apache.spark.network.client.TransportClientFactory.createClient(T >>> ransportClientFactory.java:228) >>> at >>> org.apache.spark.network.client.TransportClientFactory.createClient(T >>> ransportClientFactory.java:179) >>> at >>> org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala >>> :197) >>> at >>> org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191) >>> at >>> org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) >>> ... 4 more >>> Caused by: java.net.ConnectException: Connection refused: no further >>> information >>> : localhost/127.0.0.1:7077 >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) >>> at >>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocke >>> tChannel.java:224) >>> at >>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConne >>> ct(AbstractNioChannel.java:289) >>> at >>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.jav >>> a:528) >>> at >>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEve >>> ntLoop.java:468) >>> at >>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.ja >>> va:382) >>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >>> at >>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThread >>> EventExecutor.java:111) >>> ... 1 more >>> >>> Am I doing something wrong? >>> >>> >>> -- >>> Best Regards, >>> Ayan Guha >>> >> >>
Re: Windows - Spark 2 - Standalone - Worker not able to connect to Master
No I confirmed master is running by spark ui at localhost:8080 On 1 Aug 2016 18:22, "Nikolay Zhebet" wrote: > I think you haven't run spark master yet, or maybe port 7077 is not yours > default port for spark master. > > 2016-08-01 4:24 GMT+03:00 ayan guha : > >> Hi >> >> I just downloaded Spark 2.0 on my windows 7 to check it out. However, not >> able to set up a standalone cluster: >> >> >> Step 1: master set up (Successful) >> >> bin/spark-class org.apache.spark.deploy.master.Master >> >> It did throw an error about not able to find winutils, but started >> successfully. >> >> Step II: Set up Worker (Failed) >> >> bin/spark-class org.apache.spark.deploy.worker.Worker >> spark://localhost:7077 >> >> This step fails with following error: >> >> 16/08/01 11:21:27 INFO Worker: Connecting to master localhost:7077... >> 16/08/01 11:21:28 WARN Worker: Failed to connect to master localhost:7077 >> org.apache.spark.SparkException: Exception thrown in awaitResult >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca >> la:77) >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca >> la:75) >> at >> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.s >> cala:36) >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO >> rElse(RpcTimeout.scala:59) >> at >> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO >> rElse(RpcTimeout.scala:59) >> at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) >> at >> org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) >> at >> org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88) >> at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96) >> at >> org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deplo >> y$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:216) >> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >> Source) >> at java.util.concurrent.FutureTask.run(Unknown Source) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >> Source) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >> Source) >> at java.lang.Thread.run(Unknown Source) >> Caused by: java.io.IOException: Failed to connect to localhost/ >> 127.0.0.1:7077 >> at >> org.apache.spark.network.client.TransportClientFactory.createClient(T >> ransportClientFactory.java:228) >> at >> org.apache.spark.network.client.TransportClientFactory.createClient(T >> ransportClientFactory.java:179) >> at >> org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala >> :197) >> at >> org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191) >> at >> org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) >> ... 4 more >> Caused by: java.net.ConnectException: Connection refused: no further >> information >> : localhost/127.0.0.1:7077 >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) >> at >> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocke >> tChannel.java:224) >> at >> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConne >> ct(AbstractNioChannel.java:289) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.jav >> a:528) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEve >> ntLoop.java:468) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.ja >> va:382) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThread >> EventExecutor.java:111) >> ... 1 more >> >> Am I doing something wrong? >> >> >> -- >> Best Regards, >> Ayan Guha >> > >
Re: Windows - Spark 2 - Standalone - Worker not able to connect to Master
I think you haven't run spark master yet, or maybe port 7077 is not yours default port for spark master. 2016-08-01 4:24 GMT+03:00 ayan guha : > Hi > > I just downloaded Spark 2.0 on my windows 7 to check it out. However, not > able to set up a standalone cluster: > > > Step 1: master set up (Successful) > > bin/spark-class org.apache.spark.deploy.master.Master > > It did throw an error about not able to find winutils, but started > successfully. > > Step II: Set up Worker (Failed) > > bin/spark-class org.apache.spark.deploy.worker.Worker > spark://localhost:7077 > > This step fails with following error: > > 16/08/01 11:21:27 INFO Worker: Connecting to master localhost:7077... > 16/08/01 11:21:28 WARN Worker: Failed to connect to master localhost:7077 > org.apache.spark.SparkException: Exception thrown in awaitResult > at > org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca > la:77) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca > la:75) > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.s > cala:36) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO > rElse(RpcTimeout.scala:59) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO > rElse(RpcTimeout.scala:59) > at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) > at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) > at > org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88) > at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96) > at > org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deplo > y$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:216) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Failed to connect to localhost/ > 127.0.0.1:7077 > at > org.apache.spark.network.client.TransportClientFactory.createClient(T > ransportClientFactory.java:228) > at > org.apache.spark.network.client.TransportClientFactory.createClient(T > ransportClientFactory.java:179) > at > org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala > :197) > at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191) > at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) > ... 4 more > Caused by: java.net.ConnectException: Connection refused: no further > information > : localhost/127.0.0.1:7077 > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) > at > io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocke > tChannel.java:224) > at > io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConne > ct(AbstractNioChannel.java:289) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.jav > a:528) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEve > ntLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.ja > va:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThread > EventExecutor.java:111) > ... 1 more > > Am I doing something wrong? > > > -- > Best Regards, > Ayan Guha >
Windows - Spark 2 - Standalone - Worker not able to connect to Master
Hi I just downloaded Spark 2.0 on my windows 7 to check it out. However, not able to set up a standalone cluster: Step 1: master set up (Successful) bin/spark-class org.apache.spark.deploy.master.Master It did throw an error about not able to find winutils, but started successfully. Step II: Set up Worker (Failed) bin/spark-class org.apache.spark.deploy.worker.Worker spark://localhost:7077 This step fails with following error: 16/08/01 11:21:27 INFO Worker: Connecting to master localhost:7077... 16/08/01 11:21:28 WARN Worker: Failed to connect to master localhost:7077 org.apache.spark.SparkException: Exception thrown in awaitResult at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca la:77) at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.sca la:75) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.s cala:36) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO rElse(RpcTimeout.scala:59) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyO rElse(RpcTimeout.scala:59) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88) at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96) at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deplo y$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:216) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Failed to connect to localhost/ 127.0.0.1:7077 at org.apache.spark.network.client.TransportClientFactory.createClient(T ransportClientFactory.java:228) at org.apache.spark.network.client.TransportClientFactory.createClient(T ransportClientFactory.java:179) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala :197) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) ... 4 more Caused by: java.net.ConnectException: Connection refused: no further information : localhost/127.0.0.1:7077 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocke tChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConne ct(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.jav a:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEve ntLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.ja va:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThread EventExecutor.java:111) ... 1 more Am I doing something wrong? -- Best Regards, Ayan Guha