The master starts up now as expected but the workers are unable to connect to
the master. It looks like the master is refusing the connection messages but
I'm not sure why. The first two error lines below are from trying to connect a
worker from a separate machine and the last two error lines are from trying to
connect a worker on the same machine as the master. I verified that the workers
do not show up in the master's web ui.
MASTER:
D:\spark>spark-class org.apache.spark.deploy.master.Master
13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/05 08:08:34 INFO master.Master: Starting Spark master at
spark://ADRIBONA-DEV-1:7077
13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/05 08:08:34 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/metrics/master/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/static,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/app/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/app,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{*,null}
13/12/05 08:08:34 INFO server.AbstractConnector: Started
[email protected]:8088
13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at
http://ADRIBONA-DEV-1:8088
13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message
RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1)
for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master at
akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message
DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message
RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1)
for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master at
akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message
DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
akka://sparkMaster@ADRIBONA-DEV-1:7077
WORKER:
D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker
spark://adribona-dev-1:7077
13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker
ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/05 08:18:46 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/metrics/json,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/static,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/log,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/logPage,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/json,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{*,null}
13/12/05 08:18:46 INFO server.AbstractConnector: Started
[email protected]:8081
13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at
http://ADRIBONA-DEV-1:8081
13/12/05 08:18:46 INFO worker.Worker: Connecting to master
spark://adribona-dev-1:7077
--Adrian
From: Adrian Bonar [mailto:[email protected]]
Sent: Thursday, December 5, 2013 7:49 AM
To: [email protected]
Subject: RE: Pre-build Spark for Windows 8.1
Excellent! Thank you, Matei.
From: Matei Zaharia [mailto:[email protected]]
Sent: Wednesday, December 4, 2013 4:26 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Pre-build Spark for Windows 8.1
Hey Adrian,
Ideally you shouldn't use Cygwin to run on Windows - use the .cmd scripts we
provide instead. Cygwin might be made to work but we haven't tried to do this
so far so it's not supported. If you can fix it, that would of course be
welcome.
Also, the deploy scripts don't work on Windows - we assumed that people would
mostly use Windows for local development and testing. However, if you do want
to launch a cluster, you can use spark-class.cmd manually to launch a Master
and then Workers. Use the commands in the first part of
http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
Matei
On Dec 4, 2013, at 2:31 PM, Adrian Bonar
<[email protected]<mailto:[email protected]>> wrote:
Separate from my previous thread about building a distribution of Spark on
Win8, I am also trying to run the pre-build binaries with little success. I
downloaded and extract "spark-0.8.0-incubating-bin-hadoop1" to d:\spark and
attempted to start a master with the following error:
$ sh bin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to
/cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
failed to launch org.apache.spark.deploy.master.Master:
Error: Could not find or load main class org.apache.spark.deploy.master.Master
full log in
/cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
--Adrian (again. :))
