Hi, When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the URL (uppercase instead of lowercase). Unfortunately Akka is very specific about seeing hostnames written in the same way on each node, or else it thinks the message is for another machine!
Matei On Dec 5, 2013, at 8:27 AM, Adrian Bonar <[email protected]> wrote: > The master starts up now as expected but the workers are unable to connect to > the master. It looks like the master is refusing the connection messages but > I’m not sure why. The first two error lines below are from trying to connect > a worker from a separate machine and the last two error lines are from trying > to connect a worker on the same machine as the master. I verified that the > workers do not show up in the master’s web ui. > > MASTER: > D:\spark>spark-class org.apache.spark.deploy.master.Master > 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started > 13/12/05 08:08:34 INFO master.Master: Starting Spark master at > spark://ADRIBONA-DEV-1:7077 > 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT > 13/12/05 08:08:34 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/metrics/master/json,null} > 13/12/05 08:08:34 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/metrics/applications/json,null} > 13/12/05 08:08:34 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/static,null} > 13/12/05 08:08:34 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/app/json,null} > 13/12/05 08:08:34 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/app,null} > 13/12/05 08:08:34 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/json,null} > 13/12/05 08:08:34 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{*,null} > 13/12/05 08:08:34 INFO server.AbstractConnector: Started > [email protected]:8088 > 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at > http://ADRIBONA-DEV-1:8088 > 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message > RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1) > for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master > at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is > akka://sparkMaster@ADRIBONA-DEV-1:7077 > 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message > DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) > for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at > akka://sparkMaster@ADRIBONA-DEV-1:7077 local is > akka://sparkMaster@ADRIBONA-DEV-1:7077 > 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message > RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1) > for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master > atakka://sparkMaster@ADRIBONA-DEV-1:7077 local is > akka://sparkMaster@ADRIBONA-DEV-1:7077 > 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message > DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) > for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at > akka://sparkMaster@ADRIBONA-DEV-1:7077 local is > akka://sparkMaster@ADRIBONA-DEV-1:7077 > > WORKER: > D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker > spark://adribona-dev-1:7077 > 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started > 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker > ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM > 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark > 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT > 13/12/05 08:18:46 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/metrics/json,null} > 13/12/05 08:18:46 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/static,null} > 13/12/05 08:18:46 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/log,null} > 13/12/05 08:18:46 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/logPage,null} > 13/12/05 08:18:46 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/json,null} > 13/12/05 08:18:46 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{*,null} > 13/12/05 08:18:46 INFO server.AbstractConnector: Started > [email protected]:8081 > 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at > http://ADRIBONA-DEV-1:8081 > 13/12/05 08:18:46 INFO worker.Worker: Connecting to master > spark://adribona-dev-1:7077 > > > --Adrian > > From: Adrian Bonar [mailto:[email protected]] > Sent: Thursday, December 5, 2013 7:49 AM > To: [email protected] > Subject: RE: Pre-build Spark for Windows 8.1 > > Excellent! Thank you, Matei. > > From: Matei Zaharia [mailto:[email protected]] > Sent: Wednesday, December 4, 2013 4:26 PM > To: [email protected] > Subject: Re: Pre-build Spark for Windows 8.1 > > Hey Adrian, > > Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts we > provide instead. Cygwin might be made to work but we haven’t tried to do this > so far so it’s not supported. If you can fix it, that would of course be > welcome. > > Also, the deploy scripts don’t work on Windows — we assumed that people would > mostly use Windows for local development and testing. However, if you do want > to launch a cluster, you can use spark-class.cmd manually to launch a Master > and then Workers. Use the commands in the first part of > http://spark.incubator.apache.org/docs/latest/spark-standalone.html. > > Matei > > On Dec 4, 2013, at 2:31 PM, Adrian Bonar <[email protected]> wrote: > > > Separate from my previous thread about building a distribution of Spark on > Win8, I am also trying to run the pre-build binaries with little success. I > downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to d:\spark and > attempted to start a master with the following error: > > $ sh bin/start-master.sh > starting org.apache.spark.deploy.master.Master, logging to > /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out > failed to launch org.apache.spark.deploy.master.Master: > > Error: Could not find or load main class > org.apache.spark.deploy.master.Master > full log in > /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out > > --Adrian (again. J)
