Thanks for the info Matei. Seems like lots of other users of Akka have similar issues -- maybe at some point in the future it'll be worth making this a bit more flexible, but there are more important places to spend time right now.
On Fri, Dec 6, 2013 at 12:06 PM, Matei Zaharia <[email protected]>wrote: > Hey Andrew, unfortunately I don’t know how easy this is. Maybe future > versions of Akka have it. We can certainly ask them to do it in general but > I imagine there are some use cases where they wanted this behavior. > > Matei > > > On Dec 5, 2013, at 2:49 PM, Andrew Ash <[email protected]> wrote: > > Speaking of akka and host sensitivity... How much have you hacked on akka > to get it to support all of: myhost.mydomain.int, myhost, and 10.1.1.1? > It's kind of a pain to get the Spark URL to exactly match. I'm wondering > if there are usability gains that could be made here or if we're pretty > stuck. > > > On Thu, Dec 5, 2013 at 2:43 PM, Matei Zaharia <[email protected]>wrote: > >> Hi, >> >> When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the >> URL (uppercase instead of lowercase). Unfortunately Akka is very specific >> about seeing hostnames written in the same way on each node, or else it >> thinks the message is for another machine! >> >> Matei >> >> On Dec 5, 2013, at 8:27 AM, Adrian Bonar <[email protected]> >> wrote: >> >> The master starts up now as expected but the workers are unable to >> connect to the master. It looks like the master is refusing the connection >> messages but I’m not sure why. The first two error lines below are from >> trying to connect a worker from a separate machine and the last two error >> lines are from trying to connect a worker on the same machine as the >> master. I verified that the workers do not show up in the master’s web ui. >> >> MASTER: >> D:\spark>spark-class org.apache.spark.deploy.master.Master >> 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started >> 13/12/05 08:08:34 INFO master.Master: Starting Spark master at >> spark://ADRIBONA-DEV-1:7077 >> 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT >> 13/12/05 08:08:34 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/metrics/master/json,null} >> 13/12/05 08:08:34 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/metrics/applications/json,null} >> 13/12/05 08:08:34 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/static,null} >> 13/12/05 08:08:34 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/app/json,null} >> 13/12/05 08:08:34 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/app,null} >> 13/12/05 08:08:34 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/json,null} >> 13/12/05 08:08:34 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{*,null} >> 13/12/05 08:08:34 INFO server.AbstractConnector: Started >> [email protected]:8088 >> 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at >> http://ADRIBONA-DEV-1:8088 <http://adribona-dev-1:8088/> >> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message >> RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1) >> for non-local recipient >> akka://sparkMaster@adribona-dev-1:7077/user/Master at >> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is >> akka://sparkMaster@ADRIBONA-DEV-1:7077 >> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message >> DaemonMsgWatch(Actor[ >> akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) >> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at >> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is >> akka://sparkMaster@ADRIBONA-DEV-1:7077 >> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message >> RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1) >> for non-local recipient >> akka://sparkMaster@adribona-dev-1:7077/user/Master at >> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is >> akka://sparkMaster@ADRIBONA-DEV-1:7077 >> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message >> DaemonMsgWatch(Actor[ >> akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) >> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at >> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is >> akka://sparkMaster@ADRIBONA-DEV-1:7077 >> >> WORKER: >> D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker >> spark://adribona-dev-1:7077 >> 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started >> 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker >> ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM >> 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark >> 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT >> 13/12/05 08:18:46 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/metrics/json,null} >> 13/12/05 08:18:46 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/static,null} >> 13/12/05 08:18:46 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/log,null} >> 13/12/05 08:18:46 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/logPage,null} >> 13/12/05 08:18:46 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/json,null} >> 13/12/05 08:18:46 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{*,null} >> 13/12/05 08:18:46 INFO server.AbstractConnector: Started >> [email protected]:8081 >> 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at >> http://ADRIBONA-DEV-1:8081 <http://adribona-dev-1:8081/> >> 13/12/05 08:18:46 INFO worker.Worker: Connecting to master >> spark://adribona-dev-1:7077 >> >> >> --Adrian >> >> *From:* Adrian Bonar >> [mailto:[email protected]<[email protected]> >> ] >> *Sent:* Thursday, December 5, 2013 7:49 AM >> *To:* [email protected] >> *Subject:* RE: Pre-build Spark for Windows 8.1 >> >> Excellent! Thank you, Matei. >> >> *From:* Matei Zaharia >> [mailto:[email protected]<[email protected]> >> ] >> *Sent:* Wednesday, December 4, 2013 4:26 PM >> *To:* [email protected] >> *Subject:* Re: Pre-build Spark for Windows 8.1 >> >> Hey Adrian, >> >> Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts >> we provide instead. Cygwin might be made to work but we haven’t tried to do >> this so far so it’s not supported. If you can fix it, that would of course >> be welcome. >> >> Also, the deploy scripts don’t work on Windows — we assumed that people >> would mostly use Windows for local development and testing. However, if you >> do want to launch a cluster, you can use spark-class.cmd manually to launch >> a Master and then Workers. Use the commands in the first part of >> http://spark.incubator.apache.org/docs/latest/spark-standalone.html. >> >> Matei >> >> On Dec 4, 2013, at 2:31 PM, Adrian Bonar <[email protected]> >> wrote: >> >> >> >> Separate from my previous thread about *building* a distribution of >> Spark on Win8, I am also trying to run the pre-build binaries with little >> success. I downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to >> d:\spark and attempted to start a master with the following error: >> >> $ sh bin/start-master.sh >> starting org.apache.spark.deploy.master.Master, logging to >> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out >> failed to launch org.apache.spark.deploy.master.Master: >> >> Error: Could not find or load main class >> org.apache.spark.deploy.master.Master >> full log in >> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out >> >> --Adrian (again. J) >> >> >> > >
