Hey Andrew, unfortunately I don’t know how easy this is. Maybe future versions
of Akka have it. We can certainly ask them to do it in general but I imagine
there are some use cases where they wanted this behavior.
Matei
On Dec 5, 2013, at 2:49 PM, Andrew Ash <and...@andrewash.com> wrote:
> Speaking of akka and host sensitivity... How much have you hacked on akka to
> get it to support all of: myhost.mydomain.int, myhost, and 10.1.1.1? It's
> kind of a pain to get the Spark URL to exactly match. I'm wondering if there
> are usability gains that could be made here or if we're pretty stuck.
>
>
> On Thu, Dec 5, 2013 at 2:43 PM, Matei Zaharia <matei.zaha...@gmail.com> wrote:
> Hi,
>
> When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the URL
> (uppercase instead of lowercase). Unfortunately Akka is very specific about
> seeing hostnames written in the same way on each node, or else it thinks the
> message is for another machine!
>
> Matei
>
> On Dec 5, 2013, at 8:27 AM, Adrian Bonar <adrian.bo...@microsoft.com> wrote:
>
>> The master starts up now as expected but the workers are unable to connect
>> to the master. It looks like the master is refusing the connection messages
>> but I’m not sure why. The first two error lines below are from trying to
>> connect a worker from a separate machine and the last two error lines are
>> from trying to connect a worker on the same machine as the master. I
>> verified that the workers do not show up in the master’s web ui.
>>
>> MASTER:
>> D:\spark>spark-class org.apache.spark.deploy.master.Master
>> 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:08:34 INFO master.Master: Starting Spark master at
>> spark://ADRIBONA-DEV-1:7077
>> 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/master/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/app/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/app,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:08:34 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:8088
>> 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at
>> http://ADRIBONA-DEV-1:8088
>> 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message
>> RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1)
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master
>> at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message
>> DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message
>> RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1)
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master
>> atakka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message
>> DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>>
>> WORKER:
>> D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker
>> spark://adribona-dev-1:7077
>> 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker
>> ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
>> 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
>> 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/log,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/logPage,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:18:46 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:8081
>> 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at
>> http://ADRIBONA-DEV-1:8081
>> 13/12/05 08:18:46 INFO worker.Worker: Connecting to master
>> spark://adribona-dev-1:7077
>>
>>
>> --Adrian
>>
>> From: Adrian Bonar [mailto:adrian.bo...@microsoft.com]
>> Sent: Thursday, December 5, 2013 7:49 AM
>> To: user@spark.incubator.apache.org
>> Subject: RE: Pre-build Spark for Windows 8.1
>>
>> Excellent! Thank you, Matei.
>>
>> From: Matei Zaharia [mailto:matei.zaha...@gmail.com]
>> Sent: Wednesday, December 4, 2013 4:26 PM
>> To: user@spark.incubator.apache.org
>> Subject: Re: Pre-build Spark for Windows 8.1
>>
>> Hey Adrian,
>>
>> Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts we
>> provide instead. Cygwin might be made to work but we haven’t tried to do
>> this so far so it’s not supported. If you can fix it, that would of course
>> be welcome.
>>
>> Also, the deploy scripts don’t work on Windows — we assumed that people
>> would mostly use Windows for local development and testing. However, if you
>> do want to launch a cluster, you can use spark-class.cmd manually to launch
>> a Master and then Workers. Use the commands in the first part of
>> http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
>>
>> Matei
>>
>> On Dec 4, 2013, at 2:31 PM, Adrian Bonar <adrian.bo...@microsoft.com> wrote:
>>
>>
>> Separate from my previous thread about building a distribution of Spark on
>> Win8, I am also trying to run the pre-build binaries with little success. I
>> downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to d:\spark and
>> attempted to start a master with the following error:
>>
>> $ sh bin/start-master.sh
>> starting org.apache.spark.deploy.master.Master, logging to
>> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>> failed to launch org.apache.spark.deploy.master.Master:
>>
>> Error: Could not find or load main class
>> org.apache.spark.deploy.master.Master
>> full log in
>> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>>
>> --Adrian (again. J)
>
>
