Hi Haripiya, When you run Spark on Mesos it needs to run
spark driver mesos scheduler and both need to be visible to outside world on public iface IP you need to tell Spark and Mesos on which interface to bind - by default they resolve node hostname to ip - this is loopback address in your case Possible solutions - on slave node with public IP 192.168.56.50 1. Set export LIBPROCESS_IP=192.168.56.50 export SPARK_LOCAL_IP=192.168.56.50 2. Ensure your hostname resolves to public iface IP - (for testing) edit /etc/hosts to resolve your domain name to 192.168.56.50 3. Set correct hostname/ip in mesos configuration - see Nikolaos answer 2015-07-29 1:04 GMT+02:00 Haripriya Ayyalasomayajula <[email protected] >: > It quits before it writes any logs, when I look at the directory for logs, > it has empty files! > > On Tue, Jul 28, 2015 at 2:34 PM, Vinod Kone <[email protected]> wrote: > >> can you paste the logs? >> >> On Tue, Jul 28, 2015 at 2:31 PM, Haripriya Ayyalasomayajula < >> [email protected]> wrote: >> >>> Well, when I try doing that I get this error: >>> >>> Failed to initialize: Failed to bind on scheduler_ip_address Cannot >>> assign requested address: Cannot assign requested address [99] >>> >>> When I do a >>> >>> ps -ef | grep mesos >>> >>> on both my master and slave nodes, it works fine. And, I am also able to >>> ping both the nodes from each other - they are reachable to one another. >>> >>> On Tue, Jul 28, 2015 at 2:18 PM, Vinod Kone <[email protected]> wrote: >>> >>>> LIBPROCESS_IP is the IP address that you want the scheduler (driver) to >>>> bind to. It has nothing to do with the ZooKeeper address. >>>> >>>> In other words, do >>>> >>>> export LIBPROCESS_IP=scheduler_ip_address >>>> >>>> >>>> On Tue, Jul 28, 2015 at 2:00 PM, Haripriya Ayyalasomayajula < >>>> [email protected]> wrote: >>>> >>>>> I am trying to do this >>>>> >>>>> export LIBPROCESS_IP=zk://my_ipaddress:2181/mesos >>>>> >>>>> ./bin/spark-shell >>>>> >>>>> It gives me this error and aborts >>>>> >>>>> WARNING: Logging before InitGoogleLogging() is written to STDERR >>>>> >>>>> F0728 15:43:39.361445 13209 process.cpp:847] Parsing >>>>> LIBPROCESS_IP=zk://my_ipaddress:2181/ failed: Failed to parse the IP >>>>> >>>>> On Tue, Jul 28, 2015 at 1:51 PM, Tim Chen <[email protected]> wrote: >>>>> >>>>>> spark-env.sh works as it will be called by spark-submit/spark-shell, >>>>>> or you can just set it before you call spark-shell yourself. >>>>>> >>>>>> Tim >>>>>> >>>>>> On Tue, Jul 28, 2015 at 1:43 PM, Haripriya Ayyalasomayajula < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Where can I set the libprocess_ip env variable? spark_env.sh? Thats >>>>>>> the only place I can think of. Can you please point me to any related >>>>>>> documentation? >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 12:46 PM, Nikolaos Ballas neXus < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> If you are not using any dns like service under >>>>>>>> /etc/mesos-master/ create two files called ip and hostname and put >>>>>>>> the ip >>>>>>>> of the eth interface. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Sent from my Samsung device >>>>>>>> >>>>>>>> >>>>>>>> -------- Original message -------- >>>>>>>> From: Haripriya Ayyalasomayajula <[email protected]> >>>>>>>> Date: 28/07/2015 20:18 (GMT+01:00) >>>>>>>> To: [email protected] >>>>>>>> Subject: Problems connecting with Mesos Master >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I am trying to use Spark 1.4.1 with Mesos 0.23.0. >>>>>>>> >>>>>>>> When I try to start my spark-shell, it gives me the following >>>>>>>> warning : >>>>>>>> >>>>>>>> ************************************************** >>>>>>>> >>>>>>>> Scheduler driver bound to loopback interface! Cannot communicate >>>>>>>> with remote master(s). You might want to set 'LIBPROCESS_IP' >>>>>>>> environment >>>>>>>> variable to use a routable IP address. >>>>>>>> --- >>>>>>>> >>>>>>>> Spark-shell works fine on the node where I run master, but if I >>>>>>>> start running on any of the other slave nodes it gives me the following >>>>>>>> error: >>>>>>>> >>>>>>>> E0728 11:22:53.176515 10503 socket.hpp:107] Shutdown failed on >>>>>>>> fd=6: Transport endpoint is not connected [107] >>>>>>>> >>>>>>>> E0728 11:22:53.210146 10503 socket.hpp:107] Shutdown failed on >>>>>>>> fd=6: Transport endpoint is not connected [107] >>>>>>>> >>>>>>>> I have the following configs: >>>>>>>> >>>>>>>> >>>>>>>> - zookeeper configured to the mesos master >>>>>>>> - /etc/mesos/zk on all nodes pointing to mesos master ip. >>>>>>>> >>>>>>>> I am not sure if I have to set the ip flag and where I have to >>>>>>>> set the --ip flag? >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Haripriya Ayyalasomayajula >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Haripriya Ayyalasomayajula >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Haripriya Ayyalasomayajula >>>>> >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> Haripriya Ayyalasomayajula >>> >>> >> > > > -- > Regards, > Haripriya Ayyalasomayajula > >
