I found this issue on the mesos jira that describes the exact issue I am hitting.
https://issues.apache.org/jira/browse/MESOS-3740 It doesn't appear to be resolved. Thanks Eli > On 5 Jun 2016, at 16:46, Eli Jordan <[email protected]> wrote: > > Hmmm… that doesn’t seem to work for me. What version of mesos does this work > in? I am running 0.27.1. > > When using this approach, I still get the following error when the kafka mess > framework is starting up. > > "Scheduler driver bound to loopback interface! Cannot communicate with remote > master(s). You might want to set 'LIBPROCESS_IP' environment variable to use > a routable IP address.” > > I tried setting LIBPROCESS_IP to ‘0.0.0.0’ and LIBPROCESS_ADVERTISE_IP=‘the > public ip’ and this works. But the host variations don’t seem to work. (i.e. > set LIBPROCESS_IP=0.0.0.0 and LIBPROCESS_ADVERTISE_HOST=$HOST) > > It seems lib process doesn’t support using host names. > > I think I might have to run the framework outside of docker, but I would > really like to avoid this. > > This problem would be solved if the docker executor was able to set the same > environment variables as the command executor. Is there a way to make this > happen? > > I saw that mesos can be extended with a Hook ‘module’ to set extra > environment variables in docker containers. This might be a solution, but > seems over wrought for a simple problem. > > >> On 5 Jun 2016, at 12:50 am, Sivaram Kannan <[email protected]> wrote: >> >> >> Hi, >> >> Can you try adding && after the LIBPROCESS_HOST variable and the actual >> command. We have been using this for sometime now. >> >> "cmd": "LIBPROCESS_HOST=$HOST && ./kafka-mesos.sh .. >> >> Thanks, >> ./Siva. >> >> >>> On Sat, Jun 4, 2016 at 8:34 AM, Eli Jordan <[email protected]> wrote: >>> Hi @haosdent >>> >>> Based on my testing, this is not the case. >>> >>> I ran a task (from marathon) without using a docker container that just >>> printed out all environment variables. i.e. while [ true ]; do env; sleep >>> 2; done >>> >>> I then run a task that executed the same command inside an alpine docker >>> image. >>> >>> When running without a docker image LIBPROCESS_IP was defined along with >>> many other variables. >>> >>> Sample output when running without docker (note LIBPROCESS_IP) is defined >>> >>> Registered executor on mesos-slave0 >>> Starting task plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e >>> sh -c 'while [ true ]; do env; sleep 2; done' >>> Forked command at 16571 >>> LIBPROCESS_IP=192.168.3.16 >>> MESOS_AGENT_ENDPOINT=192.168.3.16:5051 >>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins >>> HOST=mesos-slave0 >>> SHELL=/bin/sh >>> MESOS_DIRECTORY=/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e/runs/c9b6ef86-b37d-4e3c-b1ca-bd680aed779f >>> PORT0=31082 >>> PORT_10001=31082 >>> LC_ALL=en_US.UTF-8 >>> … more >>> >>> >>> Sample output when running with docker (note LIBPROCESS_IP is not defined) >>> >>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>> --logbufsecs="0" --logging_level="INFO" >>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>> --stop_timeout="0ns" >>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>> --logbufsecs="0" --logging_level="INFO" >>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>> --stop_timeout="0ns" >>> Registered docker executor on mesos-slave0 >>> Starting task alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>> HOSTNAME=984809b0b720 >>> SHLVL=1 >>> HOME=/root >>> PORT=31295 >>> MESOS_CONTAINER_NAME=mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42 >>> MARATHON_APP_ID=/alpine-test >>> PORTS=31295 >>> PORT0=31295 >>> MARATHON_APP_DOCKER_IMAGE=alpine >>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>> MESOS_SANDBOX=/mnt/mesos/sandbox >>> MARATHON_APP_RESOURCE_DISK=0.0 >>> MARATHON_APP_RESOURCE_MEM=128.0 >>> HOST=mesos-slave0 >>> PORT_10000=31295 >>> MARATHON_APP_VERSION=2016-05-30T08:56:59.065Z >>> MARATHON_APP_LABELS= >>> PWD=/ >>> MESOS_TASK_ID=alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>> MARATHON_APP_RESOURCE_CPUS=1.0 >>> >>> Is there some other config I need to do for the docker containerizer? Any >>> help greatly appreciated. >>> >>>> On 4 Jun 2016, at 7:28 pm, haosdent <[email protected]> wrote: >>>> >>>> Hi, @Jordan. I think not matter you use MesosContainerizer or >>>> DockerContainerizer, LIBPROCESS_IP always would be set if you launch you >>>> Agent with `--ip` flag. >>>> >>>> On Fri, Jun 3, 2016 at 8:23 PM, Eli Jordan <[email protected]> >>>> wrote: >>>>> >>>>> The reason I need to set LIBPROCESS_IP is because the slaves have 2 >>>>> network interfaces, and the docker container is running in host >>>>> networking mode. So libmesos doesn’t know which IP to advertise. The >>>>> hostnames of the slaves are all resolvable. >>>>> >>>>> I have noticed that if I run a marathon app that doesn’t use docker, e.g. >>>>> while [ true ]; do env; sleep 2; done, that LIBPROCESS_IP is defined in >>>>> the environment. However, when running a docker image this variable is >>>>> not defined. >>>>> >>>>> Is there a way to have marathon pass along all environment variables >>>>> defined by mesos? >>>>> Thanks >>>>> Eli >>>>> >>>>>> On 4 Apr 2016, at 14:12, Eli Jordan <[email protected]> wrote: >>>>>> >>>>>> @haosdent I’m not sure how this works internally, but it seems the mess >>>>>> master needs to send requests to the framework for resource offers, and >>>>>> therefore needs to know the external IP (i.e. the host IP) >>>>>> >>>>>> @craig w >>>>>> Would I need to do this in the cmd portion of the marathon JSON? I >>>>>> currently have the config below >>>>>> { >>>>>> "container": ..., >>>>>> "id":"kafka-mesos-scheduler", >>>>>> "cpus": 0.5, >>>>>> "mem": 256, >>>>>> "ports": [9999], >>>>>> "cmd": "./kafka-mesos.sh scheduler --master=mesos-master:5050 >>>>>> --zk=mesos-master:2181 --api=http://mesos-slave0:9999 >>>>>> --storage=zk:/kafka-mesos", >>>>>> "instances": 1, >>>>>> "constraints": [["hostname", "LIKE", "mesos-slave0"]], >>>>>> "env": { >>>>>> "LIBPROCESS_IP": "192.168.3.16" >>>>>> } >>>>>> } >>>>>> >>>>>> @Chris Baker Currently we don’t have mess-dns setup but if this works it >>>>>> would seem to be a nice solution. However, I did try setting >>>>>> LIBPROCESS_IP to the slave hostname and it seems to produce a parse >>>>>> error. So I think it needs to be an actual IP address. >>>>>> >>>>>> I was hoping there would be a configuration for the slave that would >>>>>> automatically populate this env variable when starting the docker >>>>>> container. So I don’t need to complicate the marathon file, and can >>>>>> reuse it in different clusters. >>>>>> >>>>>> >>>>>>> On 4 Apr 2016, at 11:25 am, Chris Baker <[email protected]> wrote: >>>>>>> >>>>>>> Alternatively, because the $HOST username is indirect, which would >>>>>>> require a runtime element to "export $LIBPROCESS_IP=$HOST", another >>>>>>> alternative is to fallback on Mesos-DNS, if that's part of the cluster >>>>>>> deployment, setting $LIBPROCESS_IP to the (a priori) Mesos-DNS entry >>>>>>> corresponding to the service. >>>>>>> >>>>>>> On Sun, Apr 3, 2016 at 5:06 PM craig w <[email protected]> wrote: >>>>>>>> Hi, marathon sets the HOST env var. If it's not the ip address you can >>>>>>>> use getent with the value from HOST to figure it out. >>>>>>>> >>>>>>>> >However, in order for the frameworks to receive resource offers I >>>>>>>> >need to set the LIBPROCESS_IP environment variable to the hosts IP >>>>>>>> >address for the docker container running the frameworks. >>>>>>>> >>>>>>>> Hi, @Gmail. Could you provide more details about this? >>>>>>>> >>>>>>>>> On Sun, Apr 3, 2016 at 10:40 PM, Rad Gruchalski >>>>>>>>> <[email protected]> wrote: >>>>>>>>> Hi Gmail, >>>>>>>>> >>>>>>>>> AFAIK not. The only way to do so is setting up the env variable as >>>>>>>>> you do now. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> >>>>>>>>> Radek Gruchalski >>>>>>>>> >>>>>>>>> [email protected] >>>>>>>>> >>>>>>>>> de.linkedin.com/in/radgruchalski/ >>>>>>>>> >>>>>>>>> Confidentiality: >>>>>>>>> This communication is intended for the above-named person and may be >>>>>>>>> confidential and/or legally privileged. >>>>>>>>> If it has come to you in error you must take no action based on it, >>>>>>>>> nor must you copy or show it to anyone; please delete/destroy and >>>>>>>>> inform the sender immediately. >>>>>>>>>> On Sunday, 3 April 2016 at 16:09, Gmail wrote: >>>>>>>>>> >>>>>>>>>> I'm pretty new to mesos and marathon, and I'm running a couple of >>>>>>>>>> frameworks with marathon (Kafka and elastic search). However, in >>>>>>>>>> order for the frameworks to receive resource offers I need to set >>>>>>>>>> the LIBPROCESS_IP environment variable to the hosts IP address for >>>>>>>>>> the docker container running the frameworks. Currently I am working >>>>>>>>>> around me this by using a constraint to hard wire the slave that the >>>>>>>>>> framework gets launched on, so then I can put the slaves ip in the >>>>>>>>>> marathon json file. >>>>>>>>>> >>>>>>>>>> Obviously this is not ideal. Is there a better way to define the >>>>>>>>>> host ip Inside the docker container? >>>>>>>>>> >>>>>>>>>> Sent from my iPad >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Regards, >>>>>>>> Haosdent Huang >>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Haosdent Huang >> >> >> >> -- >> ever tried. ever failed. no matter. >> try again. fail again. fail better. >> -- Samuel Beckett >
