It's important to note that if you run a task with the command executor (I.e. Not using docker) LIBPROCESS_IP is defined, along with several other variables that are not defined in docker.
Thanks Eli > On 7 Jun 2016, at 10:05, Radoslaw Gruchalski <[email protected]> wrote: > > I think the problem is that it is not known which agent the task is running > on until the task i in the running state. > Hence the master can’t pass that as an env variable to the task. > However, I see your point. There is an agent host name avaialble in the task > as $HOST. Maybe it would be a good idea if Mesos was setting env variables > named AGENT_IP_0, AGENT_IP_1 and so on for every IP interface on the agent, > maybe AGENT_BIND_IP if bind IP is different than 0.0.0.0. OTOH, I can see how > this could be considered as some security issue. I am not sure what the > implications could be. > > Anybody else care to comment? > – > Best regards, > > Radek Gruchalski > > [email protected] > de.linkedin.com/in/radgruchalski > > Confidentiality: > This communication is intended for the above-named person and may be > confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor must > you copy or show it to anyone; please delete/destroy and inform the sender > immediately. > >> On June 7, 2016 at 1:42:46 AM, Eli Jordan ([email protected]) wrote: >> >> Thanks Radoslaw. I'm not really set on using host names, I just want a >> reliable way to start the framework. For the meantime I have gone with a >> solution similar to what you suggested. We use /etc/default/mesos file to >> configure mesos, and it has the ip defined, so I just mounted that in the >> container and read the ip. >> >> I would like to avoid having a dependency on the file system of the agents >> though. I'm not sure why I can't have the docket executor set the >> LIBPROCESS_IP variable in the same way the command executor does. >> >> Thanks >> Eli >> >> On 6 Jun 2016, at 21:44, Radoslaw Gruchalski <[email protected]> wrote: >> >>> Out of curiosity. Why are you insisting on using host names? >>> Say you have 1 master and 2 agents with these IPs: >>> >>> - mesos-master-0: 10.100.1.10 >>> - mesos-agent-0: 10.100.1.11 >>> - mesos-agent-1: 10.100.1.12 >>> >>> Your problem is that you have no way to obtain an IP address of the agent >>> in the container. Correct? >>> One way to overcome this problem is to create a shell file, say in >>> /etc/mesos-agent.sh, with contents like: >>> >>> ... >>> AGENT_IP=10.100.1.11 >>> … >>> >>> If you’re using Marathon, you can copy that file to the sandbox using >>> docker volumes: >>> >>> { >>> "containerPath": “/etc/mesos-agent.sh", >>> "hostPath": "/etc/mesos-agent.sh", >>> "mode": "RO" >>> } >>> >>> You can now source that in the container to set the LIBPROCESS_ADVERTISE_IP. >>> Other applications simply use the mesos-agent-X host name. That’s without >>> mesos-dns. >>> Things are easier with mesos-dns or consul service catalog (I prefer the >>> latter). >>> – >>> Best regards, >>> >>> Radek Gruchalski >>> >>> [email protected] >>> de.linkedin.com/in/radgruchalski >>> >>> Confidentiality: >>> This communication is intended for the above-named person and may be >>> confidential and/or legally privileged. >>> If it has come to you in error you must take no action based on it, nor >>> must you copy or show it to anyone; please delete/destroy and inform the >>> sender immediately. >>> >>>> On June 6, 2016 at 1:16:07 PM, Eli Jordan ([email protected]) wrote: >>>> >>>> The issue refers to LIBPROCESS_IP not LIBPROCESS_HOST. I haven’t been able >>>> to find the LIBPROCESS_HOST variable documented anywhere. >>>> >>>> My understanding is that the scheduler uses LIBPROCESS_IP to determine >>>> which network interface to bind to, and also which ip to advertise to the >>>> master, so that the master can send offers. There is also another variable >>>> LIBPROCESS_ADVERTISE_IP. If this is defined then LIBPROCESS_IP is used to >>>> determine which network interface to bind to, and LIBPROCESS_ADVERTISE_IP >>>> is used to determine which ip to advertise to the master. >>>> >>>> It would be great if there was a LIBPROCESS_ADVERTISE_HOST variable, then >>>> I could just use the $HOST variable to define this. >>>> >>>>> On 5 Jun 2016, at 10:41 pm, Sivaram Kannan <[email protected]> wrote: >>>>> >>>>> >>>>> I have been using this way from 0.23.0 to the 0.28.0. This has been >>>>> surely working (although for a different framework). Inside the docker >>>>> container can you see the $HOST variable defined?? >>>>> >>>>> The ticket you referred says that the apps definition needs to define >>>>> LIBPROCESS_HOST=$HOST to be make the framework take the proper IP - you >>>>> are describing a different problem. >>>>> >>>>> Thanks, >>>>> ./Siva. >>>>> >>>>>> On Sun, Jun 5, 2016 at 4:30 AM, Eli Jordan <[email protected]> >>>>>> wrote: >>>>>> I found this issue on the mesos jira that describes the exact issue I am >>>>>> hitting. >>>>>> >>>>>> https://issues.apache.org/jira/browse/MESOS-3740 >>>>>> >>>>>> It doesn't appear to be resolved. >>>>>> >>>>>> Thanks >>>>>> Eli >>>>>> >>>>>> On 5 Jun 2016, at 16:46, Eli Jordan <[email protected]> wrote: >>>>>> >>>>>>> Hmmm… that doesn’t seem to work for me. What version of mesos does this >>>>>>> work in? I am running 0.27.1. >>>>>>> >>>>>>> When using this approach, I still get the following error when the >>>>>>> kafka mess framework is starting up. >>>>>>> >>>>>>> "Scheduler driver bound to loopback interface! Cannot communicate with >>>>>>> remote master(s). You might want to set 'LIBPROCESS_IP' environment >>>>>>> variable to use a routable IP address.” >>>>>>> >>>>>>> I tried setting LIBPROCESS_IP to ‘0.0.0.0’ and >>>>>>> LIBPROCESS_ADVERTISE_IP=‘the public ip’ and this works. But the host >>>>>>> variations don’t seem to work. (i.e. set LIBPROCESS_IP=0.0.0.0 and >>>>>>> LIBPROCESS_ADVERTISE_HOST=$HOST) >>>>>>> >>>>>>> It seems lib process doesn’t support using host names. >>>>>>> >>>>>>> I think I might have to run the framework outside of docker, but I >>>>>>> would really like to avoid this. >>>>>>> >>>>>>> This problem would be solved if the docker executor was able to set the >>>>>>> same environment variables as the command executor. Is there a way to >>>>>>> make this happen? >>>>>>> >>>>>>> I saw that mesos can be extended with a Hook ‘module’ to set extra >>>>>>> environment variables in docker containers. This might be a solution, >>>>>>> but seems over wrought for a simple problem. >>>>>>> >>>>>>> >>>>>>>> On 5 Jun 2016, at 12:50 am, Sivaram Kannan <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Can you try adding && after the LIBPROCESS_HOST variable and the >>>>>>>> actual command. We have been using this for sometime now. >>>>>>>> >>>>>>>> "cmd": "LIBPROCESS_HOST=$HOST && ./kafka-mesos.sh .. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> ./Siva. >>>>>>>> >>>>>>>> >>>>>>>>> On Sat, Jun 4, 2016 at 8:34 AM, Eli Jordan <[email protected]> >>>>>>>>> wrote: >>>>>>>>> Hi @haosdent >>>>>>>>> >>>>>>>>> Based on my testing, this is not the case. >>>>>>>>> >>>>>>>>> I ran a task (from marathon) without using a docker container that >>>>>>>>> just printed out all environment variables. i.e. while [ true ]; do >>>>>>>>> env; sleep 2; done >>>>>>>>> >>>>>>>>> I then run a task that executed the same command inside an alpine >>>>>>>>> docker image. >>>>>>>>> >>>>>>>>> When running without a docker image LIBPROCESS_IP was defined along >>>>>>>>> with many other variables. >>>>>>>>> >>>>>>>>> Sample output when running without docker (note LIBPROCESS_IP) is >>>>>>>>> defined >>>>>>>>> >>>>>>>>> Registered executor on mesos-slave0 >>>>>>>>> Starting task plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e >>>>>>>>> sh -c 'while [ true ]; do env; sleep 2; done' >>>>>>>>> Forked command at 16571 >>>>>>>>> LIBPROCESS_IP=192.168.3.16 >>>>>>>>> MESOS_AGENT_ENDPOINT=192.168.3.16:5051 >>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins >>>>>>>>> HOST=mesos-slave0 >>>>>>>>> SHELL=/bin/sh >>>>>>>>> MESOS_DIRECTORY=/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e/runs/c9b6ef86-b37d-4e3c-b1ca-bd680aed779f >>>>>>>>> PORT0=31082 >>>>>>>>> PORT_10001=31082 >>>>>>>>> LC_ALL=en_US.UTF-8 >>>>>>>>> … more >>>>>>>>> >>>>>>>>> >>>>>>>>> Sample output when running with docker (note LIBPROCESS_IP is not >>>>>>>>> defined) >>>>>>>>> >>>>>>>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" >>>>>>>>> --help="false" --initialize_driver_logging="true" >>>>>>>>> --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" >>>>>>>>> --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" >>>>>>>>> --quiet="false" >>>>>>>>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>>>> --stop_timeout="0ns" >>>>>>>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" >>>>>>>>> --help="false" --initialize_driver_logging="true" >>>>>>>>> --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" >>>>>>>>> --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" >>>>>>>>> --quiet="false" >>>>>>>>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>>>> --stop_timeout="0ns" >>>>>>>>> Registered docker executor on mesos-slave0 >>>>>>>>> Starting task alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>>>>>>>> HOSTNAME=984809b0b720 >>>>>>>>> SHLVL=1 >>>>>>>>> HOME=/root >>>>>>>>> PORT=31295 >>>>>>>>> MESOS_CONTAINER_NAME=mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42 >>>>>>>>> MARATHON_APP_ID=/alpine-test >>>>>>>>> PORTS=31295 >>>>>>>>> PORT0=31295 >>>>>>>>> MARATHON_APP_DOCKER_IMAGE=alpine >>>>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>>>>>>> MARATHON_APP_RESOURCE_DISK=0.0 >>>>>>>>> MARATHON_APP_RESOURCE_MEM=128.0 >>>>>>>>> HOST=mesos-slave0 >>>>>>>>> PORT_10000=31295 >>>>>>>>> MARATHON_APP_VERSION=2016-05-30T08:56:59.065Z >>>>>>>>> MARATHON_APP_LABELS= >>>>>>>>> PWD=/ >>>>>>>>> MESOS_TASK_ID=alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>>>>>>>> MARATHON_APP_RESOURCE_CPUS=1.0 >>>>>>>>> >>>>>>>>> Is there some other config I need to do for the docker containerizer? >>>>>>>>> Any help greatly appreciated. >>>>>>>>> >>>>>>>>>> On 4 Jun 2016, at 7:28 pm, haosdent <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Hi, @Jordan. I think not matter you use MesosContainerizer or >>>>>>>>>> DockerContainerizer, LIBPROCESS_IP always would be set if you launch >>>>>>>>>> you Agent with `--ip` flag. >>>>>>>>>> >>>>>>>>>>> On Fri, Jun 3, 2016 at 8:23 PM, Eli Jordan >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> The reason I need to set LIBPROCESS_IP is because the slaves have 2 >>>>>>>>>>> network interfaces, and the docker container is running in host >>>>>>>>>>> networking mode. So libmesos doesn’t know which IP to advertise. >>>>>>>>>>> The hostnames of the slaves are all resolvable. >>>>>>>>>>> >>>>>>>>>>> I have noticed that if I run a marathon app that doesn’t use >>>>>>>>>>> docker, e.g. while [ true ]; do env; sleep 2; done, that >>>>>>>>>>> LIBPROCESS_IP is defined in the environment. However, when running >>>>>>>>>>> a docker image this variable is not defined. >>>>>>>>>>> >>>>>>>>>>> Is there a way to have marathon pass along all environment >>>>>>>>>>> variables defined by mesos? >>>>>>>>>>> Thanks >>>>>>>>>>> Eli >>>>>>>>>>> >>>>>>>>>>> On 4 Apr 2016, at 14:12, Eli Jordan <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> @haosdent I’m not sure how this works internally, but it seems the >>>>>>>>>>>> mess master needs to send requests to the framework for resource >>>>>>>>>>>> offers, and therefore needs to know the external IP (i.e. the host >>>>>>>>>>>> IP) >>>>>>>>>>>> >>>>>>>>>>>> @craig w >>>>>>>>>>>> Would I need to do this in the cmd portion of the marathon JSON? I >>>>>>>>>>>> currently have the config below >>>>>>>>>>>> { >>>>>>>>>>>> "container": ..., >>>>>>>>>>>> "id":"kafka-mesos-scheduler", >>>>>>>>>>>> "cpus": 0.5, >>>>>>>>>>>> "mem": 256, >>>>>>>>>>>> "ports": [9999], >>>>>>>>>>>> "cmd": "./kafka-mesos.sh scheduler --master=mesos-master:5050 >>>>>>>>>>>> --zk=mesos-master:2181 --api=http://mesos-slave0:9999 >>>>>>>>>>>> --storage=zk:/kafka-mesos", >>>>>>>>>>>> "instances": 1, >>>>>>>>>>>> "constraints": [["hostname", "LIKE", "mesos-slave0"]], >>>>>>>>>>>> "env": { >>>>>>>>>>>> "LIBPROCESS_IP": "192.168.3.16" >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> @Chris Baker Currently we don’t have mess-dns setup but if this >>>>>>>>>>>> works it would seem to be a nice solution. However, I did try >>>>>>>>>>>> setting LIBPROCESS_IP to the slave hostname and it seems to >>>>>>>>>>>> produce a parse error. So I think it needs to be an actual IP >>>>>>>>>>>> address. >>>>>>>>>>>> >>>>>>>>>>>> I was hoping there would be a configuration for the slave that >>>>>>>>>>>> would automatically populate this env variable when starting the >>>>>>>>>>>> docker container. So I don’t need to complicate the marathon file, >>>>>>>>>>>> and can reuse it in different clusters. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 4 Apr 2016, at 11:25 am, Chris Baker <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Alternatively, because the $HOST username is indirect, which >>>>>>>>>>>>> would require a runtime element to "export $LIBPROCESS_IP=$HOST", >>>>>>>>>>>>> another alternative is to fallback on Mesos-DNS, if that's part >>>>>>>>>>>>> of the cluster deployment, setting $LIBPROCESS_IP to the (a >>>>>>>>>>>>> priori) Mesos-DNS entry corresponding to the service. >>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Apr 3, 2016 at 5:06 PM craig w <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Hi, marathon sets the HOST env var. If it's not the ip address >>>>>>>>>>>>>> you can use getent with the value from HOST to figure it out. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >However, in order for the frameworks to receive resource offers >>>>>>>>>>>>>> >I need to set the LIBPROCESS_IP environment variable to the >>>>>>>>>>>>>> >hosts IP address for the docker container running the >>>>>>>>>>>>>> >frameworks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, @Gmail. Could you provide more details about this? >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sun, Apr 3, 2016 at 10:40 PM, Rad Gruchalski >>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> Hi Gmail, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> AFAIK not. The only way to do so is setting up the env variable >>>>>>>>>>>>>>> as you do now. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Radek Gruchalski >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> de.linkedin.com/in/radgruchalski/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Confidentiality: >>>>>>>>>>>>>>> This communication is intended for the above-named person and >>>>>>>>>>>>>>> may be confidential and/or legally privileged. >>>>>>>>>>>>>>> If it has come to you in error you must take no action based on >>>>>>>>>>>>>>> it, nor must you copy or show it to anyone; please >>>>>>>>>>>>>>> delete/destroy and inform the sender immediately. >>>>>>>>>>>>>>>> On Sunday, 3 April 2016 at 16:09, Gmail wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm pretty new to mesos and marathon, and I'm running a couple >>>>>>>>>>>>>>>> of frameworks with marathon (Kafka and elastic search). >>>>>>>>>>>>>>>> However, in order for the frameworks to receive resource >>>>>>>>>>>>>>>> offers I need to set the LIBPROCESS_IP environment variable to >>>>>>>>>>>>>>>> the hosts IP address for the docker container running the >>>>>>>>>>>>>>>> frameworks. Currently I am working around me this by using a >>>>>>>>>>>>>>>> constraint to hard wire the slave that the framework gets >>>>>>>>>>>>>>>> launched on, so then I can put the slaves ip in the marathon >>>>>>>>>>>>>>>> json file. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Obviously this is not ideal. Is there a better way to define >>>>>>>>>>>>>>>> the host ip Inside the docker container? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent from my iPad >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Haosdent Huang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Regards, >>>>>>>>>> Haosdent Huang >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ever tried. ever failed. no matter. >>>>>>>> try again. fail again. fail better. >>>>>>>> -- Samuel Beckett >>>>> >>>>> >>>>> >>>>> -- >>>>> ever tried. ever failed. no matter. >>>>> try again. fail again. fail better. >>>>> -- Samuel Beckett
