Thanks Radoslaw. I'm not really set on using host names, I just want a reliable way to start the framework. For the meantime I have gone with a solution similar to what you suggested. We use /etc/default/mesos file to configure mesos, and it has the ip defined, so I just mounted that in the container and read the ip.
I would like to avoid having a dependency on the file system of the agents though. I'm not sure why I can't have the docket executor set the LIBPROCESS_IP variable in the same way the command executor does. Thanks Eli > On 6 Jun 2016, at 21:44, Radoslaw Gruchalski <[email protected]> wrote: > > Out of curiosity. Why are you insisting on using host names? > Say you have 1 master and 2 agents with these IPs: > > - mesos-master-0: 10.100.1.10 > - mesos-agent-0: 10.100.1.11 > - mesos-agent-1: 10.100.1.12 > > Your problem is that you have no way to obtain an IP address of the agent in > the container. Correct? > One way to overcome this problem is to create a shell file, say in > /etc/mesos-agent.sh, with contents like: > > ... > AGENT_IP=10.100.1.11 > … > > If you’re using Marathon, you can copy that file to the sandbox using docker > volumes: > > { > "containerPath": “/etc/mesos-agent.sh", > "hostPath": "/etc/mesos-agent.sh", > "mode": "RO" > } > > You can now source that in the container to set the LIBPROCESS_ADVERTISE_IP. > Other applications simply use the mesos-agent-X host name. That’s without > mesos-dns. > Things are easier with mesos-dns or consul service catalog (I prefer the > latter). > – > Best regards, > > Radek Gruchalski > > [email protected] > de.linkedin.com/in/radgruchalski > > Confidentiality: > This communication is intended for the above-named person and may be > confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor must > you copy or show it to anyone; please delete/destroy and inform the sender > immediately. > >> On June 6, 2016 at 1:16:07 PM, Eli Jordan ([email protected]) wrote: >> >> The issue refers to LIBPROCESS_IP not LIBPROCESS_HOST. I haven’t been able >> to find the LIBPROCESS_HOST variable documented anywhere. >> >> My understanding is that the scheduler uses LIBPROCESS_IP to determine which >> network interface to bind to, and also which ip to advertise to the master, >> so that the master can send offers. There is also another variable >> LIBPROCESS_ADVERTISE_IP. If this is defined then LIBPROCESS_IP is used to >> determine which network interface to bind to, and LIBPROCESS_ADVERTISE_IP is >> used to determine which ip to advertise to the master. >> >> It would be great if there was a LIBPROCESS_ADVERTISE_HOST variable, then I >> could just use the $HOST variable to define this. >> >>> On 5 Jun 2016, at 10:41 pm, Sivaram Kannan <[email protected]> wrote: >>> >>> >>> I have been using this way from 0.23.0 to the 0.28.0. This has been surely >>> working (although for a different framework). Inside the docker container >>> can you see the $HOST variable defined?? >>> >>> The ticket you referred says that the apps definition needs to define >>> LIBPROCESS_HOST=$HOST to be make the framework take the proper IP - you are >>> describing a different problem. >>> >>> Thanks, >>> ./Siva. >>> >>>> On Sun, Jun 5, 2016 at 4:30 AM, Eli Jordan <[email protected]> >>>> wrote: >>>> I found this issue on the mesos jira that describes the exact issue I am >>>> hitting. >>>> >>>> https://issues.apache.org/jira/browse/MESOS-3740 >>>> >>>> It doesn't appear to be resolved. >>>> >>>> Thanks >>>> Eli >>>> >>>> On 5 Jun 2016, at 16:46, Eli Jordan <[email protected]> wrote: >>>> >>>>> Hmmm… that doesn’t seem to work for me. What version of mesos does this >>>>> work in? I am running 0.27.1. >>>>> >>>>> When using this approach, I still get the following error when the kafka >>>>> mess framework is starting up. >>>>> >>>>> "Scheduler driver bound to loopback interface! Cannot communicate with >>>>> remote master(s). You might want to set 'LIBPROCESS_IP' environment >>>>> variable to use a routable IP address.” >>>>> >>>>> I tried setting LIBPROCESS_IP to ‘0.0.0.0’ and >>>>> LIBPROCESS_ADVERTISE_IP=‘the public ip’ and this works. But the host >>>>> variations don’t seem to work. (i.e. set LIBPROCESS_IP=0.0.0.0 and >>>>> LIBPROCESS_ADVERTISE_HOST=$HOST) >>>>> >>>>> It seems lib process doesn’t support using host names. >>>>> >>>>> I think I might have to run the framework outside of docker, but I would >>>>> really like to avoid this. >>>>> >>>>> This problem would be solved if the docker executor was able to set the >>>>> same environment variables as the command executor. Is there a way to >>>>> make this happen? >>>>> >>>>> I saw that mesos can be extended with a Hook ‘module’ to set extra >>>>> environment variables in docker containers. This might be a solution, but >>>>> seems over wrought for a simple problem. >>>>> >>>>> >>>>>> On 5 Jun 2016, at 12:50 am, Sivaram Kannan <[email protected]> wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> Can you try adding && after the LIBPROCESS_HOST variable and the actual >>>>>> command. We have been using this for sometime now. >>>>>> >>>>>> "cmd": "LIBPROCESS_HOST=$HOST && ./kafka-mesos.sh .. >>>>>> >>>>>> Thanks, >>>>>> ./Siva. >>>>>> >>>>>> >>>>>>> On Sat, Jun 4, 2016 at 8:34 AM, Eli Jordan <[email protected]> >>>>>>> wrote: >>>>>>> Hi @haosdent >>>>>>> >>>>>>> Based on my testing, this is not the case. >>>>>>> >>>>>>> I ran a task (from marathon) without using a docker container that just >>>>>>> printed out all environment variables. i.e. while [ true ]; do env; >>>>>>> sleep 2; done >>>>>>> >>>>>>> I then run a task that executed the same command inside an alpine >>>>>>> docker image. >>>>>>> >>>>>>> When running without a docker image LIBPROCESS_IP was defined along >>>>>>> with many other variables. >>>>>>> >>>>>>> Sample output when running without docker (note LIBPROCESS_IP) is >>>>>>> defined >>>>>>> >>>>>>> Registered executor on mesos-slave0 >>>>>>> Starting task plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e >>>>>>> sh -c 'while [ true ]; do env; sleep 2; done' >>>>>>> Forked command at 16571 >>>>>>> LIBPROCESS_IP=192.168.3.16 >>>>>>> MESOS_AGENT_ENDPOINT=192.168.3.16:5051 >>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins >>>>>>> HOST=mesos-slave0 >>>>>>> SHELL=/bin/sh >>>>>>> MESOS_DIRECTORY=/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e/runs/c9b6ef86-b37d-4e3c-b1ca-bd680aed779f >>>>>>> PORT0=31082 >>>>>>> PORT_10001=31082 >>>>>>> LC_ALL=en_US.UTF-8 >>>>>>> … more >>>>>>> >>>>>>> >>>>>>> Sample output when running with docker (note LIBPROCESS_IP is not >>>>>>> defined) >>>>>>> >>>>>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" >>>>>>> --help="false" --initialize_driver_logging="true" >>>>>>> --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" >>>>>>> --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" >>>>>>> --quiet="false" >>>>>>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>> --stop_timeout="0ns" >>>>>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" >>>>>>> --help="false" --initialize_driver_logging="true" >>>>>>> --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" >>>>>>> --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" >>>>>>> --quiet="false" >>>>>>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>>>>> --stop_timeout="0ns" >>>>>>> Registered docker executor on mesos-slave0 >>>>>>> Starting task alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>>>>>> HOSTNAME=984809b0b720 >>>>>>> SHLVL=1 >>>>>>> HOME=/root >>>>>>> PORT=31295 >>>>>>> MESOS_CONTAINER_NAME=mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42 >>>>>>> MARATHON_APP_ID=/alpine-test >>>>>>> PORTS=31295 >>>>>>> PORT0=31295 >>>>>>> MARATHON_APP_DOCKER_IMAGE=alpine >>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>>>>> MARATHON_APP_RESOURCE_DISK=0.0 >>>>>>> MARATHON_APP_RESOURCE_MEM=128.0 >>>>>>> HOST=mesos-slave0 >>>>>>> PORT_10000=31295 >>>>>>> MARATHON_APP_VERSION=2016-05-30T08:56:59.065Z >>>>>>> MARATHON_APP_LABELS= >>>>>>> PWD=/ >>>>>>> MESOS_TASK_ID=alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>>>>>> MARATHON_APP_RESOURCE_CPUS=1.0 >>>>>>> >>>>>>> Is there some other config I need to do for the docker containerizer? >>>>>>> Any help greatly appreciated. >>>>>>> >>>>>>>> On 4 Jun 2016, at 7:28 pm, haosdent <[email protected]> wrote: >>>>>>>> >>>>>>>> Hi, @Jordan. I think not matter you use MesosContainerizer or >>>>>>>> DockerContainerizer, LIBPROCESS_IP always would be set if you launch >>>>>>>> you Agent with `--ip` flag. >>>>>>>> >>>>>>>>> On Fri, Jun 3, 2016 at 8:23 PM, Eli Jordan <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> The reason I need to set LIBPROCESS_IP is because the slaves have 2 >>>>>>>>> network interfaces, and the docker container is running in host >>>>>>>>> networking mode. So libmesos doesn’t know which IP to advertise. The >>>>>>>>> hostnames of the slaves are all resolvable. >>>>>>>>> >>>>>>>>> I have noticed that if I run a marathon app that doesn’t use docker, >>>>>>>>> e.g. while [ true ]; do env; sleep 2; done, that LIBPROCESS_IP is >>>>>>>>> defined in the environment. However, when running a docker image this >>>>>>>>> variable is not defined. >>>>>>>>> >>>>>>>>> Is there a way to have marathon pass along all environment variables >>>>>>>>> defined by mesos? >>>>>>>>> Thanks >>>>>>>>> Eli >>>>>>>>> >>>>>>>>> On 4 Apr 2016, at 14:12, Eli Jordan <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> @haosdent I’m not sure how this works internally, but it seems the >>>>>>>>>> mess master needs to send requests to the framework for resource >>>>>>>>>> offers, and therefore needs to know the external IP (i.e. the host >>>>>>>>>> IP) >>>>>>>>>> >>>>>>>>>> @craig w >>>>>>>>>> Would I need to do this in the cmd portion of the marathon JSON? I >>>>>>>>>> currently have the config below >>>>>>>>>> { >>>>>>>>>> "container": ..., >>>>>>>>>> "id":"kafka-mesos-scheduler", >>>>>>>>>> "cpus": 0.5, >>>>>>>>>> "mem": 256, >>>>>>>>>> "ports": [9999], >>>>>>>>>> "cmd": "./kafka-mesos.sh scheduler --master=mesos-master:5050 >>>>>>>>>> --zk=mesos-master:2181 --api=http://mesos-slave0:9999 >>>>>>>>>> --storage=zk:/kafka-mesos", >>>>>>>>>> "instances": 1, >>>>>>>>>> "constraints": [["hostname", "LIKE", "mesos-slave0"]], >>>>>>>>>> "env": { >>>>>>>>>> "LIBPROCESS_IP": "192.168.3.16" >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> @Chris Baker Currently we don’t have mess-dns setup but if this >>>>>>>>>> works it would seem to be a nice solution. However, I did try >>>>>>>>>> setting LIBPROCESS_IP to the slave hostname and it seems to produce >>>>>>>>>> a parse error. So I think it needs to be an actual IP address. >>>>>>>>>> >>>>>>>>>> I was hoping there would be a configuration for the slave that would >>>>>>>>>> automatically populate this env variable when starting the docker >>>>>>>>>> container. So I don’t need to complicate the marathon file, and can >>>>>>>>>> reuse it in different clusters. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 4 Apr 2016, at 11:25 am, Chris Baker <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Alternatively, because the $HOST username is indirect, which would >>>>>>>>>>> require a runtime element to "export $LIBPROCESS_IP=$HOST", another >>>>>>>>>>> alternative is to fallback on Mesos-DNS, if that's part of the >>>>>>>>>>> cluster deployment, setting $LIBPROCESS_IP to the (a priori) >>>>>>>>>>> Mesos-DNS entry corresponding to the service. >>>>>>>>>>> >>>>>>>>>>>> On Sun, Apr 3, 2016 at 5:06 PM craig w <[email protected]> wrote: >>>>>>>>>>>> Hi, marathon sets the HOST env var. If it's not the ip address you >>>>>>>>>>>> can use getent with the value from HOST to figure it out. >>>>>>>>>>>> >>>>>>>>>>>> >However, in order for the frameworks to receive resource offers I >>>>>>>>>>>> >need to set the LIBPROCESS_IP environment variable to the hosts >>>>>>>>>>>> >IP address for the docker container running the frameworks. >>>>>>>>>>>> >>>>>>>>>>>> Hi, @Gmail. Could you provide more details about this? >>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Apr 3, 2016 at 10:40 PM, Rad Gruchalski >>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>> Hi Gmail, >>>>>>>>>>>>> >>>>>>>>>>>>> AFAIK not. The only way to do so is setting up the env variable >>>>>>>>>>>>> as you do now. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Radek Gruchalski >>>>>>>>>>>>> >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> de.linkedin.com/in/radgruchalski/ >>>>>>>>>>>>> >>>>>>>>>>>>> Confidentiality: >>>>>>>>>>>>> This communication is intended for the above-named person and may >>>>>>>>>>>>> be confidential and/or legally privileged. >>>>>>>>>>>>> If it has come to you in error you must take no action based on >>>>>>>>>>>>> it, nor must you copy or show it to anyone; please delete/destroy >>>>>>>>>>>>> and inform the sender immediately. >>>>>>>>>>>>>> On Sunday, 3 April 2016 at 16:09, Gmail wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm pretty new to mesos and marathon, and I'm running a couple >>>>>>>>>>>>>> of frameworks with marathon (Kafka and elastic search). However, >>>>>>>>>>>>>> in order for the frameworks to receive resource offers I need to >>>>>>>>>>>>>> set the LIBPROCESS_IP environment variable to the hosts IP >>>>>>>>>>>>>> address for the docker container running the frameworks. >>>>>>>>>>>>>> Currently I am working around me this by using a constraint to >>>>>>>>>>>>>> hard wire the slave that the framework gets launched on, so then >>>>>>>>>>>>>> I can put the slaves ip in the marathon json file. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Obviously this is not ideal. Is there a better way to define the >>>>>>>>>>>>>> host ip Inside the docker container? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent from my iPad >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> Haosdent Huang >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Regards, >>>>>>>> Haosdent Huang >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ever tried. ever failed. no matter. >>>>>> try again. fail again. fail better. >>>>>> -- Samuel Beckett >>> >>> >>> >>> -- >>> ever tried. ever failed. no matter. >>> try again. fail again. fail better. >>> -- Samuel Beckett
