Actually not only LIBPROCESS_IP, some SSL related environment didn't pass as well. @Cody @Tim, is it OK that pass flags.executor_environment_variables to the Docker container?
[1]: https://issues.apache.org/jira/browse/MESOS-3740 On Tue, Jun 7, 2016 at 9:25 AM, Radoslaw Gruchalski <[email protected]> wrote: > Yes, because that runs in host network. This leads to a question: your > docker task, is it bridge or host network. > > -- > Best regards, > Rad > > > > > On Tue, Jun 7, 2016 at 3:21 AM +0200, "Eli Jordan" < > [email protected]> wrote: > > It's important to note that if you run a task with the command executor >> (I.e. Not using docker) LIBPROCESS_IP is defined, along with several other >> variables that are not defined in docker. >> >> Thanks >> Eli >> >> On 7 Jun 2016, at 10:05, Radoslaw Gruchalski <[email protected]> >> wrote: >> >> I think the problem is that it is not known which agent the task is >> running on until the task i in the running state. >> Hence the master can’t pass that as an env variable to the task. >> However, I see your point. There is an agent host name avaialble in the >> task as $HOST. Maybe it would be a good idea if Mesos was setting env >> variables named AGENT_IP_0, AGENT_IP_1 and so on for every IP interface on >> the agent, maybe AGENT_BIND_IP if bind IP is different than 0.0.0.0. OTOH, >> I can see how this could be considered as some security issue. I am not >> sure what the implications could be. >> >> Anybody else care to comment? >> >> – >> Best regards, >> Radek Gruchalski >> [email protected] >> de.linkedin.com/in/radgruchalski >> >> >> *Confidentiality:*This communication is intended for the above-named >> person and may be confidential and/or legally privileged. >> If it has come to you in error you must take no action based on it, nor >> must you copy or show it to anyone; please delete/destroy and inform the >> sender immediately. >> >> On June 7, 2016 at 1:42:46 AM, Eli Jordan ([email protected]) >> wrote: >> >> Thanks Radoslaw. I'm not really set on using host names, I just want a >> reliable way to start the framework. For the meantime I have gone with a >> solution similar to what you suggested. We use /etc/default/mesos file to >> configure mesos, and it has the ip defined, so I just mounted that in the >> container and read the ip. >> >> I would like to avoid having a dependency on the file system of the >> agents though. I'm not sure why I can't have the docket executor set the >> LIBPROCESS_IP variable in the same way the command executor does. >> >> Thanks >> Eli >> >> On 6 Jun 2016, at 21:44, Radoslaw Gruchalski <[email protected]> >> wrote: >> >> Out of curiosity. Why are you insisting on using host names? >> Say you have 1 master and 2 agents with these IPs: >> >> - mesos-master-0: 10.100.1.10 >> - mesos-agent-0: 10.100.1.11 >> - mesos-agent-1: 10.100.1.12 >> >> Your problem is that you have no way to obtain an IP address of the agent >> in the container. Correct? >> One way to overcome this problem is to create a shell file, say in >> /etc/mesos-agent.sh, with contents like: >> >> ... >> AGENT_IP=10.100.1.11 >> … >> >> If you’re using Marathon, you can copy that file to the sandbox using >> docker volumes: >> >> { >> "containerPath": “/etc/mesos-agent.sh", >> "hostPath": "/etc/mesos-agent.sh", >> "mode": "RO" >> } >> >> You can now source that in the container to set the >> LIBPROCESS_ADVERTISE_IP. >> Other applications simply use the mesos-agent-X host name. That’s without >> mesos-dns. >> Things are easier with mesos-dns or consul service catalog (I prefer the >> latter). >> >> – >> Best regards, >> Radek Gruchalski >> [email protected] >> de.linkedin.com/in/radgruchalski >> >> >> *Confidentiality:*This communication is intended for the above-named >> person and may be confidential and/or legally privileged. >> If it has come to you in error you must take no action based on it, nor >> must you copy or show it to anyone; please delete/destroy and inform the >> sender immediately. >> >> On June 6, 2016 at 1:16:07 PM, Eli Jordan ([email protected]) >> wrote: >> >> The issue refers to LIBPROCESS_IP not LIBPROCESS_HOST. I haven’t been >> able to find the LIBPROCESS_HOST variable documented anywhere. >> >> My understanding is that the scheduler uses LIBPROCESS_IP to determine >> which network interface to bind to, and also which ip to advertise to the >> master, so that the master can send offers. There is also another variable >> LIBPROCESS_ADVERTISE_IP. If this is defined then LIBPROCESS_IP is used to >> determine which network interface to bind to, and LIBPROCESS_ADVERTISE_IP >> is used to determine which ip to advertise to the master. >> >> It would be great if there was a LIBPROCESS_ADVERTISE_HOST variable, then >> I could just use the $HOST variable to define this. >> >> On 5 Jun 2016, at 10:41 pm, Sivaram Kannan <[email protected]> wrote: >> >> >> I have been using this way from 0.23.0 to the 0.28.0. This has been >> surely working (although for a different framework). Inside the docker >> container can you see the $HOST variable defined?? >> >> The ticket you referred says that the apps definition needs to define >> LIBPROCESS_HOST=$HOST to be make the framework take the proper IP - you are >> describing a different problem. >> >> Thanks, >> ./Siva. >> >> On Sun, Jun 5, 2016 at 4:30 AM, Eli Jordan <[email protected]> >> wrote: >> >>> I found this issue on the mesos jira that describes the exact issue I am >>> hitting. >>> >>> https://issues.apache.org/jira/browse/MESOS-3740 >>> >>> It doesn't appear to be resolved. >>> >>> Thanks >>> Eli >>> >>> On 5 Jun 2016, at 16:46, Eli Jordan <[email protected]> wrote: >>> >>> Hmmm… that doesn’t seem to work for me. What version of mesos does this >>> work in? I am running 0.27.1. >>> >>> When using this approach, I still get the following error when the kafka >>> mess framework is starting up. >>> >>> "Scheduler driver bound to loopback interface! Cannot communicate with >>> remote master(s). You might want to set 'LIBPROCESS_IP' environment >>> variable to use a routable IP address.” >>> >>> I tried setting LIBPROCESS_IP to ‘0.0.0.0’ and >>> LIBPROCESS_ADVERTISE_IP=‘the public ip’ and this works. But the host >>> variations don’t seem to work. (i.e. set LIBPROCESS_IP=0.0.0.0 and >>> LIBPROCESS_ADVERTISE_HOST=$HOST) >>> >>> It seems lib process doesn’t support using host names. >>> >>> I think I might have to run the framework outside of docker, but I would >>> really like to avoid this. >>> >>> This problem would be solved if the docker executor was able to set the >>> same environment variables as the command executor. Is there a way to make >>> this happen? >>> >>> I saw that mesos can be extended with a Hook ‘module’ to set extra >>> environment variables in docker containers. This might be a solution, but >>> seems over wrought for a simple problem. >>> >>> >>> On 5 Jun 2016, at 12:50 am, Sivaram Kannan <[email protected]> wrote: >>> >>> >>> Hi, >>> >>> Can you try adding && after the LIBPROCESS_HOST variable and the actual >>> command. We have been using this for sometime now. >>> >>> "cmd": "LIBPROCESS_HOST=$HOST && ./kafka-mesos.sh .. >>> >>> Thanks, >>> ./Siva. >>> >>> >>> On Sat, Jun 4, 2016 at 8:34 AM, Eli Jordan <[email protected]> >>> wrote: >>> >>>> Hi @haosdent >>>> >>>> Based on my testing, this is not the case. >>>> >>>> I ran a task (from marathon) without using a docker container that just >>>> printed out all environment variables. i.e. while [ true ]; do env; sleep >>>> 2; done >>>> >>>> I then run a task that executed the same command inside an alpine >>>> docker image. >>>> >>>> When running without a docker image LIBPROCESS_IP was defined along >>>> with many other variables. >>>> >>>> Sample output when running without docker (note LIBPROCESS_IP) is >>>> defined >>>> >>>> Registered executor on mesos-slave0 >>>> Starting task plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e >>>> sh -c 'while [ true ]; do env; sleep 2; done' >>>> Forked command at 16571 >>>> LIBPROCESS_IP=192.168.3.16 >>>> MESOS_AGENT_ENDPOINT=192.168.3.16:5051 >>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins >>>> HOST=mesos-slave0 >>>> SHELL=/bin/sh >>>> >>>> MESOS_DIRECTORY=/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e/runs/c9b6ef86-b37d-4e3c-b1ca-bd680aed779f >>>> PORT0=31082 >>>> PORT_10001=31082 >>>> LC_ALL=en_US.UTF-8 >>>> … more >>>> >>>> >>>> Sample output when running with docker (note LIBPROCESS_IP is not >>>> defined) >>>> >>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>> --logbufsecs="0" --logging_level="INFO" >>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>> --stop_timeout="0ns" >>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>> --logbufsecs="0" --logging_level="INFO" >>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>> --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42" >>>> --stop_timeout="0ns" >>>> Registered docker executor on mesos-slave0 >>>> Starting task alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>>> HOSTNAME=984809b0b720 >>>> SHLVL=1 >>>> HOME=/root >>>> PORT=31295 >>>> >>>> MESOS_CONTAINER_NAME=mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42 >>>> MARATHON_APP_ID=/alpine-test >>>> PORTS=31295 >>>> PORT0=31295 >>>> MARATHON_APP_DOCKER_IMAGE=alpine >>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>> MARATHON_APP_RESOURCE_DISK=0.0 >>>> MARATHON_APP_RESOURCE_MEM=128.0 >>>> HOST=mesos-slave0 >>>> PORT_10000=31295 >>>> MARATHON_APP_VERSION=2016-05-30T08:56:59.065Z >>>> MARATHON_APP_LABELS= >>>> PWD=/ >>>> MESOS_TASK_ID=alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e >>>> MARATHON_APP_RESOURCE_CPUS=1.0 >>>> >>>> Is there some other config I need to do for the docker containerizer? >>>> Any help greatly appreciated. >>>> >>>> On 4 Jun 2016, at 7:28 pm, haosdent <[email protected]> wrote: >>>> >>>> Hi, @Jordan. I think not matter you use MesosContainerizer or >>>> DockerContainerizer, LIBPROCESS_IP always would be set if you launch you >>>> Agent with `--ip` flag. >>>> >>>> On Fri, Jun 3, 2016 at 8:23 PM, Eli Jordan <[email protected]> >>>> wrote: >>>> >>>>> >>>>> The reason I need to set LIBPROCESS_IP is because the slaves have 2 >>>>> network interfaces, and the docker container is running in host networking >>>>> mode. So libmesos doesn’t know which IP to advertise. The hostnames of the >>>>> slaves are all resolvable. >>>>> >>>>> I have noticed that if I run a marathon app that doesn’t use docker, >>>>> e.g. while [ true ]; do env; sleep 2; done, that LIBPROCESS_IP is defined >>>>> in the environment. However, when running a docker image this variable is >>>>> not defined. >>>>> >>>>> Is there a way to have marathon pass along all environment variables >>>>> defined by mesos? >>>>> Thanks >>>>> Eli >>>>> >>>>> On 4 Apr 2016, at 14:12, Eli Jordan <[email protected]> wrote: >>>>> >>>>> @haosdent I’m not sure how this works internally, but it seems the >>>>> mess master needs to send requests to the framework for resource offers, >>>>> and therefore needs to know the external IP (i.e. the host IP) >>>>> >>>>> @craig w >>>>> Would I need to do this in the cmd portion of the marathon JSON? I >>>>> currently have the config below >>>>> { >>>>> "container": ..., >>>>> "id":"kafka-mesos-scheduler", >>>>> "cpus": 0.5, >>>>> "mem": 256, >>>>> "ports": [9999], >>>>> "cmd": "./kafka-mesos.sh scheduler --master=mesos-master:5050 >>>>> --zk=mesos-master:2181 --api=http://mesos-slave0:9999 >>>>> --storage=zk:/kafka-mesos", >>>>> "instances": 1, >>>>> "constraints": [["hostname", "LIKE", "mesos-slave0"]], >>>>> "env": { >>>>> "LIBPROCESS_IP": "192.168.3.16" >>>>> } >>>>> } >>>>> >>>>> @Chris Baker Currently we don’t have mess-dns setup but if this works >>>>> it would seem to be a nice solution. However, I did try setting >>>>> LIBPROCESS_IP to the slave hostname and it seems to produce a parse error. >>>>> So I think it needs to be an actual IP address. >>>>> >>>>> I was hoping there would be a configuration for the slave that would >>>>> automatically populate this env variable when starting the docker >>>>> container. So I don’t need to complicate the marathon file, and can reuse >>>>> it in different clusters. >>>>> >>>>> >>>>> On 4 Apr 2016, at 11:25 am, Chris Baker <[email protected]> wrote: >>>>> >>>>> Alternatively, because the $HOST username is indirect, which would >>>>> require a runtime element to "export $LIBPROCESS_IP=$HOST", another >>>>> alternative is to fallback on Mesos-DNS, if that's part of the cluster >>>>> deployment, setting $LIBPROCESS_IP to the (a priori) Mesos-DNS entry >>>>> corresponding to the service. >>>>> >>>>> On Sun, Apr 3, 2016 at 5:06 PM craig w <[email protected]> wrote: >>>>> >>>>>> Hi, marathon sets the HOST env var. If it's not the ip address you >>>>>> can use getent with the value from HOST to figure it out. >>>>>> >However, in order for the frameworks to receive resource offers I >>>>>> need to set the LIBPROCESS_IP environment variable to the hosts IP >>>>>> address >>>>>> for the docker container running the frameworks. >>>>>> >>>>>> Hi, @Gmail. Could you provide more details about this? >>>>>> >>>>>> On Sun, Apr 3, 2016 at 10:40 PM, Rad Gruchalski <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hi Gmail, >>>>>>> >>>>>>> AFAIK not. The only way to do so is setting up the env variable as >>>>>>> you do now. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Kind regards, >>>>>>> Radek Gruchalski >>>>>>> [email protected] <[email protected]> >>>>>>> de.linkedin.com/in/radgruchalski/ >>>>>>> >>>>>>> >>>>>>> *Confidentiality:*This communication is intended for the >>>>>>> above-named person and may be confidential and/or legally privileged. >>>>>>> If it has come to you in error you must take no action based on it, >>>>>>> nor must you copy or show it to anyone; please delete/destroy and inform >>>>>>> the sender immediately. >>>>>>> >>>>>>> On Sunday, 3 April 2016 at 16:09, Gmail wrote: >>>>>>> >>>>>>> I'm pretty new to mesos and marathon, and I'm running a couple of >>>>>>> frameworks with marathon (Kafka and elastic search). However, in order >>>>>>> for >>>>>>> the frameworks to receive resource offers I need to set the >>>>>>> LIBPROCESS_IP >>>>>>> environment variable to the hosts IP address for the docker container >>>>>>> running the frameworks. Currently I am working around me this by using a >>>>>>> constraint to hard wire the slave that the framework gets launched on, >>>>>>> so >>>>>>> then I can put the slaves ip in the marathon json file. >>>>>>> >>>>>>> Obviously this is not ideal. Is there a better way to define the >>>>>>> host ip Inside the docker container? >>>>>>> >>>>>>> Sent from my iPad >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Haosdent Huang >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Haosdent Huang >>>> >>>> >>>> >>> >>> >>> -- >>> ever tried. ever failed. no matter. >>> try again. fail again. fail better. >>> -- Samuel Beckett >>> >>> >>> >> >> >> -- >> ever tried. ever failed. no matter. >> try again. fail again. fail better. >> -- Samuel Beckett >> >> >> -- Best Regards, Haosdent Huang
