Sure, but my point what - why would mesosphere not put docker binary in the official docker image? Maintaining my own docker image of anything is the last instrument I use. That's what "official" images are for after all.
On Mon, Mar 14, 2016 at 8:30 PM, Yong Tang <[email protected]> wrote: > One way to avoid map library dependencies of docker between host and > docker containers is to install binaries of docker into the docker > container: > > > https://docs.docker.com/engine/installation/binaries/ > > > and then map /var/run/docker.sock between host and docker containers. In > this way, library dependencies conflicts between host and docker containers > could be mostly avoided. > > > Thanks > Yong > > ------------------------------ > Date: Mon, 14 Mar 2016 18:49:45 -0700 > Subject: Re: running mesos slave in a docker container > From: [email protected] > To: [email protected] > > > Enumerating each and every lib path and dealing with potential conflicts > between host and docker libc, etc - I didn't want to deal with this > option, it's quite bad imho. > > On Mon, Mar 14, 2016 at 6:42 PM, haosdent <[email protected]> wrote: > > >2. --volumes-from > So far DockerContainerizer in Mesos don't support this option. > > >1. What is the best method to point mesos-slave running in a container to > a working > Usually I mount docker binary to container from host. > > ``` > docker run --privileged -d \ > --name=mesos-slave \ > --net=host \ > -p 31000-31300:31000-31300 \ > -p 5051:5051 \ > -v /usr/bin/docker:/bin/docker \ > -v > /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1:/usr/lib/libdevmapper.so.1.02 \ > -v /lib/x86_64-linux-gnu/libpthread.so.0:/lib/libpthread.so.0 \ > -v /usr/lib/x86_64-linux-gnu/libsqlite3.so:/lib/libsqlite3.so.0 \ > -v /lib/x86_64-linux-gnu/libudev.so.1:/lib/libudev.so.1 > -v /var/run/docker.sock:/var/run/docker.sock \ > -v /sys:/sys \ > -v /tmp:/tmp \ > -e MESOS_MASTER=zk://10.10.10.9:2181/mesos \ > -e MESOS_LOG_DIR=/tmp/log \ > -e MESOS_CONTAINERIZERS=docker \ > -e MESOS_LOGGING_LEVEL=INFO \ > -e MESOS_IP=10.10.10.9 \ > -e MESOS_WORK_DIR=/tmp > mesosphere/mesos-slave mesos-slave > ``` > > On Tue, Mar 15, 2016 at 8:47 AM, Yuri Finkelstein <[email protected]> > wrote: > > Since mesosphere distributes images of mesos software in a container ( > https://hub.docker.com/r/mesosphere/mesos-slave/), I decided to try this > option. After trying this with various settings I settled on a > configuration that basically works. But I do see one problem and this is > what this message about. > > To start off, I find it strange that the image does not contain docker > distribution itself. After all, in order to use containnerizer=mesos one > needs to point mesos slave at a docker binary. If I bind-mount docker > binary to container's /usr/local/bin/mesos and use option > --mesos=/usr/local/bin/mesos I run into the problem of dynamic library > dependencies: mesos depends on a bunch of dyanmic libraries: > ====================== > ldd /usr/bin/docker > linux-vdso.so.1 => (0x00007fffaebfe000) > libsystemd-journal.so.0 => /lib/x86_64-linux-gnu/libsystemd-journal.so.0 > (0x00007f0a1458b000) > libapparmor.so.1 => /usr/lib/x86_64-linux-gnu/libapparmor.so.1 > (0x00007f0a1437f000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007f0a14160000) > libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 > (0x00007f0a13f27000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0a13b62000) > ... and many more > =========================== > Mounting */lib/x86_64-linux-gnu/ *in docker is a horrible idea which is > not worth discussing. So I wonder what is the rational behind decision to > not include docker binary into the mesosphere container and how do other > people solve this problem. > > > Here is one solution that I found. I use* docker:dind* but not as running > container but rather as a volume: > > ============================== > docker create --name "docker-proxy" -v > /var/run/docker.sock:/var/run/docker.sock -v /usr/local/bin docker:dind > =============================== > > > This container contains a fully functional docker binary in its > /usr/local/bin, and this is all I need it for. To make the mesos-slave > container see this binary I simply use *--volumes-from* option: > ========== > docker run -d --restart=unless-stopped --volumes-from > "docker-proxy" --docker=/usr/local/bin/docker --containerizers="docker,mesos" > --name > $MESOS_SLAVE $MESOS_SLAVE_IMAGE ... > ========== > > This works like a charm. But, there is the following problem. > In order for mesos-slave to function in this mode, it needs to spawn > executors in docker container as well. For that purpose mesos slave has > option *--docker_mesos_image= *that should be set to the same container > image name that's used to launch mesos slave. If I do this, > --docker_mesos_image="$MESOS_SLAVE_IMAGE" > > I see that every attempt to spawn a task fails because option > *--docker=/usr/local/bin/docker* is apparently injected into the executor > container but the *--volumes-from="docker-proxy"* option is NOT! So, the > executor becomes dysfunctional without that docker binary. > > > So, to summarize, I'm raising 2 questions: > 1. What is the best method to point mesos-slave running in a container to > a working copy of docker binary and make this work such that executor > containers will also inherit visibility of this binary. > 2. If my proposed method based on docker:dind is deemed reasonable in > general, then I wonder whether I should file a Jira to request that in > addition to *--docker_mesos_image* one gets the ability to add additional > settings to the executor container such as *--volumes-from*. This is not > easy to formulate as potentially other similar options may need to be > configured as well. > > > P.S > The full script showing how I launch mesos slave is shown below > > > > > for i in ${MESOS_SLAVE_NODES[*]}; do > eval $(docker-machine env $i) > NODE_IP=$(docker-machine ip $i) > > # mesos-slave requires access to docker binary, but the coctainer image > does not contain it. > # For that reason I'm creating (but not running!) a docker-in-a-docker > container which contains a statically linked version of the docker binary > # in /usr/local/bin. Then, using '--volumes-from' option on the mesos > container I'm making this binary visible > remove_container "docker-proxy" > docker create --name "docker-proxy" -v > /var/run/docker.sock:/var/run/docker.sock -v /usr/local/bin docker:dind > > remove_container $MESOS_SLAVE > log "Starting mesos slave on $i" > > docker run -d --restart=unless-stopped --volumes-from "docker-proxy" > --name $MESOS_SLAVE \ > --net='host' \ > --pid='host' \ > -e "TZ=$TIMEZONE" \ > --privileged \ > -v /sys/fs/cgroup:/host/sys/fs/cgroup \ > $MESOS_SLAVE_IMAGE \ > > --master="zk://$zk/mesos" \ > --advertise_ip=$NODE_IP \ > --ip=$NODE_IP \ > --resources="ports:[8000-9000, 3000-3200]" \ > --cgroups_hierarchy=/host/sys/fs/cgroup \ > --docker=/usr/local/bin/docker \ > --containerizers="docker,mesos" \ > --log_dir=/var/log/mesos \ > --logging_level=INFO \ > --docker_remove_delay=1hrs \ > --gc_delay=2hrs \ > --executor_registration_timeout=5mins > > > # --docker_mesos_image="$MESOS_SLAVE_IMAGE" \ > > > done > > > > > > > > > > > > > -- > Best Regards, > Haosdent Huang > > >
