Hey Aaron, thanks for the info, glad that I helped in some way, shape or form.
Another thing that I'm curious about is how would *we *switch / upgrade between docker runtimes, if possible? For example, I'm pretty sure that we're not ready to go onto 1.11 right after the slew of changes that we had to make for 1.10. Are there any talks/ideas floating around that allow users to choose which distro of docker gets run on the machine (assuming that we can only choose distros that are compatible with the version of CoreOS that we're on)? On Tuesday, June 7, 2016 at 4:57:05 PM UTC-4, Aaron Levy wrote: > > >> After just getting the 1.9.1 client up with the 1.10 daemon: >> > > The target would be for the client and daemon to be the same version. I'm > not sure how well it behaves with this kind of version skew. > >> >> - I *still* can't run the wrapper with an unprivileged user which is >> not ideal (have to sudo even for the version command) >> >> This is true, because ultimately we are running `rkt run`, which requires > sudo to set up the container docker is being executed in. > > We may have some other options here, however. Like maybe using rkt as the > means of distributing the docker binary (via container), but not actually > executing the client via `rkt run`. > > However, docker is already running as a privileged user, so this would > mostly be a UX, rather than a security change. > > What I'm thinking here is using rkt as the distribution method, but > possibly having the filesystem/chroot that docker uses be a bit more static > (so we may not need to re-create a full container each `docker-wrapper` > execution. I think there could be a solution something like a mix of: > > https://github.com/coreos/toolbox/blob/master/toolbox > https://github.com/kubernetes/kube-deploy/pull/34 > > I won't be able to look into this right away -- but I'd like to test out > something along those lines. > >> >> - Both root & core users have the same .dockercfg file but I cannot >> download private containers. >> >> This is likely because the .dockercfg is not being mounted into the > docker container itself. This could be added via rkt bind-mount: > > --volume dockercfg,kind=host,source=/home/core/.dockercfg \ > --mount volume=dockercfg,target=/root/.dockercfg \ > > >> Final thing to note: the current setup that you've provided still allows >> the default docker unit to start if any actions/commands hit the >> docker.socket. >> > > This is a good point. In the example I had the unit named > `test-docker.service` -- but we would in fact want to override the existing > docker.service file in its entirety. > > As far as the command hanging, it looks like the docker service started / > stopped prior to using the test-docker.service (possibly due to issue you > raised above). I'm not sure how well a 1.9 client will interact with a 1.10 > daemon, also considering the docker container storage may or may not be of > a compatible version. One thing to try would be to wipe out your > /var/lib/docker so you know you have a clean slate before starting the > test-docker service. > > Thank you for this feedback, this is super helpful. > > >> On Friday, June 3, 2016 at 7:48:41 PM UTC-4, Aaron Levy wrote: >>> >>> What is the error you are seeing? It's possible that you have an >>> existing docker daemon running. >>> >>> The steps to test would look something like: >>> >>> Assume "docker-wrapper.sh" and "docker.service" below are from the >>> linked gists. >>> >>> --- >>> systemctl stop docker >>> cp docker-wrapper.sh /opt/bin/docker-wrapper.sh >>> chmod +x /opt/bin/docker-wrapper.sh >>> cp docker.service /etc/systemd/systemd/test-docker.service >>> systemctl daemon-reload >>> systemctl start test-docker >>> >>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper version >>> --- >>> >>> Also note that there may be come incompatibilities between some versions >>> of docker (e.g. going from 1.11 to 1.9.1 likely means clearing existing >>> data in /var/lib/docker). >>> >>> On Fri, Jun 3, 2016 at 3:46 PM, Kevin Ran <[email protected]> wrote: >>> >>>> I'm just running the raw command that you have in that script with >>>> DOCKER_VERSION set to 1.9.1 on an existing node. >>>> >>>> Do I have to overwrite the existing docker.service in userdata in order >>>> for this to work? I could very well be doing something wrong. >>>> >>>> On Friday, June 3, 2016 at 5:24:08 PM UTC-4, Aaron Levy wrote: >>>>> >>>>> Hi Kevin, >>>>> >>>>> /usr/local/bin/docker-entrypoint.sh exists in the docker container >>>>> itself (docker://docker:1.9.1) -- not on the host filesystem. In this >>>>> case >>>>> we don't want to use the dockerd binary on the host >>>>> (/usr/lib/coreos/dockerd), because the goal is to be able to run an >>>>> alternative version of docker in a rkt container. >>>>> >>>>> You should be able to use the docker-wrapper.sh script as linked -- >>>>> but if you're still having trouble can you add the usage / error you're >>>>> seeing? >>>>> >>>>> On Fri, Jun 3, 2016 at 1:36 PM, Kevin Ran <[email protected]> wrote: >>>>> >>>>>> /usr/lib/coreos/dockerd also does not work. The original command >>>>>> just returns run: error parsing app image arguments: unknown flag: >>>>>> --host >>>>>> >>>>>> Tweaking it a bit: >>>>>> >>>>>> rkt run \ >>>>>> --volume etc,kind=host,source=/etc \ >>>>>> --mount volume=etc,target=/etc \ >>>>>> --volume var-lib-docker,kind=host,source=/var/lib \ >>>>>> --mount volume=var-lib-docker,target=/var/lib \ >>>>>> --volume run,kind=host,source=/run \ >>>>>> --mount volume=run,target=/run \ >>>>>> --volume var-run,kind=host,source=/var/run \ >>>>>> --mount volume=var-run,target=/var/run \ >>>>>> --volume usr-lib,kind=host,source=/usr/lib \ >>>>>> --mount volume=usr-lib,target=/usr/lib \ >>>>>> --volume usr-lib64,kind=host,source=/usr/lib64 \ >>>>>> --mount volume=usr-lib64,target=/usr/lib64 \ >>>>>> --volume lib64,kind=host,source=/lib64 \ >>>>>> --mount volume=lib64,target=/lib64 \ >>>>>> --volume usr-bin,kind=host,source=/usr/bin \ >>>>>> --mount volume=usr-bin,target=/usr/bin \ >>>>>> --volume usr-sbin,kind=host,source=/usr/sbin \ >>>>>> --mount volume=usr-sbin,target=/usr/sbin \ >>>>>> --volume etc-ssl-certs,kind=host,source=/usr/share/ca-certificates \ >>>>>> --mount volume=etc-ssl-certs,target=/etc/ssl/certs \ >>>>>> --net=host --inherit-env --insecure-options=image >>>>>> --stage1-path=/usr/share/rkt/stage1-fly.aci docker://docker:1.9.1 \ >>>>>> --exec /usr/lib/coreos/dockerd -- daemon --host=fd:// >>>>>> image: using image from file /usr/share/rkt/stage1-fly.aci >>>>>> image: using image from local store for url docker://docker:1.9.1 >>>>>> run: can't execute "/usr/lib/coreos/dockerd": no such file or >>>>>> directory >>>>>> >>>>>> On Friday, June 3, 2016 at 4:29:55 PM UTC-4, Kevin Ran wrote: >>>>>>> >>>>>>> /usr/local/bin/docker-entrypoint.sh isn't a file on the latest AMI. >>>>>>> Should that be replaced with the daemon bin? >>>>>>> >>>>>>> On Thursday, June 2, 2016 at 6:58:09 PM UTC-4, Aaron Levy wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I've been doing some testing of running Docker inside a rkt (fly) >>>>>>>> container. Part of the reason for this is to be able to run various >>>>>>>> Docker >>>>>>>> versions that are not currently shipping within the CoreOS release. >>>>>>>> >>>>>>>> I wanted to share the (somewhat hack-ish) process so far, and see >>>>>>>> if anyone else is interested in testing / has feedback / finds >>>>>>>> interesting >>>>>>>> ways this is broken. >>>>>>>> >>>>>>>> The starting point (thanks Brandon): >>>>>>>> https://gist.github.com/philips/4ba6f9888499266b0ab09d95991e6784 >>>>>>>> >>>>>>>> I've broken the above into two pieces, a wrapper script and the >>>>>>>> service file, to allow for running docker commands on the cli easier: >>>>>>>> >>>>>>>> >>>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-wrapper-sh >>>>>>>> >>>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-service >>>>>>>> >>>>>>>> For example: >>>>>>>> >>>>>>>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper version >>>>>>>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper run -it busybox /bin/sh >>>>>>>> >>>>>>>> I've also added a bind mount for root certificates: >>>>>>>> >>>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-wrapper-sh-L34 >>>>>>>> >>>>>>>> At this point it seems to be working reasonably well based on some >>>>>>>> initial testing with the coreos-kubernetes single-node installation >>>>>>>> (running CoreOS alpha with Docker 1.9.1): >>>>>>>> >>>>>>>> >>>>>>>> https://github.com/coreos/coreos-kubernetes/compare/master...aaronlevy:docker-in-rkt >>>>>>>> >>>>>>> >>>>> >>> >
