On Wed, Jun 22, 2016 at 10:17 AM Kevin Ran <[email protected]> wrote:
> Another thing that I'm curious about is how would *we *switch / upgrade > between docker runtimes, if possible? > The current design idea is that CoreOS declares the "default" application and then allows for users to override. See the design doc here: https://docs.google.com/document/d/1W0vqVctpxpALhOyyZAHaYEPKukGSQOf_L_B3Qx_NR9w/edit > Are there any talks/ideas floating around that allow users to choose which > distro of docker gets run on the machine (assuming that we can only choose > distros that are compatible with the version of CoreOS that we're on)? > Yes, the image and version will be changeable by the user. Hope that makes sense. Looking forward to any feedback. Brandon > On Tuesday, June 7, 2016 at 4:57:05 PM UTC-4, Aaron Levy wrote: >> >> >>> After just getting the 1.9.1 client up with the 1.10 daemon: >>> >> >> The target would be for the client and daemon to be the same version. I'm >> not sure how well it behaves with this kind of version skew. >> >>> >>> - I *still* can't run the wrapper with an unprivileged user which is >>> not ideal (have to sudo even for the version command) >>> >>> This is true, because ultimately we are running `rkt run`, which >> requires sudo to set up the container docker is being executed in. >> >> We may have some other options here, however. Like maybe using rkt as the >> means of distributing the docker binary (via container), but not actually >> executing the client via `rkt run`. >> >> However, docker is already running as a privileged user, so this would >> mostly be a UX, rather than a security change. >> >> What I'm thinking here is using rkt as the distribution method, but >> possibly having the filesystem/chroot that docker uses be a bit more static >> (so we may not need to re-create a full container each `docker-wrapper` >> execution. I think there could be a solution something like a mix of: >> >> https://github.com/coreos/toolbox/blob/master/toolbox >> https://github.com/kubernetes/kube-deploy/pull/34 >> >> I won't be able to look into this right away -- but I'd like to test out >> something along those lines. >> >>> >>> - Both root & core users have the same .dockercfg file but I cannot >>> download private containers. >>> >>> This is likely because the .dockercfg is not being mounted into the >> docker container itself. This could be added via rkt bind-mount: >> >> --volume dockercfg,kind=host,source=/home/core/.dockercfg \ >> --mount volume=dockercfg,target=/root/.dockercfg \ >> >> >>> Final thing to note: the current setup that you've provided still allows >>> the default docker unit to start if any actions/commands hit the >>> docker.socket. >>> >> >> This is a good point. In the example I had the unit named >> `test-docker.service` -- but we would in fact want to override the existing >> docker.service file in its entirety. >> >> As far as the command hanging, it looks like the docker service started / >> stopped prior to using the test-docker.service (possibly due to issue you >> raised above). I'm not sure how well a 1.9 client will interact with a 1.10 >> daemon, also considering the docker container storage may or may not be of >> a compatible version. One thing to try would be to wipe out your >> /var/lib/docker so you know you have a clean slate before starting the >> test-docker service. >> >> Thank you for this feedback, this is super helpful. >> >> >>> On Friday, June 3, 2016 at 7:48:41 PM UTC-4, Aaron Levy wrote: >>>> >>>> What is the error you are seeing? It's possible that you have an >>>> existing docker daemon running. >>>> >>>> The steps to test would look something like: >>>> >>>> Assume "docker-wrapper.sh" and "docker.service" below are from the >>>> linked gists. >>>> >>>> --- >>>> systemctl stop docker >>>> cp docker-wrapper.sh /opt/bin/docker-wrapper.sh >>>> chmod +x /opt/bin/docker-wrapper.sh >>>> cp docker.service /etc/systemd/systemd/test-docker.service >>>> systemctl daemon-reload >>>> systemctl start test-docker >>>> >>>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper version >>>> --- >>>> >>>> Also note that there may be come incompatibilities between some >>>> versions of docker (e.g. going from 1.11 to 1.9.1 likely means clearing >>>> existing data in /var/lib/docker). >>>> >>>> On Fri, Jun 3, 2016 at 3:46 PM, Kevin Ran <[email protected]> wrote: >>>> >>>>> I'm just running the raw command that you have in that script with >>>>> DOCKER_VERSION set to 1.9.1 on an existing node. >>>>> >>>>> Do I have to overwrite the existing docker.service in userdata in >>>>> order for this to work? I could very well be doing something wrong. >>>>> >>>>> On Friday, June 3, 2016 at 5:24:08 PM UTC-4, Aaron Levy wrote: >>>>>> >>>>>> Hi Kevin, >>>>>> >>>>>> /usr/local/bin/docker-entrypoint.sh exists in the docker container >>>>>> itself (docker://docker:1.9.1) -- not on the host filesystem. In this >>>>>> case >>>>>> we don't want to use the dockerd binary on the host >>>>>> (/usr/lib/coreos/dockerd), because the goal is to be able to run an >>>>>> alternative version of docker in a rkt container. >>>>>> >>>>>> You should be able to use the docker-wrapper.sh script as linked -- >>>>>> but if you're still having trouble can you add the usage / error you're >>>>>> seeing? >>>>>> >>>>>> On Fri, Jun 3, 2016 at 1:36 PM, Kevin Ran <[email protected]> wrote: >>>>>> >>>>>>> /usr/lib/coreos/dockerd also does not work. The original command >>>>>>> just returns run: error parsing app image arguments: unknown flag: >>>>>>> --host >>>>>>> >>>>>>> Tweaking it a bit: >>>>>>> >>>>>>> rkt run \ >>>>>>> --volume etc,kind=host,source=/etc \ >>>>>>> --mount volume=etc,target=/etc \ >>>>>>> --volume var-lib-docker,kind=host,source=/var/lib \ >>>>>>> --mount volume=var-lib-docker,target=/var/lib \ >>>>>>> --volume run,kind=host,source=/run \ >>>>>>> --mount volume=run,target=/run \ >>>>>>> --volume var-run,kind=host,source=/var/run \ >>>>>>> --mount volume=var-run,target=/var/run \ >>>>>>> --volume usr-lib,kind=host,source=/usr/lib \ >>>>>>> --mount volume=usr-lib,target=/usr/lib \ >>>>>>> --volume usr-lib64,kind=host,source=/usr/lib64 \ >>>>>>> --mount volume=usr-lib64,target=/usr/lib64 \ >>>>>>> --volume lib64,kind=host,source=/lib64 \ >>>>>>> --mount volume=lib64,target=/lib64 \ >>>>>>> --volume usr-bin,kind=host,source=/usr/bin \ >>>>>>> --mount volume=usr-bin,target=/usr/bin \ >>>>>>> --volume usr-sbin,kind=host,source=/usr/sbin \ >>>>>>> --mount volume=usr-sbin,target=/usr/sbin \ >>>>>>> --volume etc-ssl-certs,kind=host,source=/usr/share/ca-certificates >>>>>>> \ >>>>>>> --mount volume=etc-ssl-certs,target=/etc/ssl/certs \ >>>>>>> --net=host --inherit-env --insecure-options=image >>>>>>> --stage1-path=/usr/share/rkt/stage1-fly.aci docker://docker:1.9.1 \ >>>>>>> --exec /usr/lib/coreos/dockerd -- daemon --host=fd:// >>>>>>> image: using image from file /usr/share/rkt/stage1-fly.aci >>>>>>> image: using image from local store for url docker://docker:1.9.1 >>>>>>> run: can't execute "/usr/lib/coreos/dockerd": no such file or >>>>>>> directory >>>>>>> >>>>>>> On Friday, June 3, 2016 at 4:29:55 PM UTC-4, Kevin Ran wrote: >>>>>>>> >>>>>>>> /usr/local/bin/docker-entrypoint.sh isn't a file on the latest AMI. >>>>>>>> Should that be replaced with the daemon bin? >>>>>>>> >>>>>>>> On Thursday, June 2, 2016 at 6:58:09 PM UTC-4, Aaron Levy wrote: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I've been doing some testing of running Docker inside a rkt (fly) >>>>>>>>> container. Part of the reason for this is to be able to run various >>>>>>>>> Docker >>>>>>>>> versions that are not currently shipping within the CoreOS release. >>>>>>>>> >>>>>>>>> I wanted to share the (somewhat hack-ish) process so far, and see >>>>>>>>> if anyone else is interested in testing / has feedback / finds >>>>>>>>> interesting >>>>>>>>> ways this is broken. >>>>>>>>> >>>>>>>>> The starting point (thanks Brandon): >>>>>>>>> https://gist.github.com/philips/4ba6f9888499266b0ab09d95991e6784 >>>>>>>>> >>>>>>>>> I've broken the above into two pieces, a wrapper script and the >>>>>>>>> service file, to allow for running docker commands on the cli easier: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-wrapper-sh >>>>>>>>> >>>>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-service >>>>>>>>> >>>>>>>>> For example: >>>>>>>>> >>>>>>>>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper version >>>>>>>>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper run -it busybox >>>>>>>>> /bin/sh >>>>>>>>> >>>>>>>>> I've also added a bind mount for root certificates: >>>>>>>>> >>>>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-wrapper-sh-L34 >>>>>>>>> >>>>>>>>> At this point it seems to be working reasonably well based on some >>>>>>>>> initial testing with the coreos-kubernetes single-node installation >>>>>>>>> (running CoreOS alpha with Docker 1.9.1): >>>>>>>>> >>>>>>>>> >>>>>>>>> https://github.com/coreos/coreos-kubernetes/compare/master...aaronlevy:docker-in-rkt >>>>>>>>> >>>>>>>> >>>>>> >>>> >>
