Got it working after a bunch of tweaks to the commands that you provided
(sudo and such). A couple of things / concerns:
Following is a terminal copy-paste with the setup that I manually
provisioned. It's just hanging on the version command.
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $ cat
/opt/bin/docker-wrapper
#!/bin/bash
# Wrapper for launching Docker via rkt-fly stage1.
set -e
if [ -z "${DOCKER_VERSION}" ]; then
echo "ERROR: must set DOCKER_VERSION"
exit 1
fi
mkdir --parents /var/lib/docker
exec /usr/bin/rkt run \
--volume etc,kind=host,source=/etc \
--mount volume=etc,target=/etc \
--volume var-lib-docker,kind=host,source=/var/lib \
--mount volume=var-lib-docker,target=/var/lib \
--volume run,kind=host,source=/run \
--mount volume=run,target=/run \
--volume var-run,kind=host,source=/var/run \
--mount volume=var-run,target=/var/run \
--volume usr-lib,kind=host,source=/usr/lib \
--mount volume=usr-lib,target=/usr/lib \
--volume usr-lib64,kind=host,source=/usr/lib64 \
--mount volume=usr-lib64,target=/usr/lib64 \
--volume lib64,kind=host,source=/lib64 \
--mount volume=lib64,target=/lib64 \
--volume usr-bin,kind=host,source=/usr/bin \
--mount volume=usr-bin,target=/usr/bin \
--volume usr-sbin,kind=host,source=/usr/sbin \
--mount volume=usr-sbin,target=/usr/sbin \
--volume etc-ssl-certs,kind=host,source=/usr/share/ca-certificates \
--mount volume=etc-ssl-certs,target=/etc/ssl/certs \
--net=host --inherit-env --insecure-options=image
--stage1-path=/usr/share/rkt/stage1-fly.aci \
docker://docker:${DOCKER_VERSION} --exec
/usr/local/bin/docker-entrypoint.sh -- "$@"
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $ systemctl cat
test-docker
# /etc/systemd/system/test-docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=docker.socket early-docker.target network.target
Requires=docker.socket early-docker.target
[Service]
EnvironmentFile=-/run/flannel_docker_opts.env
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
Environment=DOCKER_VERSION=1.9.1
ExecStart=/opt/bin/docker-wrapper daemon --host=fd://
[Install]
WantedBy=multi-user.target
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $ systemctl status
docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib64/systemd/system/docker.service; disabled;
vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─10-wait-docker.conf, 20-dm-storage.conf, increase-ulimit.conf
Active: failed (Result: exit-code) since Mon 2016-06-06 17:44:17 UTC;
1min 32s ago
Docs: http://docs.docker.com
Process: 18406 ExecStart=/usr/lib/coreos/dockerd daemon --host=fd://
$DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU
$DOCKER_OPT_IPMASQ (code=exited, status=2)
Main PID: 18406 (code=exited, status=2)
Jun 06 17:43:41 ip-10-74-131-184.ec2.internal systemd[1]: Started Docker
Application Container Engine.
Jun 06 17:43:42 ip-10-74-131-184.ec2.internal dockerd[18406]:
time="2016-06-06T17:43:42.196194725Z" level=warning msg="devmapper: Base
device already exists and has filesystem xfs on it. Us...be ignored."
Jun 06 17:44:17 ip-10-74-131-184.ec2.internal systemd[1]: Stopping Docker
Application Container Engine...
Jun 06 17:44:17 ip-10-74-131-184.ec2.internal systemd[1]: docker.service:
Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 06 17:44:17 ip-10-74-131-184.ec2.internal systemd[1]: Stopped Docker
Application Container Engine.
Jun 06 17:44:17 ip-10-74-131-184.ec2.internal systemd[1]: docker.service:
Unit entered failed state.
Jun 06 17:44:17 ip-10-74-131-184.ec2.internal systemd[1]: docker.service:
Failed with result 'exit-code'.
Jun 06 17:44:57 ip-10-74-131-184.ec2.internal systemd[1]: Stopped Docker
Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $ systemctl status
test-docker
● test-docker.service - Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/test-docker.service; disabled;
vendor preset: disabled)
Active: active (running) since Mon 2016-06-06 17:45:06 UTC; 50s ago
Docs: http://docs.docker.com
Main PID: 18494 (run)
Memory: 113.6M
CPU: 48.424s
CGroup: /system.slice/test-docker.service
└─18494
/var/lib/rkt/pods/run/306648b9-7882-4022-aebe-4bec57d0c477/stage1/rootfs/run
--net=host --local-config=/etc/rkt 306648b9-7882-4022-aebe-4bec57d0c477
Jun 06 17:45:06 ip-10-74-131-184.ec2.internal systemd[1]: Started Docker
Application Container Engine.
Jun 06 17:45:07 ip-10-74-131-184.ec2.internal docker-wrapper[18494]: image:
using image from file /usr/share/rkt/stage1-fly.aci
Jun 06 17:45:07 ip-10-74-131-184.ec2.internal docker-wrapper[18494]: image:
using image from local store for url docker://docker:1.9.1
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $ sudo
docker-wrapper version
ERROR: must set DOCKER_VERSION
core@ip-10-74-131-184 sandbox-kran-us-east-1d-control ~ $ sudo -E
docker-wrapper version
image: using image from file /usr/share/rkt/stage1-fly.aci
image: using image from local store for url docker://docker:1.9.1
After just getting the 1.9.1 client up with the 1.10 daemon:
- I *still* can't run the wrapper with an unprivileged user which is not
ideal (have to sudo even for the version command)
- Performing a pull takes a while (might just be due to the 1.10.3
client interacting with the 1.9.1 daemon, but it's really painful).
- Both root & core users have the same .dockercfg file but I cannot
download private containers.
Final thing to note: the current setup that you've provided still allows
the default docker unit to start if any actions/commands hit the
docker.socket.
On Friday, June 3, 2016 at 7:48:41 PM UTC-4, Aaron Levy wrote:
>
> What is the error you are seeing? It's possible that you have an existing
> docker daemon running.
>
> The steps to test would look something like:
>
> Assume "docker-wrapper.sh" and "docker.service" below are from the linked
> gists.
>
> ---
> systemctl stop docker
> cp docker-wrapper.sh /opt/bin/docker-wrapper.sh
> chmod +x /opt/bin/docker-wrapper.sh
> cp docker.service /etc/systemd/systemd/test-docker.service
> systemctl daemon-reload
> systemctl start test-docker
>
> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper version
> ---
>
> Also note that there may be come incompatibilities between some versions
> of docker (e.g. going from 1.11 to 1.9.1 likely means clearing existing
> data in /var/lib/docker).
>
> On Fri, Jun 3, 2016 at 3:46 PM, Kevin Ran <[email protected] <javascript:>
> > wrote:
>
>> I'm just running the raw command that you have in that script with
>> DOCKER_VERSION set to 1.9.1 on an existing node.
>>
>> Do I have to overwrite the existing docker.service in userdata in order
>> for this to work? I could very well be doing something wrong.
>>
>> On Friday, June 3, 2016 at 5:24:08 PM UTC-4, Aaron Levy wrote:
>>>
>>> Hi Kevin,
>>>
>>> /usr/local/bin/docker-entrypoint.sh exists in the docker container
>>> itself (docker://docker:1.9.1) -- not on the host filesystem. In this case
>>> we don't want to use the dockerd binary on the host
>>> (/usr/lib/coreos/dockerd), because the goal is to be able to run an
>>> alternative version of docker in a rkt container.
>>>
>>> You should be able to use the docker-wrapper.sh script as linked -- but
>>> if you're still having trouble can you add the usage / error you're seeing?
>>>
>>> On Fri, Jun 3, 2016 at 1:36 PM, Kevin Ran <[email protected]> wrote:
>>>
>>>> /usr/lib/coreos/dockerd also does not work. The original command just
>>>> returns run: error parsing app image arguments: unknown flag: --host
>>>>
>>>> Tweaking it a bit:
>>>>
>>>> rkt run \
>>>> --volume etc,kind=host,source=/etc \
>>>> --mount volume=etc,target=/etc \
>>>> --volume var-lib-docker,kind=host,source=/var/lib \
>>>> --mount volume=var-lib-docker,target=/var/lib \
>>>> --volume run,kind=host,source=/run \
>>>> --mount volume=run,target=/run \
>>>> --volume var-run,kind=host,source=/var/run \
>>>> --mount volume=var-run,target=/var/run \
>>>> --volume usr-lib,kind=host,source=/usr/lib \
>>>> --mount volume=usr-lib,target=/usr/lib \
>>>> --volume usr-lib64,kind=host,source=/usr/lib64 \
>>>> --mount volume=usr-lib64,target=/usr/lib64 \
>>>> --volume lib64,kind=host,source=/lib64 \
>>>> --mount volume=lib64,target=/lib64 \
>>>> --volume usr-bin,kind=host,source=/usr/bin \
>>>> --mount volume=usr-bin,target=/usr/bin \
>>>> --volume usr-sbin,kind=host,source=/usr/sbin \
>>>> --mount volume=usr-sbin,target=/usr/sbin \
>>>> --volume etc-ssl-certs,kind=host,source=/usr/share/ca-certificates \
>>>> --mount volume=etc-ssl-certs,target=/etc/ssl/certs \
>>>> --net=host --inherit-env --insecure-options=image
>>>> --stage1-path=/usr/share/rkt/stage1-fly.aci docker://docker:1.9.1 \
>>>> --exec /usr/lib/coreos/dockerd -- daemon --host=fd://
>>>> image: using image from file /usr/share/rkt/stage1-fly.aci
>>>> image: using image from local store for url docker://docker:1.9.1
>>>> run: can't execute "/usr/lib/coreos/dockerd": no such file or directory
>>>>
>>>> On Friday, June 3, 2016 at 4:29:55 PM UTC-4, Kevin Ran wrote:
>>>>>
>>>>> /usr/local/bin/docker-entrypoint.sh isn't a file on the latest AMI.
>>>>> Should that be replaced with the daemon bin?
>>>>>
>>>>> On Thursday, June 2, 2016 at 6:58:09 PM UTC-4, Aaron Levy wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I've been doing some testing of running Docker inside a rkt (fly)
>>>>>> container. Part of the reason for this is to be able to run various
>>>>>> Docker
>>>>>> versions that are not currently shipping within the CoreOS release.
>>>>>>
>>>>>> I wanted to share the (somewhat hack-ish) process so far, and see if
>>>>>> anyone else is interested in testing / has feedback / finds interesting
>>>>>> ways this is broken.
>>>>>>
>>>>>> The starting point (thanks Brandon):
>>>>>> https://gist.github.com/philips/4ba6f9888499266b0ab09d95991e6784
>>>>>>
>>>>>> I've broken the above into two pieces, a wrapper script and the
>>>>>> service file, to allow for running docker commands on the cli easier:
>>>>>>
>>>>>>
>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-wrapper-sh
>>>>>>
>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-service
>>>>>>
>>>>>> For example:
>>>>>>
>>>>>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper version
>>>>>> DOCKER_VERSION=1.9.1 /opt/bin/docker-wrapper run -it busybox /bin/sh
>>>>>>
>>>>>> I've also added a bind mount for root certificates:
>>>>>>
>>>>>> https://gist.github.com/aaronlevy/1211c6bb69d9cc60d8965f8733963c13#file-docker-wrapper-sh-L34
>>>>>>
>>>>>> At this point it seems to be working reasonably well based on some
>>>>>> initial testing with the coreos-kubernetes single-node installation
>>>>>> (running CoreOS alpha with Docker 1.9.1):
>>>>>>
>>>>>>
>>>>>> https://github.com/coreos/coreos-kubernetes/compare/master...aaronlevy:docker-in-rkt
>>>>>>
>>>>>
>>>
>
