Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Giuseppe Scrivano]

Daniel J Walsh  writes:

> We have to have a version by Tuesday for RHEL.
>
> On 03/16/2017 01:03 PM, Mrunal Patel wrote:
>
>  If we can wait a bit, we should have a new 1.0.0.rc3 for runc soon. 

I've tried to find a workaround for this issue on Atomic Host but it
seems that runC works only with rootfsPropagation = "rprivate",
otherwise it leaks mounts.

For a quick reproducer, it is enough to add "rootfsPropagation":"private" to 
the default
configuration file generated by runC:

bash-4.3# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
 Version: 25.80 (2017-03-13 23:35:50)
  Commit: 
24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
  OSName: fedora-atomic

# cd /var/lib/containers/atomic/docker.0/ (can really be any container)
# rm config.json
# runc spec config.json
# sed -i -e 's|\("linux": {\)|\1\n\t\t"rootfsPropagation": "private",|g' 
config.json
# wc -l /proc/self/mountinfo
34 /proc/self/mountinfo
# /usr/bin/runc run test ; wc -l /proc/self/mountinfo 
container_linux.go:247: starting container process caused 
"process_linux.go:359: container init caused \"rootfs_linux.go:89: jailing 
process inside rootfs caused \\\"pivot_root invalid argument\\\"\""
# wc -l /proc/self/mountinfo
71 /proc/self/mountinfo

Regards,
Giuseppe

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Daniel J Walsh]

We have to have a version by Tuesday for RHEL.


On 03/16/2017 01:03 PM, Mrunal Patel wrote:
> If we can wait a bit, we should have a new 1.0.0.rc3 for runc soon. 
>
> On Thu, Mar 16, 2017 at 8:51 AM, Daniel J Walsh  > wrote:
>
> Mrunal which version of runc should we be shipping?
>
>
> On 03/16/2017 10:01 AM, Giuseppe Scrivano wrote:
> > Daniel J Walsh > writes:
> >
> >>> Could we get an updated runC package?  There is also another fix
> >>> that would be nice to have for the Flannel system container:
> >>>
> >>>
> https://github.com/projectatomic/atomic-system-containers/pull/24
> 
> >>>
> >> What OS Needs updating?
> > Fedora Atomic Host.  I see that the last runC tag v1.0.0-rc2 is from
> > last September.  Do we follow the tags from the runC upstream
> > repository?  Would be nice to have something much newer than that.
> >
> > The RHEL build seems to be based on
> ee992e5ff7143ea3fedb1bb4aa88a41d65a0bd66,
> > which is still quite old "Wed Oct 12 2016".  It doesn't include
> the fix
> > needed for Atomic Host.
> >
> > Thanks,
> > Giuseppe
>
>

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Mrunal Patel]

If we can wait a bit, we should have a new 1.0.0.rc3 for runc soon.

On Thu, Mar 16, 2017 at 8:51 AM, Daniel J Walsh  wrote:

> Mrunal which version of runc should we be shipping?
>
>
> On 03/16/2017 10:01 AM, Giuseppe Scrivano wrote:
> > Daniel J Walsh  writes:
> >
> >>> Could we get an updated runC package?  There is also another fix
> >>> that would be nice to have for the Flannel system container:
> >>>
> >>> https://github.com/projectatomic/atomic-system-containers/pull/24
> >>>
> >> What OS Needs updating?
> > Fedora Atomic Host.  I see that the last runC tag v1.0.0-rc2 is from
> > last September.  Do we follow the tags from the runC upstream
> > repository?  Would be nice to have something much newer than that.
> >
> > The RHEL build seems to be based on ee992e5ff7143ea3fedb1bb4aa88a4
> 1d65a0bd66,
> > which is still quite old "Wed Oct 12 2016".  It doesn't include the fix
> > needed for Atomic Host.
> >
> > Thanks,
> > Giuseppe
>
>

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Daniel J Walsh]

Mrunal which version of runc should we be shipping?


On 03/16/2017 10:01 AM, Giuseppe Scrivano wrote:
> Daniel J Walsh  writes:
>
>>> Could we get an updated runC package?  There is also another fix
>>> that would be nice to have for the Flannel system container:
>>>
>>> https://github.com/projectatomic/atomic-system-containers/pull/24
>>>
>> What OS Needs updating?
> Fedora Atomic Host.  I see that the last runC tag v1.0.0-rc2 is from
> last September.  Do we follow the tags from the runC upstream
> repository?  Would be nice to have something much newer than that.
>
> The RHEL build seems to be based on ee992e5ff7143ea3fedb1bb4aa88a41d65a0bd66,
> which is still quite old "Wed Oct 12 2016".  It doesn't include the fix
> needed for Atomic Host.
>
> Thanks,
> Giuseppe

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Giuseppe Scrivano]

Daniel J Walsh  writes:

>> Could we get an updated runC package?  There is also another fix
>> that would be nice to have for the Flannel system container:
>>
>> https://github.com/projectatomic/atomic-system-containers/pull/24
>>
> What OS Needs updating?

Fedora Atomic Host.  I see that the last runC tag v1.0.0-rc2 is from
last September.  Do we follow the tags from the runC upstream
repository?  Would be nice to have something much newer than that.

The RHEL build seems to be based on ee992e5ff7143ea3fedb1bb4aa88a41d65a0bd66,
which is still quite old "Wed Oct 12 2016".  It doesn't include the fix
needed for Atomic Host.

Thanks,
Giuseppe

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Daniel J Walsh]

We have updated the runc package for RHEL

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12783794



On 03/16/2017 09:16 AM, Daniel J Walsh wrote:
>
> On 03/16/2017 08:06 AM, Giuseppe Scrivano wrote:
>> Colin Walters  writes:
>>
>>> # atomic host status
>>> State: idle
>>> Deployments:
>>> ● fedora-atomic:fedora-atomic/25/x86_64/docker-host
>>>  Version: 25.80 (2017-03-13 23:35:50)
>>>   Commit: 
>>> 24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
>>>   OSName: fedora-atomic
>>> # atomic install --system gscrivano/docker-fedora
>>> # systemctl start docker-fedora
>>> ...
>>> Mar 15 16:02:02 localhost.localdomain runc[4867]: container_linux.go:247: 
>>> starting container process caused "process_linux.go:359: container init 
>>> caused \"rootfs_linux.go:89: jailing process inside rootfs caused 
>>> \\\"pivot_root invalid argument\\\"\""
>>> ```
>> it seems like a bug in runC.  The same version of runC works on Rawhide
>> though, I think the difference is that /var is a bind mount on Atomic
>> Host.  The issue is fixed in runC upstream, looking at the git log, by
>> this:
>>
>> commit 6c147f86496c02f1c28315d1e86ea8be08049ceb
>> Author: Vivek Goyal 
>> Date:   Tue Oct 25 11:15:11 2016 -0400
>>
>> Make parent mount private before bind mounting rootfs
>> 
>> This reverts part of the commit eb0a144b5e383
>>
>> Could we get an updated runC package?  There is also another fix
>> that would be nice to have for the Flannel system container:
>>
>> https://github.com/projectatomic/atomic-system-containers/pull/24
>>
>> Regards,
>> Giuseppe
> What OS Needs updating?
>

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Daniel J Walsh]

On 03/16/2017 08:06 AM, Giuseppe Scrivano wrote:
> Colin Walters  writes:
>
>> # atomic host status
>> State: idle
>> Deployments:
>> ● fedora-atomic:fedora-atomic/25/x86_64/docker-host
>>  Version: 25.80 (2017-03-13 23:35:50)
>>   Commit: 
>> 24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
>>   OSName: fedora-atomic
>> # atomic install --system gscrivano/docker-fedora
>> # systemctl start docker-fedora
>> ...
>> Mar 15 16:02:02 localhost.localdomain runc[4867]: container_linux.go:247: 
>> starting container process caused "process_linux.go:359: container init 
>> caused \"rootfs_linux.go:89: jailing process inside rootfs caused 
>> \\\"pivot_root invalid argument\\\"\""
>> ```
> it seems like a bug in runC.  The same version of runC works on Rawhide
> though, I think the difference is that /var is a bind mount on Atomic
> Host.  The issue is fixed in runC upstream, looking at the git log, by
> this:
>
> commit 6c147f86496c02f1c28315d1e86ea8be08049ceb
> Author: Vivek Goyal 
> Date:   Tue Oct 25 11:15:11 2016 -0400
>
> Make parent mount private before bind mounting rootfs
> 
> This reverts part of the commit eb0a144b5e383
>
> Could we get an updated runC package?  There is also another fix
> that would be nice to have for the Flannel system container:
>
> https://github.com/projectatomic/atomic-system-containers/pull/24
>
> Regards,
> Giuseppe
What OS Needs updating?

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Giuseppe Scrivano]

Colin Walters  writes:

> # atomic host status
> State: idle
> Deployments:
> ● fedora-atomic:fedora-atomic/25/x86_64/docker-host
>  Version: 25.80 (2017-03-13 23:35:50)
>   Commit: 
> 24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
>   OSName: fedora-atomic
> # atomic install --system gscrivano/docker-fedora
> # systemctl start docker-fedora
> ...
> Mar 15 16:02:02 localhost.localdomain runc[4867]: container_linux.go:247: 
> starting container process caused "process_linux.go:359: container init 
> caused \"rootfs_linux.go:89: jailing process inside rootfs caused 
> \\\"pivot_root invalid argument\\\"\""
> ```

it seems like a bug in runC.  The same version of runC works on Rawhide
though, I think the difference is that /var is a bind mount on Atomic
Host.  The issue is fixed in runC upstream, looking at the git log, by
this:

commit 6c147f86496c02f1c28315d1e86ea8be08049ceb
Author: Vivek Goyal 
Date:   Tue Oct 25 11:15:11 2016 -0400

Make parent mount private before bind mounting rootfs

This reverts part of the commit eb0a144b5e383

Could we get an updated runC package?  There is also another fix
that would be nice to have for the Flannel system container:

https://github.com/projectatomic/atomic-system-containers/pull/24

Regards,
Giuseppe

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Giuseppe Scrivano]

Colin Walters  writes:

> Does anyone know what the status of
> https://github.com/projectatomic/atomic-system-containers
> is in general, and in particular I'm interested in the
> "containerized docker" approach.
>
> Can someone who knows a bit more about this add
> e.g. a `README.md` with getting started instructions?

We have tried different solutions to get it done, the current version in
atomic-system-containers is using a chroot for running the Docker
container, but I don't really like this approach and anyway it brings
its own set of issues as maintaining/creating the rootfs for the
container manually.

Yes, it is a bit messy, I am going to work on this and try to make it
clearer.  The gscrivano/docker-fedora and gscrivano/docker-centos
containers are based on the PR here:

  https://github.com/projectatomic/atomic-system-containers/pull/38

As soon as it gets a bit more stable, we will need to move somewhere
else than my Docker hub account, as we did for Flannel and etcd.

The biggest issue is how to support live-restore.  Docker remounts
/var/lib/docker/devicemapper/* as MS_PRIVATE when it runs, so whatever
mount it creates there, it will not be accessible once the namespace is
destroyed without some dirty tricks.
I've filed an issue for Docker upstream, as in general it is not
possible to run Docker in its own mount namespace and support
live-restore (could be useful even for things like systemd
InaccessiblePaths=):

  https://github.com/docker/docker/issues/31489

There are some workarounds in the PR, but I got it to work somehow.  The
limitation is that you can't exec in a Docker container that is living
more than two Docker containers update (as a system container keeps only
two deployments).  The reason for this limitation is that when you do
"docker exec ...", it will use the runc that was installed as part of
the system container deployment that after two updates is removed.

I wouldn't worry too much of it for now. IIUIC in libcontainerd
master there is some work to let the shim process attached to a docker
container do the exec itself, so we won't have the limitation with
upgrading the Docker system containers as won't be required to run runc
from the mount namespace the container was created.

> # atomic host status
> State: idle
> Deployments:
> ● fedora-atomic:fedora-atomic/25/x86_64/docker-host
>  Version: 25.80 (2017-03-13 23:35:50)
>   Commit: 
> 24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
>   OSName: fedora-atomic
> # atomic install --system gscrivano/docker-fedora
> # systemctl start docker-fedora
> ...
> Mar 15 16:02:02 localhost.localdomain runc[4867]: container_linux.go:247: 
> starting container process caused "process_linux.go:359: container init 
> caused \"rootfs_linux.go:89: jailing process inside rootfs caused 
> \\\"pivot_root invalid argument\\\"\""

looks bad, I am going to have a look.

Regards,
Giuseppe

Re: [atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Jerry Zhang]

Hi Colin,

> Hey,
> 
> Does anyone know what the status of
> https://github.com/projectatomic/atomic-system-containers
> is in general, and in particular I'm interested in the
> "containerized docker" approach.
> 

Most of the containers that are in the repo are operational
but not fully ready for production, minus etcd and flannel
which are more stable and tested compared to the other ones.

> Can someone who knows a bit more about this add
> e.g. a `README.md` with getting started instructions?
> 
> I did find https://hub.docker.com/r/gscrivano/docker-fedora/
> which has some info, but it appears generic and not specific
> to this container.  The tradeoffs/implementation details
> of containerizing Docker in particular seem worth having
> a specific doc.
> 

Giuseppe's repo is not technically the offical repo, although
you are right in that we need better docs. Once the issues
are more flattened out I'd imagine Giuseppe would add the
docs for docker.

> (Also, that image is auto-built from github:giuseppe/atomic-oci-containers
>  which is different from the projectatomic one?)
> 

No that is the same repo. The repo in projectatomic was moved over
from giuseppe's. If you click the link it will direct you to the
projectatomic repo.

> I just tried this:
> 
> ```
> # atomic host status
> State: idle
> Deployments:
> ● fedora-atomic:fedora-atomic/25/x86_64/docker-host
>  Version: 25.80 (2017-03-13 23:35:50)
>   Commit:
>   24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
>   OSName: fedora-atomic
> # atomic install --system gscrivano/docker-fedora
> # systemctl start docker-fedora
> ...
> Mar 15 16:02:02 localhost.localdomain runc[4867]: container_linux.go:247:
> starting container process caused "process_linux.go:359: container init
> caused \"rootfs_linux.go:89: jailing process inside rootfs caused
> \\\"pivot_root invalid argument\\\"\""
> ```
> 
> 
> 

Try instead: https://github.com/projectatomic/atomic-system-containers/pull/38
I build that locally and docker runs fine from a system container (running
on f25 cloud):
# atomic containers list
   CONTAINER ID IMAGECOMMAND  CREATED  
STATE BACKENDRUNTIME   
   flannel  gscrivano/flannel/usr/bin/flanneld-ru 2017-03-15 16:37 
running   ostree runc  
   docker   local/docker /usr/bin/init.sh 2017-03-15 16:37 
running   ostree runc  
   etcd local/etcd   /usr/bin/etcd-env.sh 2017-03-15 16:37 
running   ostree runc

# systemctl status docker
● docker.service - Docker service
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: 
disabled)
  Drop-In: /etc/systemd/system/docker.service.d
   └─flannel.conf
   Active: active (running) since Wed 2017-03-15 16:37:58 UTC; 7min ago

For testing convenience I've build that branch to 
https://hub.docker.com/r/jerzhang/docker/,
so you can pull with `atomic pull --storage ostree jerzhang/docker`
Please let me know if that works for you.

Regards,
Yu Qi Zhang

[atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Colin Walters]

Hey,

Does anyone know what the status of
https://github.com/projectatomic/atomic-system-containers
is in general, and in particular I'm interested in the
"containerized docker" approach.

Can someone who knows a bit more about this add
e.g. a `README.md` with getting started instructions?

I did find https://hub.docker.com/r/gscrivano/docker-fedora/
which has some info, but it appears generic and not specific
to this container.  The tradeoffs/implementation details
of containerizing Docker in particular seem worth having
a specific doc.

(Also, that image is auto-built from github:giuseppe/atomic-oci-containers
 which is different from the projectatomic one?)

I just tried this:

```
# atomic host status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
 Version: 25.80 (2017-03-13 23:35:50)
  Commit: 
24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
  OSName: fedora-atomic
# atomic install --system gscrivano/docker-fedora
# systemctl start docker-fedora
...
Mar 15 16:02:02 localhost.localdomain runc[4867]: container_linux.go:247: 
starting container process caused "process_linux.go:359: container init caused 
\"rootfs_linux.go:89: jailing process inside rootfs caused \\\"pivot_root 
invalid argument\\\"\""
```