Re: root on btrfs and lvmthinp for docker backing confusion

[Jason Brooks]

- Original Message -
> From: "Chris Murphy" 
> To: "Fedora Cloud SIG" 
> Sent: Thursday, December 10, 2015 12:39:26 AM
> Subject: root on btrfs and lvmthinp for docker backing confusion
> 
> OK instead of hijacking some other thread...(sorta too late).
> 
> # lsblk -f
> NAME  FSTYPE  LABEL UUID
> MOUNTPOINT
> sda
> =E2=94=9C=E2=94=80sda1vfat  5956-63D8
> /boot/efi
> =E2=94=9C=E2=94=80sda2btrfs   f23a
> 8b0c4840-4fc7-4782-a4c0-25fec8a40dd4   /sysroot
> =E2=94=9C=E2=94=80sda3ext4
> 908cb4df-410b-47e4-afb1-872255bd1244   /boot
> =E2=94=9C=E2=94=80sda4LVM2_member
> 9WkBEW-XwcN-8f99-nQOP-E1w8-QiBs-6Xio7a
> =E2=94=94=E2=94=80sda5swap
> 53f95874-06db-4433-a74b-8dc31105c528   [SWAP]
> 
> -bash-4.3# pvs
>   PV VG   Fmt  Attr PSize   PFree
>   /dev/sda4  VG   lvm2 a--  426.51g 126.36g
> -bash-4.3# vgs
>   VG   #PV #LV #SN Attr   VSize   VFree
>   VG 1   1   0 wz--n- 426.51g 126.36g
> -bash-4.3# lvs
>   LV  VG   Attr   LSize   Pool Origin Data%  Meta%  Move
> Log Cpy%Sync Convert
>   docker-pool VG   twi-a-tz-- 300.00g 0.00   0.44
> 
> 
> What I'd like to know how to do, is get docker to use the thin pool
> setup here as backing, instead of Btrfs.
> 
> What I tried:
> 
> # cp /usr/lib/docker-storage-setup/docker-storage-setup
> /etc/sysconfig/docker-storage-setup
> 
> I then edited that file making one change such that VG=3DVG since that's
> the name of the VG. So it still says to use
> STORAGE_DRIVER=3Ddevicemapper.
> 
> 
> # systemctl status docker-storage-setup
> =E2=97=8F docker-storage-setup.service - Docker Storage Setup
>Loaded: loaded
> (/usr/lib/systemd/system/docker-storage-setup.service; enabled; vendor
> preset: disabled)
>Active: inactive (dead) since Thu 2015-12-10 01:09:06 MST; 45s ago
>   Process: 8008 ExecStart=3D/usr/bin/docker-storage-setup (code=3Dexited,
> status=3D0/SUCCESS)
>  Main PID: 8008 (code=3Dexited, status=3D0/SUCCESS)
> 
> Dec 10 01:09:05 f23a.localdomain systemd[1]: Starting Docker Storage Setup.=
> ..
> Dec 10 01:09:05 f23a.localdomain docker-storage-setup[8008]: INFO:
> Volume group backing root filesystem could not be determined
> Dec 10 01:09:06 f23a.localdomain docker-storage-setup[8008]: Logical
> volume "docker-pool" changed.
> Dec 10 01:09:06 f23a.localdomain systemd[1]: Started Docker Storage Setup.
> 
> 
> # systemctl status docker
> =E2=97=8F docker.service - Docker Application Container Engine
>Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled;
> vendor preset: disabled)
>   Drop-In: /usr/lib/systemd/system/docker.service.d
>=E2=94=94=E2=94=80flannel.conf
>Active: failed (Result: exit-code) since Thu 2015-12-10 01:09:06 MST; 48=
> s ago
>  Docs: http://docs.docker.com
>   Process: 8032 ExecStart=3D/usr/bin/docker daemon $OPTIONS
> $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $INSECURE_REGISTRY
> (code=3Dexited, status=3D1/FAILURE)
>  Main PID: 8032 (code=3Dexited, status=3D1/FAILURE)
> 
> Dec 10 01:09:06 f23a.localdomain systemd[1]: Starting Docker
> Application Container Engine...
> Dec 10 01:09:06 f23a.localdomain docker[8032]:
> time=3D"2015-12-10T01:09:06.240354137-07:00" level=3Dinfo
> msg=3D"[graphdriver] using prior storage driver \"btrfs\""
> Dec 10 01:09:06 f23a.localdomain docker[8032]:
> time=3D"2015-12-10T01:09:06.249931298-07:00" level=3Dfatal msg=3D"Error
> starting daemon: SELinux is not supporte...h driver"
> Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Main
> process exited, code=3Dexited, status=3D1/FAILURE
> Dec 10 01:09:06 f23a.localdomain systemd[1]: Failed to start Docker
> Application Container Engine.
> Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Unit
> entered failed state.
> Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Failed
> with result 'exit-code'.
> Hint: Some lines were ellipsized, use -l to show in full.
> 
> 
> So there's a disconnect between docker-storage-setup, which is set to
> use devicemapper and the volume group named VG, and docker.service
> which continues to use the btrfs driver (and then fails for the known
> reason that --selinux-enabled is set by default and that's not yet
> working until the automatic chcon patch lands).
> 
> In the meantime I can just remove --selinux-enabled. But I'd like to
> know how to redirect docker to use lvm thinp if that's possible.

You can ignore docker-storage-setup and edit /etc/sysconfig/docker-storage
yourself.

Here's what it looks like from the f23 vagrant box:

DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs --storage-opt 
dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt 
dm.use_deferred_removal=true

Jason

> 
> --
> Chris Murphy
> ___
> cloud mailing list
> 

Re: root on btrfs and lvmthinp for docker backing confusion

[Chris Murphy]

On Thu, Dec 10, 2015 at 9:54 AM, Dusty Mabe  wrote:

> So you created the VG VG and the docker-pool LV on your own before
> docker-storage-setup is run?

Yes.

> What I would do is leave sda4 blank and then put the following in your
> config file:
>
> DEVS=/dev/sda4
> VG=vgdocker

Fails with

Dec 10 11:10:13 f23a.localdomain docker-storage-setup[1135]: Partition
specification unsupported at this time.


> What I think this will do is create a PV out of /dev/sda4 and create a
> VG (named vgdocker) on top of it. It will then create the docker-pool
> LV for you and have the docker daemon use that as the backing store.
>
> Let me know if this is what you were looking for or not!

Yes.


-- 
Chris Murphy
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: [cloud] #144: f23 atomic iso configures docker loopback storage

[Fedora Cloud Trac Tickets]

#144: f23 atomic iso configures docker loopback storage
-+-
 Reporter:  jasonbrooks  |   Owner:
 Type:  defect   |  Status:  new
 Priority:  major|   Milestone:  Future
Component:  ---  |  Resolution:
 Keywords:   |
-+-

Comment (by jasonbrooks):

 https://github.com/projectatomic/fedora-productimg-atomic is responsible
 for setting this up correctly. There's a bug
 (https://bugzilla.redhat.com/show_bug.cgi?id=1290257) and pending fixed
 pkg pending... not sure if it'll fix this issue as well.

-- 
Ticket URL: 
cloud 
Fedora Cloud Working Group Ticketing System
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: root on btrfs and lvmthinp for docker backing confusion

[Dusty Mabe]

On 12/10/2015 08:35 PM, Chris Murphy wrote:
> On Thu, Dec 10, 2015 at 11:47 AM, Jason Brooks  wrote:
> 
>> You can ignore docker-storage-setup and edit /etc/sysconfig/docker-storage
>> yourself.
> 
> OK I've done "systemctl disable docker-storage-setup"
> 
>>
>> Here's what it looks like from the f23 vagrant box:
>>
>> DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs --storage-opt 
>> dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt 
>> dm.use_deferred_removal=true
> 
> 
> Short version: This works. Docker service starts, no errors, and it's
> not using a loopback device.
> 
> Followup question: Does Docker directly use a thin pool without
> creating (virtual size) logical volumes? Because I don't see any other
> LV's created, and no XFS filesystem appears on the host using the
> mount command. And yet I see XFS mount and umount kernel messages on
> the host. This is sort of an esoteric question. However, I have no
> access to container files from the host like I can see inside each
> btrfs subvolume when btrfs is the backing method. And that suggests
> possibly rather different backup strategies depending on the backing.

I believe it chops it up using low level device mapper stuff. I think
you don't see the mounts on your host because they are in a different
mount namespace (part of the magic behind containers).

For more info on docker + device mapper look at slides 37-44 of [1]

[1] - http://www.slideshare.net/Docker/docker-storage-drivers
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: root on btrfs and lvmthinp for docker backing confusion

[Chris Murphy]

On Thu, Dec 10, 2015 at 11:47 AM, Jason Brooks  wrote:

> You can ignore docker-storage-setup and edit /etc/sysconfig/docker-storage
> yourself.

OK I've done "systemctl disable docker-storage-setup"

>
> Here's what it looks like from the f23 vagrant box:
>
> DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs --storage-opt 
> dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt 
> dm.use_deferred_removal=true


Short version: This works. Docker service starts, no errors, and it's
not using a loopback device.

Followup question: Does Docker directly use a thin pool without
creating (virtual size) logical volumes? Because I don't see any other
LV's created, and no XFS filesystem appears on the host using the
mount command. And yet I see XFS mount and umount kernel messages on
the host. This is sort of an esoteric question. However, I have no
access to container files from the host like I can see inside each
btrfs subvolume when btrfs is the backing method. And that suggests
possibly rather different backup strategies depending on the backing.


Long version of the question:


# systemctl stop docker

Copied generic docker-storage and edited as follows:

DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs
--storage-opt dm.thinpooldev=/dev/mapper/vgfedora-docker--pool
--storage-opt dm.use_deferred_removal=true

# systemctl start docker
Dec 10 13:05:02 f23a.localdomain systemd[1]: Starting Docker
Application Container Engine...
Dec 10 13:05:06 f23a.localdomain docker[1695]:
time="2015-12-10T13:05:06.377269077-07:00" level=info msg="Firewalld
running: false"
Dec 10 13:05:06 f23a.localdomain docker[1695]:
time="2015-12-10T13:05:06.791189262-07:00" level=info msg="Default
bridge (docker0) is assigned with an IP address 172.17.0.1/16. Daemon
option --bip can be used to set a preferred IP address"
Dec 10 13:05:07 f23a.localdomain docker[1695]:
time="2015-12-10T13:05:07.572415354-07:00" level=info msg="Loading
containers: start."
Dec 10 13:05:07 f23a.localdomain docker[1695]: ..
Dec 10 13:05:07 f23a.localdomain docker[1695]:
time="2015-12-10T13:05:07.576094399-07:00" level=info msg="Loading
containers: done."
Dec 10 13:05:07 f23a.localdomain docker[1695]:
time="2015-12-10T13:05:07.576663930-07:00" level=info msg="Daemon has
completed initialization"
Dec 10 13:05:07 f23a.localdomain docker[1695]:
time="2015-12-10T13:05:07.576770908-07:00" level=info msg="Docker
daemon" commit=f7c1d52-dirty execdriver=native-0.2
graphdriver=devicemapper version=1.9.1-fc23
Dec 10 13:05:07 f23a.localdomain docker[1695]:
time="2015-12-10T13:05:07.577004141-07:00" level=info msg="API listen
on /var/run/docker.sock"
Dec 10 13:05:07 f23a.localdomain systemd[1]: Started Docker
Application Container Engine.

host# docker pull fedora
host# docker images
REPOSITORY  TAG IMAGE IDCREATED
 VIRTUAL SIZE
fedora  latest  597717fc21bd2 weeks
ago 204 MB

So that all works, and then also the thin pool data% is growing after
each step, according to lvs. But there is no logical volume, no file
system.

host# docker run -i -t fedora /bin/bash

host# mount | grep xfs
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
host# mount | grep ext
/dev/sda3 on /boot type ext4 (rw,relatime,seclabel,stripe=4,data=ordered)

It's working. But is Docker directly using the thin pool without
creating a thin logical volume and file system? That's unexpected.

host# lvs
  LV  VG   Attr   LSize   Pool Origin Data%  Meta%
Move Log Cpy%Sync Convert
  docker-pool vgfedora twi-aotz-- 250.00g 0.11   0.53
host# docker pull ubuntu
[...snip pull output...]
-bash-4.3# lvs
  LV  VG   Attr   LSize   Pool Origin Data%  Meta%
Move Log Cpy%Sync Convert
  docker-pool vgfedora twi-aotz-- 250.00g 0.20   0.63

Yeah, it seems to directly use the thin pool without create a virtual
size LV first. Kernel messages show a bunch of items like this:

[ 7818.790685] XFS (dm-4): Mounting V5 Filesystem
[ 7819.038211] XFS (dm-4): Ending clean mount
[ 7834.219132] XFS (dm-4): Unmounting Filesystem

So it's using XFS on something, that isn't appearing in mount on the
host or in the container. And multiple containers all appear to use
XFS on the same device mapper device (dm-4) which does not appear on
the host in the /dev/ directory. So this is really... obscured.

-- 
Chris Murphy
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Docker 1.9.1-4 needs testing

[Joe Brockmeier]

Hey all,

We need a few more eyeballs on this:

https://bodhi.fedoraproject.org/updates/FEDORA-2015-5917f166c6

Version currently in stable has a few annoying bugs, like "docker
search" not working with default settings.

If you have time today to test + give karma (if you find it worthy)
please do!

Thanks,

jzb
-- 
Joe Brockmeier | Community Team, OSAS
j...@redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://dissociatedpress.net/



signature.asc
Description: OpenPGP digital signature
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: Add net.ifnames=0 to Atomic host

[Colin Walters]

On Thu, Dec 10, 2015, at 11:09 AM, Dusty Mabe wrote:

> This commit is against master.. Can you apply the change to the f23 branch?

Done: https://git.fedorahosted.org/cgit/spin-kickstarts.git/commit/?h=f23
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: Add net.ifnames=0 to Atomic host

[Kushal Das]

On 09/12/15, Colin Walters wrote:
> On Wed, Dec 9, 2015, at 11:40 AM, Dusty Mabe wrote:
> >
> >
> >
> On 12/04/2015 02:16 PM, Colin Walters wrote:
> > > On Mon, Nov 23, 2015, at 10:35 AM, Dusty Mabe wrote:
> >>
> >>
> >> Right, this would affect them all. As you mention below we were
> >> intending for this ("eth0") to be the case for a long time so I
> >> figured we should fix it. You are right that this would be a change
> >> over how we released F23 atomic and it does have implications for
> >> other environments.
> >
> > Based on that, my instinct here is to only do this change for the
> > Vagrant box.  WDYT?
> >
> >
> Ok can you merge the attached patch then?
> 
> Done, thank you!
> 
> https://git.fedorahosted.org/cgit/spin-kickstarts.git/commit/?id=80eeba2146f8165b274348113b54b81070116ba7

Same failure [1] in today's tests.

[1] https://apps.fedoraproject.org/autocloud/jobs/828/output

Kushal
-- 
Fedora Cloud Engineer
CPython Core Developer
CentOS Cloud SIG lead
http://kushaldas.in
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: Docker 1.9.1-4 needs testing

[Joe Brockmeier]

On 12/10/2015 09:55 AM, Haïkel wrote:
> I tested it this morning and gave it good karma so it should be pushed.

Thanks!

If we can team up on these updates it will help a lot.

Best,

jzb
-- 
Joe Brockmeier | Community Team, OSAS
j...@redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://dissociatedpress.net/



signature.asc
Description: OpenPGP digital signature
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: Docker 1.9.1-4 needs testing

[Haïkel]

I tested it this morning and gave it good karma so it should be pushed.

H.
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: Add net.ifnames=0 to Atomic host

[Dusty Mabe]

On 12/09/2015 01:45 PM, Colin Walters wrote:

On Wed, Dec 9, 2015, at 11:40 AM, Dusty Mabe wrote:

On 12/04/2015 02:16 PM, Colin Walters wrote:
> On Mon, Nov 23, 2015, at 10:35 AM, Dusty Mabe wrote: >> >> >> Right, this would affect them all. As you mention below we 
were >> intending for this ("eth0") to be the case for a long time so 
I >> figured we should fix it. You are right that this would be a 
change >> over how we released F23 atomic and it does have 
implications for >> other environments. > > Based on that, my 
instinct here is to only do this change for > the Vagrant box.  WDYT?

Ok can you merge the attached patch then?

Done, thank you!
https://git.fedorahosted.org/cgit/spin-kickstarts.git/commit/?id=80eeba2146f8165b274348113b54b81070116ba7


This commit is against master.. Can you apply the change to the f23 branch?

Dusty
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org

Re: Docker 1.9.1-4 needs testing

[Haïkel]

Sure, I wished Bodhi had the ability to create trackers so that we
could list the updates that we need to focus.
We can use etherpad for that purpose.

H.
___
cloud mailing list
cloud@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org