Re: [lxc-users] Strange freezes with btrfs backend

[Ron Kelley]

My 0.02

We have been using btrfs in production for more than a year on other 
projects and about 6mos with LXD.  It has been rock solid.  I have multiple 
LXD servers each with >20 containers. We have a separate btrfs filesystem 
(with compression enabled) to store the LXD containers. I take nightly 
snapshots for all containers, and each server probably has 2000 snapshots. 
The only issue thus far is the IO hit when deleting lots of snapshots at 
one time.  You need to delete a few (10 at a time), pause for 60secs, then 
delete the next 10.


I have used ZFS in Linux in the past and could never get adequate 
performance - regardless of tuning or amount of RAM given to ZFS.  In fact, 
I started using ZFS for our backup server (64TB raw storage with 32GB RAM) 
but had to move back to XFS due to severe performance issues.  Nothing 
fancy; I did a by-the-bok install and enabled compression and snapshots. I 
tried every tuning option available (including SSD for L2-ARC). Nothing 
would improve the performance.


To the OP: are you sure btrfs is causing your issues?  Have you traced the 
OP activity during the hiccup moments?





On December 3, 2016 7:37:21 AM "Fajar A. Nugraha"  wrote:


On Sat, Dec 3, 2016 at 6:01 PM, Sergiusz Pawlowicz 
wrote:


> You'd need to set arc to be as small as possible:
> # cat /etc/modprobe.d/zfs-arc-max.conf
> options zfs zfs_arc_max=67108865

What is a sense of using ZFS if you don't use its cache? Non sense. it



- excellent integration with lxd
- data integrity verification using checksum
- thin-lvm-like space management
- send/receive
- compression
- much more mature compared to btrfs



will work slower and less reliable than ext4.



I never said it was faster.

In general, zfs WILL be slower - to some extent - compared to ext4. Just
like ext4 (presumably with LVM and raid/mirror) will be slower compared to
writing to raw disk directly - especially if you also exclude any kind of
raid/mirror and volume manager.

To give more perspective to my particular use case, my EC2 zfsroot AMI only
use 1GB EBS thanks to lz4 compression. And that's with around 400 MB free
space. Thanks to zfs snapshot/clone, I can also use clones of my root as
containers (which is more efficient compared to LVM snapshots or overlay)

Is it a suitable solution for everyone? No.
Does it work for my use case? Yes. MUCH more so compared to ext4 or btrfs.
Will it work for Pierce's use case? I believe so.

--
Fajar



--
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Strange freezes with btrfs backend

[Fajar A. Nugraha]

On Sat, Dec 3, 2016 at 6:01 PM, Sergiusz Pawlowicz 
wrote:

> > You'd need to set arc to be as small as possible:
> > # cat /etc/modprobe.d/zfs-arc-max.conf
> > options zfs zfs_arc_max=67108865
>
> What is a sense of using ZFS if you don't use its cache? Non sense. it
>

- excellent integration with lxd
- data integrity verification using checksum
- thin-lvm-like space management
- send/receive
- compression
- much more mature compared to btrfs


> will work slower and less reliable than ext4.
>
>
I never said it was faster.

In general, zfs WILL be slower - to some extent - compared to ext4. Just
like ext4 (presumably with LVM and raid/mirror) will be slower compared to
writing to raw disk directly - especially if you also exclude any kind of
raid/mirror and volume manager.

To give more perspective to my particular use case, my EC2 zfsroot AMI only
use 1GB EBS thanks to lz4 compression. And that's with around 400 MB free
space. Thanks to zfs snapshot/clone, I can also use clones of my root as
containers (which is more efficient compared to LVM snapshots or overlay)

Is it a suitable solution for everyone? No.
Does it work for my use case? Yes. MUCH more so compared to ext4 or btrfs.
Will it work for Pierce's use case? I believe so.

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Strange freezes with btrfs backend

[Sergiusz Pawlowicz]

> You'd need to set arc to be as small as possible:
> # cat /etc/modprobe.d/zfs-arc-max.conf
> options zfs zfs_arc_max=67108865

What is a sense of using ZFS if you don't use its cache? Non sense. it
will work slower and less reliable than ext4.

S.
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Strange freezes with btrfs backend

[Fajar A. Nugraha]

On Sat, Dec 3, 2016 at 11:30 AM, Pierce Ng  wrote:

> Hi all,
>
> I'm running LXD on a Ubuntu 16.04 VPS with ~1GB RAM. My setup uses a disk
> image
> file, running on the default ext4 base filesystem, as the btrfs backend.
> The
> server runs four containers, of which only one is on the high side of
> lightly
> loaded.
>
> I'm getting random freezes such that all containers err, although the host
> OS
> continues running. When I reboot, the server gets stuck at the boot prompt
> requiring a manual fsck to get going.
>
> Anyone seeing this?
>
> Should I switch to ZFS? Is it sensible to run ZFS on a 1GB RAM VPS?
> Realistically, I'll be using ZFS in the same manner - via a disk image
> file.
> Is this a stupid idea?
>
>

1GB for 4 containers is probably overkill, depending on what you're trying
to achive. "random freeze" might be caused by out of memory (OOM)
condition, where the host is too busy swapping. This is why I disable swap
on most of my systems: at least when OOM killer is active it'd immediately
kill processes instead of keeping the disk busy.

That being said, 1GB with zfs should work. I have several variants running
on EC2 (with swap disabled):
- one with xfs root (or you can use whatever rootfs in the AMI), with
separate zfs EBS for lxd
- one with zfs root (requires an effort to create this, but now I have this
as an AMI). One of the dataset (on the same root pool) is managed by lxd.

You'd need to set arc to be as small as possible:
# cat /etc/modprobe.d/zfs-arc-max.conf
options zfs zfs_arc_max=67108865

Anything lower than that will make the setting ignored. After that, it's
simply a matter of checking whether your system have enough memory to run
the containers (try "htop", and check syslog for OOM killer)

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Strange freezes with btrfs backend

[Tomasz Chmielewski]

On 2016-12-03 18:31, Pierce Ng wrote:

On Sat, Dec 03, 2016 at 11:49:02AM +0700, Sergiusz Pawlowicz wrote:

With 1GB of memory is is not recommended to use ZFS not BTRFS,
especially via a disk image file. Just forget about it.


The same VPS - as same as a VPS can be :-) - ran LXC on Ubuntu 14.04 
fine. For
my work load perhaps the directory or LVM backends for LXD are good 
enough.


Don't use a file-based image for a different filesystem - your 
performance will be poor, and you risk loosing the whole filesystem if 
something wrong happens to your main fs.


Exceptions to this rule are perhaps:

- testing
- recovery
- KVM set up to use a file-based image for a VM (still, not perfect)


If you want to use LXD with btrfs, it's still quite easy to do, though 
stick to the following rules:


- btrfs should be placed on a separate block device (disk/partition) - 
so, let's say /dev/xvdb1 mounted to /var/lib/lxd


- btrfs is still not very stable (no RAID and RAID-1 levels are fine for 
basic workloads, though you may still hit an occasional hiccup) - use 
the most recent stable kernel with btrfs (for Ubuntu, you can get it 
from http://kernel.ubuntu.com/~kernel-ppa/mainline/ - you'll need 
"raw.lxc: lxc.aa_allow_incomplete=1" in container's config); kernel 4.4 
used in Ubuntu 16.04 is too old and you will run into problems after 
some time!


- chattr +C on database (mysql, postgres, mongo, elasticsearch) 
directories, otherwise, performance will be very poor (note chattr +C 
only works on directories and empty files; for existing files, you'll 
have to move them out and copy back in)



Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Strange freezes with btrfs backend

[Pierce Ng]

On Sat, Dec 03, 2016 at 11:49:02AM +0700, Sergiusz Pawlowicz wrote:
> With 1GB of memory is is not recommended to use ZFS not BTRFS,
> especially via a disk image file. Just forget about it.

The same VPS - as same as a VPS can be :-) - ran LXC on Ubuntu 14.04 fine. For
my work load perhaps the directory or LVM backends for LXD are good enough.

Pierce
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users