Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
26.01.2021 22:09, Gena Makhomed пишет: On 26.01.2021 18:41, Scott Dowdle wrote: Have you tried LXD? Not yet. My first post on this mailing list asked if anyone was using LXC in production: Does anyone use LXC and/or systemd-nspawn containers on RHEL 8 / CentOS 8 for production? Well, I use lxc on oracle linux 8 ( just because centos 8 is killed by redhat) in production, I just need to run yet another asterisk instance with different network settings on the same host. Never tried systemd-nspawn, though. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
On 26.01.2021 20:24, Scott Dowdle wrote: Ok, so you are turning off SELinux and using ZFS too? And you still want to stay with EL? Why? RHEL is more stable than Ubuntu, it has 10 year support and rpm installs silently without additional questions and dialogues, as it in deb world. dnf / yum is very useful toolkit for managing operating system packages. And there are many other reasons that will be off-topic to discuss here. Ubuntu and LXD do support ZFS and Canonical's lawyers seem happy to allow ZFS to be bundled with Ubuntu by default. You should get along nicely. I feel comfortable using RHEL, because I still remember CVE-2008-0166. -- Best regards, Gena ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
Greetings, - Original Message - > Can you share your experience with LXC and/or systemd-nspawn > for RHEL 8 / CentOS 8 operating system on the hardware node? > I can't use host network for [system] containers. > Each container must have its own private network. In that case, perhaps you'd like docker/podman's private networking? > Backuping persistent containers and restoring from backup - issue. > I don't want have deal with a mash of different images and layers. I haven't thought of backups. I assume there are a number of backup solutions for docker/podman containers but I'm completely ignorant. > Each my systemd-nspawn container located in separate filesystem: > > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > tank 531G 1.13T 96K /tank > tank/containers 528G 1.13T 168K /tank/containers > tank/containers/119.1G 1.13T 8.00G /tank/containers/1 Ok, so you are turning off SELinux and using ZFS too? And you still want to stay with EL? Why? > Upstream also doesn't support ZFS, but this is extraordinary file system > with excellent feature set. Ubuntu and LXD do support ZFS and Canonical's lawyers seem happy to allow ZFS to be bundled with Ubuntu by default. You should get along nicely. TYL, -- Scott Dowdle 704 Church Street Belgrade, MT 59714 (406)388-0827 [home] (406)994-3931 [work] ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
On 26.01.2021 18:41, Scott Dowdle wrote: Have you tried LXD? Not yet. My first post on this mailing list asked if anyone was using LXC in production: Does anyone use LXC and/or systemd-nspawn containers on RHEL 8 / CentOS 8 for production? What are advantages and disadvantages of each of these technologies? Can you share your experience with LXC and/or systemd-nspawn for RHEL 8 / CentOS 8 operating system on the hardware node? podman is replacement for Docker, it is not replacement for OpenVZ 6 containers. Docker definitely targets "Application Containers"... with one service per container. podman says they can also do "System Containers" by running systemd as the entry point. Of course the vast majority of pre-made container images you'll find in container image repositories aren't built for that, but you can use distro provided images and build a system container image out of them. I have a simple recipe for Fedora, CentOS, and Ubuntu. I don't know how many people are using podman in this capacity yet, and I don't know if it is mature or not for production... but the limited testing I've done with it, has worked out fairly well... using Fedora or CentOS Stream 8 as the host OS... No problem, systemd-nspawn also has worked out fairly well, without extra complexity, introduced by podman "System Containers" images. Yes, podman does still use it's own private network addressing, but I guess that can be overcome by telling it to use the host network. I haven't tried that. Not exactly like OpenVZ's container networking for sure. I can't use host network for [system] containers. Each container must have its own private network. I have containers with 1.6 TiB of valuable data - podman not designed to work in this mode and in such conditions. Persistent data really isn't an issue. You just have to understand how it works. Plenty of people run long-term / persistent-data Docker and podman containers... Backuping persistent containers and restoring from backup - issue. I don't want have deal with a mash of different images and layers. Each my systemd-nspawn container located in separate filesystem: # zfs list NAME USED AVAIL REFER MOUNTPOINT tank 531G 1.13T 96K /tank tank/containers 528G 1.13T 168K /tank/containers tank/containers/119.1G 1.13T 8.00G /tank/containers/1 tank/containers/100 7.59G 1.13T 6.59G /tank/containers/100 tank/containers/111 169G 1.13T 27.6G /tank/containers/111 tank/containers/120 3.05G 1.13T 1.31G /tank/containers/120 tank/containers/121 10.2G 1.13T 9.20G /tank/containers/121 tank/containers/122 8.80G 1.13T 7.23G /tank/containers/122 tank/containers/124 3.20G 1.13T 2.21G /tank/containers/124 tank/containers/125 3.08G 1.13T 2.12G /tank/containers/125 tank/containers/126 87.1G 1.13T 64.1G /tank/containers/126 tank/containers/127 145G 1.13T 125G /tank/containers/127 tank/containers/128 7.46G 1.13T 5.62G /tank/containers/128 tank/containers/129 6.04G 1.13T 3.92G /tank/containers/129 tank/containers/130 5.03G 1.13T 3.01G /tank/containers/130 tank/containers/131 6.41G 1.13T 2.94G /tank/containers/131 tank/containers/132 4.55G 1.13T 2.98G /tank/containers/132 tank/containers/133 22.7G 1.13T 20.6G /tank/containers/133 tank/containers/134 3.36G 1.13T 1.61G /tank/containers/134 tank/containers/135 3.82G 1.13T 1.73G /tank/containers/135 tank/containers/25 1.74G 1.13T 960M /tank/containers/25 tank/containers/30 2.15G 1.13T 1.35G /tank/containers/30 tank/containers/97 5.90G 1.13T 2.06G /tank/containers/97 tank/containers/99 3.15G 1.13T 2.20G /tank/containers/99 Each filesystem has many snapshots (24 hourly and 30 daily), which are replicated to backup server, without the need to stop each systemd-nspawn container for creating snapshot/backup of it. So I have only two alternatives for OS-level virtualization: LXC or systemd-nspawn. If CentOS is your target host, I'd guess that neither of those really are a good solutions... simply because they aren't supported and upstream doesn't care about anything other than podman for containers. Upstream also doesn't support ZFS, but this is extraordinary file system with excellent feature set. LXC varies from one distro to the next... with different kernels, and different versions of libraries and management scripts. Again, LXD on an Ubuntu LTS host is probably the most stable... with Proxmox VE as a close second. Both of those upstreams care about system containers and put in a lot of effort to make it work. LXC/LXD for CentOS 8 and other Linux distros distributed in the form of snap package. Inside snap - ordinary Ubuntu. Google "Install LXC CentOS 8" for more details about this. Good luck. Thank you. Luck is
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
Greetings, - Original Message - > > LXD is a management layer on top of it which provides for easy > > clustering and even managing VMs. I think it is the closest thing > > to vzctl/prlctl from OpenVZ. > > "Yes, you could use LXC without LXD. But you probably would not want to. > On its own, LXC will give you only a basic subset of features. > For a production environment, you’ll want to use LXD". Have you tried LXD? Again, I'd only recommend it on Ubuntu LTS and I believe your target is CentOS so that is probably why you are excluding it, eh? > podman is replacement for Docker, > it is not replacement for OpenVZ 6 containers. Docker definitely targets "Application Containers"... with one service per container. podman says they can also do "System Containers" by running systemd as the entry point. Of course the vast majority of pre-made container images you'll find in container image repositories aren't built for that, but you can use distro provided images and build a system container image out of them. I have a simple recipe for Fedora, CentOS, and Ubuntu. I don't know how many people are using podman in this capacity yet, and I don't know if it is mature or not for production... but the limited testing I've done with it, has worked out fairly well... using Fedora or CentOS Stream 8 as the host OS... and yes, even running the container as a regular user after doing: setsebool -P container_manage_cgroup on Yes, podman does still use it's own private network addressing, but I guess that can be overcome by telling it to use the host network. I haven't tried that. Not exactly like OpenVZ's container networking for sure. > I have containers with 1.6 TiB of valuable data - podman > not designed to work in this mode and in such conditions. Persistent data really isn't an issue. You just have to understand how it works. Plenty of people run long-term / persistent-data Docker and podman containers... although granted, most folks say if you are using persistent data containers, you are doing it wrong. I guess I prefer to do it wrong. :) > So I have only two alternatives for OS-level virtualization: > LXC or systemd-nspawn. If CentOS is your target host, I'd guess that neither of those really are a good solutions... simply because they aren't supported and upstream doesn't care about anything other than podman for containers. LXC varies from one distro to the next... with different kernels, and different versions of libraries and management scripts. Again, LXD on an Ubuntu LTS host is probably the most stable... with Proxmox VE as a close second. Both of those upstreams care about system containers and put in a lot of effort to make it work. Good luck. TYL, -- Scott Dowdle 704 Church Street Belgrade, MT 59714 (406)388-0827 [home] (406)994-3931 [work] ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
On 26.01.2021 0:05, Scott Dowdle wrote: OpenVZ 7 has no updates, and therefore is not suitable for production. The free updates lag behind the paid Virtuozzo 7 version and plenty of people are using it in production. I'm not one of those. See all released OpenVZ 7 updates: http://ftp.netinch.com/pub/openvz/virtuozzo/releases/ Lag between two serial updates can be up to 4-5 month. OpenVZ 7 has many other disadvantages, so I can't use it for production. LXC/LXD is the same technology, as I understand from linuxcontainers.org LXD is a management layer on top of it which provides for easy clustering and even managing VMs. I think it is the closest thing to vzctl/prlctl from OpenVZ. "Yes, you could use LXC without LXD. But you probably would not want to. On its own, LXC will give you only a basic subset of features. For a production environment, you’ll want to use LXD". podman can't be a replacement for OpenVZ 6 / systemd-nspawn because it destroys the root filesystem on the container stop, and all changes made in container configs and other container files will be lost. This is a nightmare for the website hosting server with containers. No, it does NOT destroy the delta disk (that's what I call where changes are stored) upon container stop and I'm not sure why you think it does. You can even export a systemd unit file to manage the container as a systemd service or user service. volumes are a nice way to handle persistence of data if you want to nuke the existing container and make a new one from scratch without losing your data. While it is true you have to approach the container a little differently, podman systemd containers are fairly reasonable "system containers". podman is replacement for Docker, it is not replacement for OpenVZ 6 containers. I have containers with 1.6 TiB of valuable data - podman not designed to work in this mode and in such conditions. So I have only two alternatives for OS-level virtualization: LXC or systemd-nspawn. -- Best regards, Gena ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
Greetings, - Original Message - > OpenVZ 7 has no updates, and therefore is not suitable for > production. The free updates lag behind the paid Virtuozzo 7 version and plenty of people are using it in production. I'm not one of those. > LXC/LXD is the same technology, as I understand from > linuxcontainers.org linuxcontainers.org is owned by Canonical and yes it documents LXC... but LXD is a management layer on top of it which provides for easy clustering and even managing VMs. I think it is the closest thing to vzctl/prlctl from OpenVZ. > podman can't be a replacement for OpenVZ 6 / systemd-nspawn because > it destroys the root filesystem on the container stop, and all > changes made in container configs and other container files will be lost. > This is a nightmare for the website hosting server with containers. No, it does NOT destroy the delta disk (that's what I call where changes are stored) upon container stop and I'm not sure why you think it does. You can even export a systemd unit file to manage the container as a systemd service or user service. volumes are a nice way to handle persistence of data if you want to nuke the existing container and make a new one from scratch without losing your data. While it is true you have to approach the container a little differently, podman systemd containers are fairly reasonable "system containers". TYL, -- Scott Dowdle 704 Church Street Belgrade, MT 59714 (406)388-0827 [home] (406)994-3931 [work] ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
On 25.01.2021 22:24, Scott Dowdle wrote: I found only two possible free/open source alternatives for OpenVZ 6: - LXC - systemd-nspawn Some you seem to have overlooked?!? 1) OpenVZ 7 2) LXD from Canonical that is part of Ubuntu 3) podman containers with systemd installed (set /sbin/init as the entry point) OpenVZ 7 has no updates, and therefore is not suitable for production. LXC/LXD is the same technology, as I understand from linuxcontainers.org podman can't be a replacement for OpenVZ 6 / systemd-nspawn because it destroys the root filesystem on the container stop, and all changes made in container configs and other container files will be lost. This is a nightmare for the website hosting server with containers. systemd-nspawn probably is the best fit for my tasks. But systemd-nspawn also have some major disadvantages in the current RHEL-stable and RHEL-beta versions: https://bugzilla.redhat.com/show_bug.cgi?id=1913734 https://bugzilla.redhat.com/show_bug.cgi?id=1913806 Answering to your previous question: > in the reproduction steps, disabling SELinux is a step? SELinux must be disabled, because if SELinux is enabled - it prevents systemd-nspawn containers from starting. SELinux permissive mode is useless because it consumes more resources compared to completely disabled SELinux. -- Best regards, Gena ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] OS-level virtualization using LXC and systemd-nspawn containers
Greetings, - Original Message - > I found only two possible free/open source alternatives for OpenVZ 6: > > - LXC > - systemd-nspawn Some you seem to have overlooked?!? 1) OpenVZ 7 2) LXD from Canonical that is part of Ubuntu 3) podman containers with systemd installed (set /sbin/init as the entry point) I use LXC on Proxmox VE (which I guess should be #4 above) some although I primarily use it for VMs. Oh, LXD is supposedly packaged for other distros but given that they aren't much into SELinux and they are into snaps, I'd not really recommend it outside of Ubuntu. TYL, -- Scott Dowdle 704 Church Street Belgrade, MT 59714 (406)388-0827 [home] (406)994-3931 [work] ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
