,----[ Initial Thought/Message ]
|  God I wish we had a .32 OpenVZ kernel then this discussion wouldn't
|  even take place ... I appreciate all the excellent work done by all
|  OpenVZ folks! Kir, you rock! Well, here it is:

Hello folks!

We have evaluated the situation once again and made the decision going
forward with LXC. Yes, unfortunately that means ditching OpenVZ. This is
nothing personal (although I am a bit sad since I have invested a lot of
time) but purely logical.

We want/need investment security therefore stuff needs to be in
mainline. There is no argument against that, this is 2010 and not 1995
anymore i.e. out of tree is of no interest from a business point of view

The next stable releases for Debian and Ubuntu are to be scheduled for
March/April. At this moment no one knows if a new OpenVZ kernel will be
available by then. This, we cannot have.

We were looking at KVM and LXC. Linux-VServer and OpenVZ are not in
mainline so they are not considered. KVM is to "fat" for what we want
plus as it looks like after a few tests, switching from OpenVZ to LXC is
quite feasible and the few LXC test systems we created during the last
few days run smoothly on .32.

As with OpenVZ, our hosts as well as containers will run Debian. There
is quite good support for Debian already e.g.
/usr/share/doc/lxc/examples/lxc-debian.gz for example.

Michael> I use to use Linux-vserver years and years ago but when they
Michael> broke IPv6 support moving from 1.x to 2.x I was forced to
Michael> abandon Linux-vserver and switch a number of VM's over to
Michael> OpenVZ. To this day IPv6 remains an "experimental patch" for
Michael> Linux-vserver and I see that question come up on their list
Michael> periodically, so I couldn't migrate back there, even if I
Michael> wanted to. That being said, IPv6 support in the OpenVZ vnet
Michael> device is nothing to brag about either and I have had to
Michael> strictly use the veth devices.

Before OpenVZ we/I used Linux-VServer too. It is excellent I think but
then here is the problem: LVS is basically a one-man show by Herbert
Poetzl. He's a great guy and I meet him a few times in Vienna (were I
live too). What happens if Herbert is run over by a train (which of
course hopefully does not happen but you get the idea)?! This is a
problem, so we switched to OpenVZ.

Michael> However... There is a new kid on the block, depending on your
Michael> requirements. Linux Containers or LXC. It still has a few rough
Michael> edges and some differences with OpenVZ but has the big
Michael> advantage that it's all in the mainline kernel (2.6.29 and
Michael> above), so no more patches (yeah!), it is supported under
Michael> libvirt, and the utilities are in the major cutting edge
Michael> distros like Fedora and Ubuntu.

Michael, you are nothing but right here. Stuff must be in mainline, I
cannot get tired of saying that enough these days. The energy spend
sketching possible scenarios about what we are going to do if and when
will resolve immediately once we use LXC. You just know what will be the
case in X months for now ... that is an irreplaceable peace of mind.
That is true for any Distros out there, host or container ...

Michael> I found that with a couple of scripts, I could directly convert
Michael> OpenVZ config files to LXC config files and start my old OpenVZ
Michael> containers as a container under LXC with no further
Michael> modification inside the container.

Please provide your scripts to the public. I would love to see them,
help improve things and maybe others will join in so nobody needs to be
alone by switching to LXC.

Dietmar, since we are both interested on making this work for Debian
plus, we are in Austria, maybe we should work on this together a bit?
Maybe even have a sprint? My email is suno.ano[at]sunoano.org just in
case ...

Here is what I found so far http://sysadmin-cookbook.rot13.org/#lxc , go
down to ve2lxc. I have already started a very rough/ugly collection of
bits and pieces of information for my personal matters which can be
found at http://sunoano.name/ws/public_xhtml/linux_containers.html

Michael> Other than a couple of initial test containers I was
Michael> experimenting with, once I got my scripts settled down and
Michael> tested, I migrated over 3 dozen VM's from OpenVZ to LXC in a
Michael> single day with none of the containers experiencing more that a
Michael> minute or so of down time (transfer time between hosts).
Michael> Because there were no changes in the containers themselves, I
Michael> could migrate them back, if I needed to, just as fast.

I want this! Tell us more please. Details sir ;-)

Michael> 1) /proc/mounts shows mounts outside of the container (ugly but
Michael>    not fatal). Fixed in git.

Is this true for kernels >= .32 ?

Michael> 2) Possible to break out of a container file system (related to #1
Michael> above). It's possible to break out of chrooted jails. Fixed in
Michael> git by using pivot root. This is serious and if you have
Michael> potential hostiles in a container, I wouldn't use LXC yet or
Michael> use the utilities from git.

Also, is this true for kernels >= .32 ?

Michael> 3) Halt and Reboot of a container not working. You have to
Michael>    manually shut down and restart the container from the host.
Michael>    Being worked on right now. I use a script that detects when
Michael>    there's only one process running (init) in the container and
Michael>    the container runlevel is 0 or 6 to decide to shut it down
Michael>    or restart it. Ugly but works.

Can you please provide the scipt/resolution you are using. This is still
true for >= .32 yes? Hm, my containers started automatically when
rebooting the host. I am on .32, Debian standard kernel in unstable:

,----[ uname -a ]
| Linux wks 2.6.32-trunk-amd64 #1 SMP Sun Jan 10 22:40:40 UTC 2010 x86_64 

Michael> 4) There still seems to be a lot of development work going on
Michael>    in the kernel wrt checkpoint and restore. Since I don't use
Michael>    that much, I haven't paid that much attention but LXC does
Michael>    support freezing and unfreezing containers.

True. I can live without that for a while i.e. it is no problem if I do
not have that feature available for the next half year or so.

Michael> * LXC supports virtual consoles you can connect to and log into
Michael> (lxc-console).

Michael> * LXC does not (yet) support the equivalent of "vzctl enter" (under
Michael> discussion - some possible patches).

Personally I do not care since I setup sshd with each container. It sure
is a nice to have thingy though. http://sunoano.name/ws/public_xhtml/ssh.html

Michael> * Does not have the same level of fine grained resource control
Michael> available with OpenVZ (something that is not a requirement for
Michael> me) and the user_beancounters (some controls in the cgroups
Michael> resources but not as many).

I second that but then I think it is just a matter of time until
resource control will improve to the current state of OpenVZ.

Michael> * Handles the bridge management for the eth interfaces
Michael>   automatically, so no need for extra config files in the host.

I hate bridges therefore I use lxc.network.type = macvlan which is the
equivalent to OpenVZ's venet device ... basically a pipe-like connection
between container and host. No bridge involved. Imho a bride just
complicates a setup and introduces an additional layer of indirection.

Michael> Primary disadvantage to LXC is that the utlities are at 0.6.4
Michael> from source forge and 0.6.3 from Fedora and really really under
Michael> active development and change. Features and facilities are
Michael> still subject to discussion and change and it's not fully
Michael> mature on that level. I don't know how long that will take but
Michael> I wouldn't use anything less that what's in their git repo
Michael> right now or 0.6.5 or higher, when it comes out.

I had no problems on Debian testing/sid; stuff is quite recent here

,----[ dpkg -l lxc* | grep ii ]
| ii  lxc   0.6.4-2       Linux containers userspace tools

Michael> Sooo... If you WANT to run a newer leading edge distro like
Michael> Ubuntu or Fedora for your host system and you can deal with
Michael> those differences, LXC MIGHT be an option.

It is not just an option from my point of view, it is one of two ways
(lxc, kvm) to go about the future of virtualization. Only things that
ship with mainline are worth being considered.

Michael> Exactly. Which is why the burning need to get to a mainstream
Michael> kernel. Containers, namespaces, and cgroups in the kernel have
Michael> matured to the point where they are eminently usable.

I agree. It looks good to me, even after some stress tests on the
network layer I did everything continued to fine.

Michael> I can't speak for the developers here but, I would not be
Michael> surprised if this were a real reason for some of the lack of
Michael> recent progress on newer kernels. Why invest the effort at all
Michael> if you are going to be able to take advantage of mainline
Michael> features in the near future? Skip the transition period and get
Michael> ready for the big jump. Better to organize and prepare for when
Michael> it reaches that level of maturity. I would like to see OpenVZ
Michael> running on a recent linux kernel just using the whole cgroup
Michael> and namespaces facility, even if not all of the granular
Michael> user_beancounters are fully supported (and may never be fully
Michael> supported to that degree of granularity).

Yes, it is paradox. Kir and the rest of the core developers around
OpenVZ have contributed a ton of good stuff to cgroup, namespaces and
the networking part, all of which in mainline now, but then OpenVZ
itself is horribly outdated (.27 where .32 is available).

On Debian I am on .26 on most server around here and yesterday updating
the host failed because the current udev version does not work with .26
but only newer kernels. It now really is the time to act, even if it is
a bit melancholic because OpenVZ is great, great but to old :-/

Michael> If I had the maturity and stability of the OpenVZ utilities
Michael> running on the mainline kernel using namespaces and cgroups and
Michael> no custom patch, that would be my ideal combination right now.

I totally agree even though my experience was that the utilities work
fine. Mainline is mainline is mainline ... :-)

Users mailing list

Reply via email to