[Lxc-users] Cluster Resource Agent
Hi, I'm wondering if anybody is using LXC in a high availability cluster. I tried to use it in a Pacemaker Cluster together with DRBD. In theory there would be the VirtualDomain Resource Agent supporting libvirt. But since my libvirt experience together with LXC was not promising, I think the best option is to use lxc-tools. It worked for me when using the lxc init script (/etc/init.d/lxc) for active/passive configurations. As far as I found out only /etc/lxc/ and corresponding lxc rootfs dirs have to be shared. /var/lib/lxc should not be necessary, because lxc init script doesn't use lxc-create/lxc-destroy. Anything else to take care of when moving LXC containers around machines? For active/active and more advanced configurations an OCF Resource Agent for LXC would be nice. It could be similar to the ManageVE RA for OpenVZ: http://hg.linux-ha.org/agents/raw-file/tip/heartbeat/ManageVE Regards, Christoph -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-clone
On 04/05/2011 02:52 PM, Serge Hallyn wrote: Quoting Daniel Lezcano (daniel.lezc...@free.fr): I think more configuration tweaking will be needed but this patch looks good for me. I'm hoping to set up a machine on which to play with lxc on btrfs snapshots sometime this week. (In the background, starting the build now). Nice :) +echo Tweaking configuration +cp $lxc_path/$lxc_orig/config $lxc_path/$lxc_new/config +sed -i '/lxc.utsname/d' $lxc_path/$lxc_new/config +echo lxc.utsname = $hostname $lxc_path/$lxc_new/config We should not assume lxc.utsname is in the configuration file in order to not write a hostname in all the cases. The user may want to let the container to setup itself the hostname. What do you think is the best way to do this? We could allow the user to specify a 'firstboot' script, which gets copied into root directory of the container. Maybe boot the container when it's done, run /firstboot.sh, and shut down. Or just let that happen when the user first boots. We could use a /etc/init.d/lxc-firstboot script, but that will only work if the container's init system actually looks at sysvinit scripts. Obviously sysvinit and upstart do, and I must assume that systemd does. lxc-init I assume doesn't. Mmh, that's look a bit complicate for the user. I was thinking about something simpler like: grep -q lxc.utsname $lxc_path/$lxc_new/config if [ $? == 0 ]; then sed -e s/lxc.utsname/lxc.utsname=$hostname $lxc_path/$lxc_new/config else echo lxc.utsname = $hostname $lxc_path/$lxc_new/config fi and so for the rest of the configuration variables. I suspect the configuration tweaks will be more complicated than that.. For example: lxc.network.veth.peer = veth1234 should be removed or replaced by a random name. But we can improve the different configuration tweaks later. Thanks for looking it over, Daniel. I'm using this heavily every day, so am really keen on polishing it :) Thanks for this command, I think it is very useful. By the way, did the lxc.rootfs=lvm_image worked ? Thanks -- Daniel -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] native (non-NAT) routing?
On 04/05/2011 02:56 PM, Ulli Horlacher wrote: On Tue 2011-04-05 (14:53), Daniel Lezcano wrote: Can you give the bridge setup ? (brctl show) root@zoo:/lxc# brctl show bridge name bridge id STP enabled interfaces br0 8000.0050568e0003 no eth0 Hi Ulli, is your container up when you show the bridge information ? is it possible you give the ip addr result on the host ? Thanks -- Daniel ps : In the past, I did exactly the same configuration than you and it worked, so I know what you are trying to achieve should work. -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Lxc-attach status update?
On 03/31/2011 12:04 PM, Goran Cetusic wrote: I applied the patch to 2.6.38.2 along with the patch for unionfs 2.6.38-rc7. Now I get the old error message root@ct1:/home/cetko# lxc-attach -n i07dc0.n0 lxc-attach: Function not implemented - failed to set namespace 'pid' lxc-attach: failed to enter the namespace The lxc patch was applied with quilt push -a Unionfs was applied with gunzip -c unionfs.diff.gz | patch -p1 I checked and all namespaces are enabled in the kernel .config file. The template for the .config file is a ubuntu 2.6.35.8 kernel. The working one with the 2.6.35 lxc patch. I compiled the kernel with make-kpkg --initrd --append-to-version=-imunes kernel_image kernel_headers. I'm cross compiling on a more powerful machine so it's convenient. Am I missing something? How can I gather more info to debug? Can I help in any way? Lxc is a crucial part of my master's thesis. I noticed there is no man page on Debian for lxc-attach. Since lxc-attach is of great use to me I would like to write it if nobody minds. Goran can you check the commit : http://lxc.git.sourceforge.net/git/gitweb.cgi?p=lxc/lxc;a=commit;h=63e0f402aa028c02d15fc5fa7066b77a45877e83 fix your problem ? Thanks -- Daniel -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-fstab vs /etc/fstab vs /lib/init/fstab
Quoting Trent W. Buck (t...@cyber.com.au): Serge Hallyn serge.hal...@canonical.com writes: Next, upstart's mountall consults /lib/init/fstab. That's the one which will usually prevent container startup from proceeding. The lxcguest package for ubuntu will force upstart to mount an empty version of that file before mountall runs. So if you install lxcguest then mountall can safely run, which makes your container safer against package updates. Interesting approach. IIRC I just dpkg-divert --rename /lib/init/fstab : /lib/init/fstab Yeah that can be a good trick. That doesn't meet my goal, though, of having a single disk image able to boot both as container and kvm host (and physical host). Which is particularly useful when dealing with something like openstack, where Chuck now has a single image which is used to fire off both kvm based and libvirt-lxc based images. thanks, -serge signature.asc Description: Digital signature -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-clone
Quoting Daniel Lezcano (daniel.lezc...@free.fr): What do you think is the best way to do this? We could allow the user to specify a 'firstboot' script, which gets copied into root directory of the container. Maybe boot the container when it's done, run /firstboot.sh, and shut down. Or just let that happen when the user first boots. We could use a /etc/init.d/lxc-firstboot script, but that will only work if the container's init system actually looks at sysvinit scripts. Obviously sysvinit and upstart do, and I must assume that systemd does. lxc-init I assume doesn't. Mmh, that's look a bit complicate for the user. I was thinking about something simpler like: grep -q lxc.utsname $lxc_path/$lxc_new/config if [ $? == 0 ]; then sed -e s/lxc.utsname/lxc.utsname=$hostname $lxc_path/$lxc_new/config else echo lxc.utsname = $hostname $lxc_path/$lxc_new/config fi and so for the rest of the configuration variables. Ok, yeah, that'll be necessary and independent of the distro. But I do think a firstboot option will be useful both for -clone and -create. It can be totally optional so as not to confuse those who don't want it. I suspect the configuration tweaks will be more complicated than that.. For example: lxc.network.veth.peer = veth1234 should be removed or replaced by a random name. Sigh, yeah. But we can improve the different configuration tweaks later. Phes :) Thanks for looking it over, Daniel. I'm using this heavily every day, so am really keen on polishing it :) Thanks for this command, I think it is very useful. By the way, did the lxc.rootfs=lvm_image worked ? Yup! Now, to do btrfs (I got sidetracked and won't be able to get to it this week probably), we will probably want to be able to specify a bind mount source. That way we can mount a btrfs somewhere like /opt/lxc. Otherwise IIRC the way btrfs works we won't be able to create a snapshot from /var/lib/lxc/natty/rootfs for /var/lib/lxc/n2/rootfs, because the snapshots are are created as sibling directories. Can you think of a clean way to do that or add that to the current config format? thanks, -serge signature.asc Description: Digital signature -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Lxc-attach status update?
On 03/31/2011 12:04 PM, Goran Cetusic wrote: I noticed there is no man page on Debian for lxc-attach. Since lxc-attach is of great use to me I would like to write it if nobody minds. I was going to do it but please do, btw, -- is not documented in lxc-execute.1 C. -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Cluster Resource Agent
On 4/6/2011 4:56 AM, Christoph Mitasch wrote: Hi, I'm wondering if anybody is using LXC in a high availability cluster. I tried to use it in a Pacemaker Cluster together with DRBD. In theory there would be the VirtualDomain Resource Agent supporting libvirt. But since my libvirt experience together with LXC was not promising, I think the best option is to use lxc-tools. It worked for me when using the lxc init script (/etc/init.d/lxc) for active/passive configurations. As far as I found out only /etc/lxc/ and corresponding lxc rootfs dirs have to be shared. /var/lib/lxc should not be necessary, because lxc init script doesn't use lxc-create/lxc-destroy. Anything else to take care of when moving LXC containers around machines? For active/active and more advanced configurations an OCF Resource Agent for LXC would be nice. It could be similar to the ManageVE RA for OpenVZ: http://hg.linux-ha.org/agents/raw-file/tip/heartbeat/ManageVE Regards, Christoph What lxc init script? I think we all write our own and there is no official one yet. I write one for openSUSE but it's not in the suse lxc package nor in any other official suse package, just in a stand-alone rclxc package in my buildservice repo. If the official packages for other distros includes an init script, it will be different for each one since containers are such a low-level feature that can be used for so many different kinds of jobs, it's hard to imagine what an official init script would even be like. -- bkw -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
[Lxc-users] Container with different architecture like arm on x86 [How-to]
Hi, I tried to run an arm container under a x86_64 host and it works Little how-to : build a static compiled qemu-arm take qemu sources and build it with : ./configure --static --target-list=arm-linux-user; make U will find static qemu for arm at ./arm-linux-user/qemu-arm use the binfmt_misc kernel module mount the pseudofs : mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc have an arm container let's say it is at /lxc/armcontainer copy qemu in the container : cp ./arm-linux-user/qemu-arm /lxc/armcontainer/ enable binfmt : echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/bin/qemu-arm:'/proc/sys/fs/binfmt_misc/register launch your container normaly. I found this cool, I hope it be useful someone else. I have made this how-to from bash history, I could have made some mistakes. feel free to ask if you're in troubles. regards, Guillaume ZITTA -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Container with different architecture like arm on x86 [How-to]
On 4/6/2011 3:26 PM, l...@zitta.fr wrote: Hi, I tried to run an arm container under a x86_64 host and it works Little how-to : build a static compiled qemu-arm take qemu sources and build it with : ./configure --static --target-list=arm-linux-user; make U will find static qemu for arm at ./arm-linux-user/qemu-arm use the binfmt_misc kernel module mount the pseudofs : mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc have an arm container let's say it is at /lxc/armcontainer copy qemu in the container : cp ./arm-linux-user/qemu-arm /lxc/armcontainer/ enable binfmt : echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/bin/qemu-arm:'/proc/sys/fs/binfmt_misc/register launch your container normaly. I found this cool, I hope it be useful someone else. I have made this how-to from bash history, I could have made some mistakes. feel free to ask if you're in troubles. regards, Guillaume ZITTA A few questions, The echo command references /usr/local/bin/qemu-arm, but I don't see that anywhere else in the recipe. Is that a x86_64 binary on the host or is that supposed to be a arm binary in the container, or is it simply ignored in this case and doesn't matter that it doesn't exist? It sort of looks like you are telling the host x86_64 kernel to run a x86_64 qemu-arm executable any time it encounters an arm elf executable, and then since you are placing an arm qemu-arm executable in the container fs I guess you are implying that the arm executable you will be trying to run will be that arm qemu executable? Why would you do that? foo - qemu - qemu - kernel ?? ie: arm-executable foo - arm executable qemu-arm - x86_64 executable qemu-arm - x86_64 host kernel ?? Assuming that even works. Doesn't there have to be an arm kernel in there somewhere? Like: arm-foo - arm-kernel - x86_64-qemu-arm - x86_64-host-kernel I don't see the point in this. As long as you have qemu in there anywhere it means you are doing full cpu virtualization, avoiding which is pretty much the sole purpose of containers. If it's really true that you can have qemu provide _only_ cpu virtualization yet somehow have the host kernel support the arm executables through that I guess that's a win since you have a single kernel doling out resources directly to all processes instead of kernels within kernels. Then again wouldn't that result in every single arm executable running inside it's own instance of qemu, auto launched by the binfmt? That might be ok for application containers that only run one process but that would be terrible for a full system container unless that container really only ran one process directly, an arm kernel. And in that case I don't see the point of doing that inside a container. It's already even more isolated inside qemu than what the container provides and the container layer just becomes pointless overhead. But doesn't the arm kernel have rather a lot more differences than merely understanding the arm binary format and cpu? I would have thought the container would have to run an x86_64 (or i386) binary, which would be qemu, and that qemu would have to run an arm kernel, and all other arm processes would have to run in that arm kernel. I think I need an example to illustrate a use case for this. -- bkw -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Container with different architecture like arm on x86 [How-to]
On 4/6/2011 5:30 PM, Justin Cormack wrote: On Wed, 2011-04-06 at 16:45 -0400, Brian K. White wrote: A few questions, The echo command references /usr/local/bin/qemu-arm, but I don't see that anywhere else in the recipe. Is that a x86_64 binary on the host or is that supposed to be a arm binary in the container, or is it simply ignored in this case and doesn't matter that it doesn't exist? It sort of looks like you are telling the host x86_64 kernel to run a x86_64 qemu-arm executable any time it encounters an arm elf executable, and then since you are placing an arm qemu-arm executable in the container fs I guess you are implying that the arm executable you will be trying to run will be that arm qemu executable? Why would you do that? foo - qemu - qemu - kernel ?? ie: arm-executable foo - arm executable qemu-arm - x86_64 executable qemu-arm - x86_64 host kernel ?? Assuming that even works. Doesn't there have to be an arm kernel in there somewhere? Like: arm-foo - arm-kernel - x86_64-qemu-arm - x86_64-host-kernel I don't see the point in this. As long as you have qemu in there anywhere it means you are doing full cpu virtualization, avoiding which is pretty much the sole purpose of containers. If it's really true that you can have qemu provide _only_ cpu virtualization yet somehow have the host kernel support the arm executables through that I guess that's a win since you have a single kernel doling out resources directly to all processes instead of kernels within kernels. Then again wouldn't that result in every single arm executable running inside it's own instance of qemu, auto launched by the binfmt? That might be ok for application containers that only run one process but that would be terrible for a full system container unless that container really only ran one process directly, an arm kernel. And in that case I don't see the point of doing that inside a container. It's already even more isolated inside qemu than what the container provides and the container layer just becomes pointless overhead. But doesn't the arm kernel have rather a lot more differences than merely understanding the arm binary format and cpu? I would have thought the container would have to run an x86_64 (or i386) binary, which would be qemu, and that qemu would have to run an arm kernel, and all other arm processes would have to run in that arm kernel. I think I need an example to illustrate a use case for this. Qemu is just being used as an arm instruction set interpreter, making x86 system calls to the native kernel. binfmt_misc lets you run other architecture binaries via emulation just by executing the binary. Obviously its slow, but if you want to build an arm distro say it gives another option other than cross compiling or a native compile on a slow machine. Justin Back to the first question, this actually works for binaries other than hello.c ? How many binaries live entirely within the high level calls that are really fully abstracted by the kernel? I guess I have to try this because I don't believe it. qemu just emulates hardware, as in the cpu and some of the supporting system. You can't run user executable on hardware. Only specially crafted free-standing things which is pretty much just bios/efi, bootloaders, kernels, and memtest86. Not ls for instance. I'm familiar with binfmt since I used to use iBCS and then linux-abi ages ago to run SCO binaries on linux, and similarly to run linux binaries within lxrun on SCO, and similarly osrcompat to run unixware binaries on open server and vice-versa, and linux on freebsd, etc ...) But in all those cases, the following always is true: * The executables have the same cpu instruction set as the host kernel. * The executables have the same endianness as the host kernel and filesystem and utilities. And at least one or more of the following is also always true: * The emulation layer explicitly goes to lengths to handle various unavoidable differences and conflicts. Like remapping syscalls that take different numbers or types of arguments, and exhibit different behavior, even though they are named the same and do nominally the same job. Faking various system level environmental things like fake device nodes, /proc entries, etc... maybe cpu registers or memory/io addresses too I don't know everything but I know it's not exactly trivial or ignorable. * The emulation layer provides fail-as-graceful-as-possible stubs for things that can't or haven't yet been really translated. * Users of the emulation layer simply know that only a small subset of things will actually work. Anything might have any sort of problem, and if it breaks you get to keep the pieces. It's useful in a few very special cases and requires significant hacks and workarounds and compromises, but isn't generally useful. I mean it's not just a few things, it's things everywhere you turn, filesystems that return values
