Re: [Lxc-users] Container with different architecture like arm on x86 [How-to]
On 4/7/2011 5:23 AM, l...@zitta.fr wrote: qemu has 2 modes system and user You described system mode and I used user mode That resolves a lot of the mystery right there. I hadn't realized qemu had such a mode. The other issues are either * obviated by the fact that you're already doing it and the world didn't blow up (kernel/abi/environment compatibility), * or the usefulness outweighs the cost (each process runs in it's own qemu instance). Pretty slick. -- bkw -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
[Lxc-users] Container with different architecture like arm on x86 [How-to]
Hi, I tried to run an arm container under a x86_64 host and it works Little how-to : build a static compiled qemu-arm take qemu sources and build it with : ./configure --static --target-list=arm-linux-user; make U will find static qemu for arm at ./arm-linux-user/qemu-arm use the binfmt_misc kernel module mount the pseudofs : mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc have an arm container let's say it is at /lxc/armcontainer copy qemu in the container : cp ./arm-linux-user/qemu-arm /lxc/armcontainer/ enable binfmt : echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/bin/qemu-arm:'/proc/sys/fs/binfmt_misc/register launch your container normaly. I found this cool, I hope it be useful someone else. I have made this how-to from bash history, I could have made some mistakes. feel free to ask if you're in troubles. regards, Guillaume ZITTA -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Container with different architecture like arm on x86 [How-to]
On 4/6/2011 3:26 PM, l...@zitta.fr wrote: Hi, I tried to run an arm container under a x86_64 host and it works Little how-to : build a static compiled qemu-arm take qemu sources and build it with : ./configure --static --target-list=arm-linux-user; make U will find static qemu for arm at ./arm-linux-user/qemu-arm use the binfmt_misc kernel module mount the pseudofs : mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc have an arm container let's say it is at /lxc/armcontainer copy qemu in the container : cp ./arm-linux-user/qemu-arm /lxc/armcontainer/ enable binfmt : echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/bin/qemu-arm:'/proc/sys/fs/binfmt_misc/register launch your container normaly. I found this cool, I hope it be useful someone else. I have made this how-to from bash history, I could have made some mistakes. feel free to ask if you're in troubles. regards, Guillaume ZITTA A few questions, The echo command references /usr/local/bin/qemu-arm, but I don't see that anywhere else in the recipe. Is that a x86_64 binary on the host or is that supposed to be a arm binary in the container, or is it simply ignored in this case and doesn't matter that it doesn't exist? It sort of looks like you are telling the host x86_64 kernel to run a x86_64 qemu-arm executable any time it encounters an arm elf executable, and then since you are placing an arm qemu-arm executable in the container fs I guess you are implying that the arm executable you will be trying to run will be that arm qemu executable? Why would you do that? foo - qemu - qemu - kernel ?? ie: arm-executable foo - arm executable qemu-arm - x86_64 executable qemu-arm - x86_64 host kernel ?? Assuming that even works. Doesn't there have to be an arm kernel in there somewhere? Like: arm-foo - arm-kernel - x86_64-qemu-arm - x86_64-host-kernel I don't see the point in this. As long as you have qemu in there anywhere it means you are doing full cpu virtualization, avoiding which is pretty much the sole purpose of containers. If it's really true that you can have qemu provide _only_ cpu virtualization yet somehow have the host kernel support the arm executables through that I guess that's a win since you have a single kernel doling out resources directly to all processes instead of kernels within kernels. Then again wouldn't that result in every single arm executable running inside it's own instance of qemu, auto launched by the binfmt? That might be ok for application containers that only run one process but that would be terrible for a full system container unless that container really only ran one process directly, an arm kernel. And in that case I don't see the point of doing that inside a container. It's already even more isolated inside qemu than what the container provides and the container layer just becomes pointless overhead. But doesn't the arm kernel have rather a lot more differences than merely understanding the arm binary format and cpu? I would have thought the container would have to run an x86_64 (or i386) binary, which would be qemu, and that qemu would have to run an arm kernel, and all other arm processes would have to run in that arm kernel. I think I need an example to illustrate a use case for this. -- bkw -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Container with different architecture like arm on x86 [How-to]
On 4/6/2011 5:30 PM, Justin Cormack wrote: On Wed, 2011-04-06 at 16:45 -0400, Brian K. White wrote: A few questions, The echo command references /usr/local/bin/qemu-arm, but I don't see that anywhere else in the recipe. Is that a x86_64 binary on the host or is that supposed to be a arm binary in the container, or is it simply ignored in this case and doesn't matter that it doesn't exist? It sort of looks like you are telling the host x86_64 kernel to run a x86_64 qemu-arm executable any time it encounters an arm elf executable, and then since you are placing an arm qemu-arm executable in the container fs I guess you are implying that the arm executable you will be trying to run will be that arm qemu executable? Why would you do that? foo - qemu - qemu - kernel ?? ie: arm-executable foo - arm executable qemu-arm - x86_64 executable qemu-arm - x86_64 host kernel ?? Assuming that even works. Doesn't there have to be an arm kernel in there somewhere? Like: arm-foo - arm-kernel - x86_64-qemu-arm - x86_64-host-kernel I don't see the point in this. As long as you have qemu in there anywhere it means you are doing full cpu virtualization, avoiding which is pretty much the sole purpose of containers. If it's really true that you can have qemu provide _only_ cpu virtualization yet somehow have the host kernel support the arm executables through that I guess that's a win since you have a single kernel doling out resources directly to all processes instead of kernels within kernels. Then again wouldn't that result in every single arm executable running inside it's own instance of qemu, auto launched by the binfmt? That might be ok for application containers that only run one process but that would be terrible for a full system container unless that container really only ran one process directly, an arm kernel. And in that case I don't see the point of doing that inside a container. It's already even more isolated inside qemu than what the container provides and the container layer just becomes pointless overhead. But doesn't the arm kernel have rather a lot more differences than merely understanding the arm binary format and cpu? I would have thought the container would have to run an x86_64 (or i386) binary, which would be qemu, and that qemu would have to run an arm kernel, and all other arm processes would have to run in that arm kernel. I think I need an example to illustrate a use case for this. Qemu is just being used as an arm instruction set interpreter, making x86 system calls to the native kernel. binfmt_misc lets you run other architecture binaries via emulation just by executing the binary. Obviously its slow, but if you want to build an arm distro say it gives another option other than cross compiling or a native compile on a slow machine. Justin Back to the first question, this actually works for binaries other than hello.c ? How many binaries live entirely within the high level calls that are really fully abstracted by the kernel? I guess I have to try this because I don't believe it. qemu just emulates hardware, as in the cpu and some of the supporting system. You can't run user executable on hardware. Only specially crafted free-standing things which is pretty much just bios/efi, bootloaders, kernels, and memtest86. Not ls for instance. I'm familiar with binfmt since I used to use iBCS and then linux-abi ages ago to run SCO binaries on linux, and similarly to run linux binaries within lxrun on SCO, and similarly osrcompat to run unixware binaries on open server and vice-versa, and linux on freebsd, etc ...) But in all those cases, the following always is true: * The executables have the same cpu instruction set as the host kernel. * The executables have the same endianness as the host kernel and filesystem and utilities. And at least one or more of the following is also always true: * The emulation layer explicitly goes to lengths to handle various unavoidable differences and conflicts. Like remapping syscalls that take different numbers or types of arguments, and exhibit different behavior, even though they are named the same and do nominally the same job. Faking various system level environmental things like fake device nodes, /proc entries, etc... maybe cpu registers or memory/io addresses too I don't know everything but I know it's not exactly trivial or ignorable. * The emulation layer provides fail-as-graceful-as-possible stubs for things that can't or haven't yet been really translated. * Users of the emulation layer simply know that only a small subset of things will actually work. Anything might have any sort of problem, and if it breaks you get to keep the pieces. It's useful in a few very special cases and requires significant hacks and workarounds and compromises, but isn't generally useful. I mean it's not just a few things, it's things everywhere you turn, filesystems that return values
