Re: [Lxc-users] read only rootfs
On Wed, 2011-07-20 at 07:10 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): [root@forest ~]# lxc-start --name Plover lxc-start: Invalid argument - pivot_root syscall failed sort of unrelated, but Rob Landley had mentioned he wanted to fix chroot to prevent the chdir-based chroot escape, allowing lxc to use chroot in place of pivot_root. As you see above, pivot_root has some very stringent constaints regarding the ms_shared state of the mounpoints *and* their parents, so this would be a very good thing. Interesting. Very interesting. Especially since I remember being involved in the whole transition from chroot over to pivot_root way back when. Explains a couples of other anomalies I observed. And would prevent the above. I'm going to have to refresh my memory of the reasons for switching from chroot over to pivot_root in the first place. There was more than one and the chdir-based chroot escape was only one. As far as the main topic of this thread, I feel I can't really do it justice without trying harder to reproduce, which I can't do today. I'm going to try and find time tomorrow or friday to do so (if you haven't gotten to the bottom of this before that). We also might want to point dhansen at the mail archive of this thread and get his input. Ok... I've still got some testing and experimenting to do based on those hints C Anthony gave me. ITMT, I'm not sure who you are referring to as dhansen (so I can't point them anywhere) but, the more the merrier if it helps this thing along. -serge Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- 10 Tips for Better Web Security Learn 10 ways to better secure your business today. Topics covered include: Web security, SSL, hacker attacks Denial of Service (DoS), private keys, security Microsoft Exchange, secure Instant Messaging, and much more. http://www.accelacomm.com/jaw/sfnl/114/51426210/___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Mon, Jul 18, 2011 at 7:36 AM, Serge E. Hallyn se...@hallyn.com wrote: Quoting C Anthony Risinger (anth...@xtfx.me): On Jul 15, 2011 12:01 PM, Michael H. Warfield m...@wittsend.com wrote: Unfortunately, I also still find that if there's a -o remount,ro in the halt/reboot script, it still sets /dev/pts to ro and that still propagates to the host and to the other containers triggering random acts of terrorism like unable to create pty/0 in the containers and inability to start new containers in the host. Not sure if we can apply a bind to that or not. Doesn't `-o newinstance` mount option to devpts mounts prevent this? It I haven't looked further than reading Michael's email, but a plausible sequence is that (a) the container's rootfs is just a bind mount from the parent's, not a good idea if so :-) (b) the mount -o remount,ro is not being done with 'bind' and so affects the fs, not the mount (as helpfully pointed out a few weeks ago on irc by dhansen), ahhh, i did not know it behaved like that ... hrmm. i knew you needed `-o bind` but i never even considered acting on the bind would *ever* affect the original mountpoint ... interesting, thanks. and so (c) the fs on which the host's /var/lib/lxc/container/rootfs is mounted gets recursively mounted ro, and the host's /dev/pts is under that. per your other comment, i didn't realize some distros --make-[r]shared / by default (or are mount points themselves shared by default? how to do that?) ... thats probably why i don't see this behavior; all my machines are Archlinux and we probably aren't changing the upstream defaults. should privatize the devices for each ... its best to mount host this way too -- then set symlink for each: /dev/ptmx - /dev/pts/ptmx The kernel should also prohibit, totally, the propagation of remount options from inside a container to the outer host or to other containers. That is tantamount to a security vulnerability and clearly a violation of container isolation. But not all use cases are system containers, eg 100% isolated. Isn't a slave mount enough to prevent this? I'd have to check but I *thought* bind mounts only responded to the `ro` flag ... and the new mount NS I'd think would play a role too ... not sure details offhand. See '(b)' above. You're sort of mixing mounts propagation with bind mounts subtleties. Your second sentence in that paragraph is 100% correct. yes i see that. pretty much all i know was from hours of trial an error understanding the semantics ... definitely a few gotchas in there it would seem. however, while i could *maybe* see the rootfs being an unconditional slave, i would NOT want to see any lxc default/enforcement preventing container - host propagation on a globally recursive scale. im of the opinion that the implementor should decide the best tactic ... especially in light of the fact the one distro may not even have the same problems as say ubutnu/fedora/etc because they keep mount points private by default. The third is non sequitur :) beh :-) fair enough. i've been known to ramble on ... slowly degrading toward incohesive mumblings of interwoven thoughts in real life, so i suppose this is the digital manifestation of these tendencies ... continues incoherently ... C Anthony -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
Quoting C Anthony Risinger (anth...@xtfx.me): there it would seem. however, while i could *maybe* see the rootfs being an unconditional slave, i would NOT want to see any lxc default/enforcement preventing container - host propagation on a globally recursive scale. im of the opinion that the implementor should decide the best tactic ... especially in light of the fact the one distro may not even have the same problems as say ubutnu/fedora/etc because they keep mount points private by default. Good point. (I don't see it on ubuntu either fwiw) Perhaps there should be a toggle in the per-container config file? -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Tue, 2011-07-19 at 13:34 -0500, Serge E. Hallyn wrote: Quoting C Anthony Risinger (anth...@xtfx.me): there it would seem. however, while i could *maybe* see the rootfs being an unconditional slave, i would NOT want to see any lxc default/enforcement preventing container - host propagation on a globally recursive scale. im of the opinion that the implementor should decide the best tactic ... especially in light of the fact the one distro may not even have the same problems as say ubutnu/fedora/etc because they keep mount points private by default. Good point. (I don't see it on ubuntu either fwiw) Perhaps there should be a toggle in the per-container config file? Quick question. Is there any way to test for these flags (SHARED, PRIVATE, SLAVE)? I don't see them showing up anywhere from mount, in proc mounts or mountstats. How do you check to see if they are set? Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
Quoting Michael H. Warfield (m...@wittsend.com): On Tue, 2011-07-19 at 13:34 -0500, Serge E. Hallyn wrote: Quoting C Anthony Risinger (anth...@xtfx.me): there it would seem. however, while i could *maybe* see the rootfs being an unconditional slave, i would NOT want to see any lxc default/enforcement preventing container - host propagation on a globally recursive scale. im of the opinion that the implementor should decide the best tactic ... especially in light of the fact the one distro may not even have the same problems as say ubutnu/fedora/etc because they keep mount points private by default. Good point. (I don't see it on ubuntu either fwiw) Perhaps there should be a toggle in the per-container config file? Quick question. Is there any way to test for these flags (SHARED, PRIVATE, SLAVE)? I don't see them showing up anywhere from mount, in proc mounts or mountstats. How do you check to see if they are set? /proc/self/mountinfo is supposed to tell that. i.e. if you do a --make-shared on /mnt, it'll show 'shared' next to the /mnt entry. (I say 'is supposed to' bc --make-rslave just shows nothing, but maybe that's bc the way i did it it wasn't a slave to anything, so it was actually private) -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Tue, 2011-07-19 at 16:50 -0400, Michael H. Warfield wrote: On Tue, 2011-07-19 at 15:32 -0500, Serge E. Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Tue, 2011-07-19 at 13:34 -0500, Serge E. Hallyn wrote: Quoting C Anthony Risinger (anth...@xtfx.me): there it would seem. however, while i could *maybe* see the rootfs being an unconditional slave, i would NOT want to see any lxc default/enforcement preventing container - host propagation on a globally recursive scale. im of the opinion that the implementor should decide the best tactic ... especially in light of the fact the one distro may not even have the same problems as say ubutnu/fedora/etc because they keep mount points private by default. Good point. (I don't see it on ubuntu either fwiw) Perhaps there should be a toggle in the per-container config file? Quick question. Is there any way to test for these flags (SHARED, PRIVATE, SLAVE)? I don't see them showing up anywhere from mount, in proc mounts or mountstats. How do you check to see if they are set? /proc/self/mountinfo is supposed to tell that. i.e. if you do a --make-shared on /mnt, it'll show 'shared' next to the /mnt entry. (I say 'is supposed to' bc --make-rslave just shows nothing, but maybe that's bc the way i did it it wasn't a slave to anything, so it was actually private) Ok... This may be telling us something. What? Oh, meant to include the info on / on each, sorry... On the host Forest: 22 1 8:5 / / rw,relatime - ext4 /dev/sda5 rw,barrier=1,data=ordered [root@forest ~]# cat /proc/self/mountinfo | grep export 50 22 8:18 / /export rw,relatime - ext4 /dev/sdb2 rw,barrier=1,data=ordered [root@forest ~]# mount --make-shared /export [root@forest ~]# cat /proc/self/mountinfo | grep export 50 22 8:18 / /export rw,relatime shared:1 - ext4 /dev/sdb2 rw,barrier=1,data=ordered [root@forest ~]# mount --make-slave /export [root@forest ~]# cat /proc/self/mountinfo | grep export 50 22 8:18 / /export rw,relatime - ext4 /dev/sdb2 rw,barrier=1,data=ordered [root@forest ~]# mount --make-private /export [root@forest ~]# cat /proc/self/mountinfo | grep export 50 22 8:18 / /export rw,relatime - ext4 /dev/sdb2 rw,barrier=1,data=ordered So, shared looks like it worked and the other two didn't? Does something have to be done to enable them? You say maybe that's bc the way i did it it wasn't a slave to anything meaning we're missing a step. What's the missing step. On the guest Alcove (with your patch to add MS_REC | MS_SLAVE): 105 55 8:17 /lxc/private/Alcove / rw,relatime - ext4 /dev/sdb1 rw,barrier=1,data=ordered [root@alcove mhw]# cat /proc/self/mountinfo | grep devpts 107 105 0:10 /6 /dev/console rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 108 105 0:10 /0 /dev/tty1 rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 109 105 0:10 /1 /dev/tty2 rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 110 105 0:10 /2 /dev/tty3 rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 111 105 0:10 /3 /dev/tty4 rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 112 105 0:10 /4 /dev/tty5 rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 113 105 0:10 /5 /dev/tty6 rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 56 105 0:44 / /dev/pts rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 64 56 0:10 / /dev/pts rw,relatime - devpts devpts rw,mode=600,ptmxmode=666 I'd say that's not good. Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Tue, Jul 19, 2011 at 4:17 PM, Michael H. Warfield m...@wittsend.com wrote: On Tue, 2011-07-19 at 15:32 -0500, Serge E. Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Tue, 2011-07-19 at 13:34 -0500, Serge E. Hallyn wrote: Quoting C Anthony Risinger (anth...@xtfx.me): there it would seem. however, while i could *maybe* see the rootfs being an unconditional slave, i would NOT want to see any lxc default/enforcement preventing container - host propagation on a globally recursive scale. im of the opinion that the implementor should decide the best tactic ... especially in light of the fact the one distro may not even have the same problems as say ubutnu/fedora/etc because they keep mount points private by default. Good point. (I don't see it on ubuntu either fwiw) Perhaps there should be a toggle in the per-container config file? Quick question. Is there any way to test for these flags (SHARED, PRIVATE, SLAVE)? I don't see them showing up anywhere from mount, in proc mounts or mountstats. How do you check to see if they are set? /proc/self/mountinfo is supposed to tell that. i.e. if you do a --make-shared on /mnt, it'll show 'shared' next to the /mnt entry. (I say 'is supposed to' bc --make-rslave just shows nothing, but maybe that's bc the way i did it it wasn't a slave to anything, so it was actually private) Ok... This just gets weirder. For giggles, I set my /srv partition (where all my VM's are located) to shared. Now. the first machine starts up fine but the second one, Plover, and all subsequent ones blow up with this: [root@forest ~]# lxc-start --name Plover lxc-start: Invalid argument - pivot_root syscall failed lxc-start: failed to setup pivot root lxc-start: failed to set rootfs for 'Plover' lxc-start: failed to setup the container lxc-start: invalid sequence number 1. expected 2 lxc-start: failed to spawn 'Plover' lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/Plover' And mount -t devpts shows ALL the devpts mounts for all the attempted VM's. Ok... Guess that wasn't a good idea. But... I got this for the root system on Alcove. 106 55 8:17 /lxc/private/Alcove / rw,relatime master:1 - ext4 /dev/sdb1 rw,barrier=1,data=ordered Ok... That now says master:1. Not sure what it signifies... Shut him down and changed /srv to be slave and all the containers come up but the remount still propagates back. Changed ran --make-rslave on it and no influence. Seems like we're missing a piece of the puzzle here. maybe not the best context for this response, but i wanted to point out one thing that confused me for awhile since it might be related ... ... that fact that the shared/slave context only exists with BOTH sides are mount points. eg. if DIR is only a directory: mount --bind ./DIR ./TARGET ... it will never propagate mounts to TARGET (AFAICT), and does not respond to --make-* ... before OR after the --bind. in order to get propagation, one must: mount --bind ./DIR ./DIR mount --make-shared ./DIR mount --bind ./DIR ./TARGET [mount --make-slave ./TARGET] ... this tripped me up for awhile as it seemed like the semantics were changing. C Anthony -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
(sorry, just realized postfix has been messing up my email, hope this comes through ok) Quoting C Anthony Risinger (anth...@xtfx.me): On Jul 15, 2011 12:01 PM, Michael H. Warfield m...@wittsend.com wrote: Unfortunately, I also still find that if there's a -o remount,ro in the halt/reboot script, it still sets /dev/pts to ro and that still propagates to the host and to the other containers triggering random acts of terrorism like unable to create pty/0 in the containers and inability to start new containers in the host. Not sure if we can apply a bind to that or not. Doesn't `-o newinstance` mount option to devpts mounts prevent this? It I haven't looked further than reading Michael's email, but a plausible sequence is that (a) the container's rootfs is just a bind mount from the parent's, (b) the mount -o remount,ro is not being done with 'bind' and so affects the fs, not the mount (as helpfully pointed out a few weeks ago on irc by dhansen), and so (c) the fs on which the host's /var/lib/lxc/container/rootfs is mounted gets recursively mounted ro, and the host's /dev/pts is under that. should privatize the devices for each ... its best to mount host this way too -- then set symlink for each: /dev/ptmx - /dev/pts/ptmx The kernel should also prohibit, totally, the propagation of remount options from inside a container to the outer host or to other containers. That is tantamount to a security vulnerability and clearly a violation of container isolation. But not all use cases are system containers, eg 100% isolated. Isn't a slave mount enough to prevent this? I'd have to check but I *thought* bind mounts only responded to the `ro` flag ... and the new mount NS I'd think would play a role too ... not sure details offhand. See '(b)' above. You're sort of mixing mounts propagation with bind mounts subtleties. Your second sentence in that paragraph is 100% correct. The third is non sequitur :) See the patch I just sent in response to Michael's email. thanks, -serge -- Storage Efficiency Calculator This modeling tool is based on patent-pending intellectual property that has been used successfully in hundreds of IBM storage optimization engage- ments, worldwide. Store less, Store more with what you own, Move data to the right place. Try It Now! http://www.accelacomm.com/jaw/sfnl/114/51427378/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Mon, 2011-07-04 at 22:16 +0200, Matto Fransen wrote: Hi, On Mon, Jun 27, 2011 at 06:05:13PM +0200, Samuel Maftoul wrote: I'm searching for a solution to have a read only rootfs inside an LXC container. I have a webserver running this way :) I created a container with the busybox template, this container works. As soon as I try to mount it read only I have this message in the logs: Create a rootfs outside the container. In the config of your container you add lines like: lxc.mount.entry=/path/to/rootfs/lib /var/lib/lxc/container/rootfs/lib none ro,bind 0 0 and so on for all the dir's you want to mount readonly Also create some system directories: # system mounts lxc.mount.entry=proc /var/lib/lxc/container/rootfs/proc proc none defaults 0 0 lxc.mount.entry=shmfs /var/lib/lxc/container/rootfs/dev/shm tmpfs mode=0644 0 0 lxc.mount.entry=sysfs /var/lib/lxc/container/rootfs/sys sysfs defaults 0 0 I find the bind mount is providing protection for propagating mount point option changes from the guest to the host or to other guests. That's good. Unfortunately, I also still find that if there's a -o remount,ro in the halt/reboot script, it still sets /dev/pts to ro and that still propagates to the host and to the other containers triggering random acts of terrorism like unable to create pty/0 in the containers and inability to start new containers in the host. Not sure if we can apply a bind to that or not. I've been commenting those remounts out of the halt and reboot scripts but I recently upgraded some containers from F12 to F13 and then to F14 and that installed new scripts. Since I was commenting out those lines automatically when I started the containers, I didn't catch it and it clobbered the host. Fortunately, I caught it and just did a mount -o remount,rw /dev/pts in the host and everything was back to normal. Guess I'll do a preshutdown script that runs in the containers to catch that and prevent it but it doesn't change the fact a the container can have a serious impact on the host. And add the following line to the config of your container: lxc.cap.drop=sys_admin Doing this also triggers random acts of terrorism in my containers. The sys_admin cap is far to broad and affects far to many other things like setting the host names. It would also prevent using the loopback device and setting crypto keys on loopback devices and prevent mounting other file systems like iso images and encrypted file systems or nfs file systems from within the container. I can only assume that autofs would be totally broken. This last line prevents that one can jumo out of the readonly bind mounts from inside the container :) We need a better way than that. This needs kernel work in the container logic but, if something is mounted ro outside the container, the container should treat it as ro hardware and disallow rw remounting. That's one thing. The kernel should also prohibit, totally, the propagation of remount options from inside a container to the outer host or to other containers. That is tantamount to a security vulnerability and clearly a violation of container isolation. It amounts to a semi-local Denial of Service (DoD) vulnerability (semi-local because it's local to that machine and its constellation of guests but it propagates between guests). I don't see any clean fix for this that doesn't involve creating other problems (like dropping sys_admin) which violates the ability to virtualize the machines properly. Cheers, Matto -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev___ Lxc-users mailing list Lxc-users@lists.sourceforge.net
Re: [Lxc-users] read only rootfs
On Jul 15, 2011 12:01 PM, Michael H. Warfield m...@wittsend.com wrote: Unfortunately, I also still find that if there's a -o remount,ro in the halt/reboot script, it still sets /dev/pts to ro and that still propagates to the host and to the other containers triggering random acts of terrorism like unable to create pty/0 in the containers and inability to start new containers in the host. Not sure if we can apply a bind to that or not. Doesn't `-o newinstance` mount option to devpts mounts prevent this? It should privatize the devices for each ... its best to mount host this way too -- then set symlink for each: /dev/ptmx - /dev/pts/ptmx The kernel should also prohibit, totally, the propagation of remount options from inside a container to the outer host or to other containers. That is tantamount to a security vulnerability and clearly a violation of container isolation. But not all use cases are system containers, eg 100% isolated. Isn't a slave mount enough to prevent this? I'd have to check but I *thought* bind mounts only responded to the `ro` flag ... and the new mount NS I'd think would play a role too ... not sure details offhand. C Anthony [mobile] -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
Hi, Thanks all of you, I managed to make it work ! I understand there are some security concerns, and Matto, you're pointing to a very interesting detail, dropping capability is really what I want ! Thanks -- Samuel On Mon, Jul 4, 2011 at 10:16 PM, Matto Fransen ma...@matto.nl wrote: Hi, On Mon, Jun 27, 2011 at 06:05:13PM +0200, Samuel Maftoul wrote: I'm searching for a solution to have a read only rootfs inside an LXC container. I have a webserver running this way :) I created a container with the busybox template, this container works. As soon as I try to mount it read only I have this message in the logs: Create a rootfs outside the container. In the config of your container you add lines like: lxc.mount.entry=/path/to/rootfs/lib /var/lib/lxc/container/rootfs/lib none ro,bind 0 0 and so on for all the dir's you want to mount readonly Also create some system directories: # system mounts lxc.mount.entry=proc /var/lib/lxc/container/rootfs/proc proc none defaults 0 0 lxc.mount.entry=shmfs /var/lib/lxc/container/rootfs/dev/shm tmpfs mode=0644 0 0 lxc.mount.entry=sysfs /var/lib/lxc/container/rootfs/sys sysfs defaults 0 0 And add the following line to the config of your container: lxc.cap.drop=sys_admin This last line prevents that one can jumo out of the readonly bind mounts from inside the container :) Cheers, Matto -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
Hi, On Mon, Jun 27, 2011 at 06:05:13PM +0200, Samuel Maftoul wrote: I'm searching for a solution to have a read only rootfs inside an LXC container. I have a webserver running this way :) I created a container with the busybox template, this container works. As soon as I try to mount it read only I have this message in the logs: Create a rootfs outside the container. In the config of your container you add lines like: lxc.mount.entry=/path/to/rootfs/lib /var/lib/lxc/container/rootfs/lib none ro,bind 0 0 and so on for all the dir's you want to mount readonly Also create some system directories: # system mounts lxc.mount.entry=proc /var/lib/lxc/container/rootfs/proc proc none defaults 0 0 lxc.mount.entry=shmfs /var/lib/lxc/container/rootfs/dev/shm tmpfs mode=0644 0 0 lxc.mount.entry=sysfs /var/lib/lxc/container/rootfs/sys sysfs defaults 0 0 And add the following line to the config of your container: lxc.cap.drop=sys_admin This last line prevents that one can jumo out of the readonly bind mounts from inside the container :) Cheers, Matto -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Mon, 2011-06-27 at 18:05 +0200, Samuel Maftoul wrote: I tried several ways to have the rootfs mounted RO. First I removed the lxc.rootfs from my config file and the tried: - lxc-start -n vm0 -o /tmp/lxc-vm0.log -l DEBUG -s lxc.mount.entry=/ /var/lib/lxc/vm0/rootfs none ro,bind 0 0 Then I tried: - echo / /var/lib/lxc/vm0/rootfs none ro,bind 0 0 /var/lib/lxc/vm0/fstab ; lxc-start -n vm0 -o /tmp/lxc-vm0.log -l DEBUG -s lxc.mount = /var/lib/lxc/vm0/fstab Finally I tried to boot with lxc.rootfs pointing to the same content, but on it's block device, mounted read-only The system starts, I have a console, but in the logs I get: lxc_conf - ignoring mount point '/var/lib/lxc/vm0/rootfs/lib' lxc_conf - ignoring mount point '/var/lib/lxc/vm0/rootfs/usr/lib' and of course, If I ls these directories, I have nothing inside. Bind mounting the root fs is fine, but it will not bind mount file systems under this, so you will need to add these to your fstab too. It looks like you have /lib and /usr/lib mounted on separate file systems and need to bind mount these too? Justin -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Mon, 2011-06-27 at 17:20 +0100, Justin Cormack wrote: On Mon, 2011-06-27 at 18:05 +0200, Samuel Maftoul wrote: I tried several ways to have the rootfs mounted RO. First I removed the lxc.rootfs from my config file and the tried: - lxc-start -n vm0 -o /tmp/lxc-vm0.log -l DEBUG -s lxc.mount.entry=/ /var/lib/lxc/vm0/rootfs none ro,bind 0 0 Then I tried: - echo / /var/lib/lxc/vm0/rootfs none ro,bind 0 0 /var/lib/lxc/vm0/fstab ; lxc-start -n vm0 -o /tmp/lxc-vm0.log -l DEBUG -s lxc.mount = /var/lib/lxc/vm0/fstab Finally I tried to boot with lxc.rootfs pointing to the same content, but on it's block device, mounted read-only The system starts, I have a console, but in the logs I get: lxc_conf - ignoring mount point '/var/lib/lxc/vm0/rootfs/lib' lxc_conf - ignoring mount point '/var/lib/lxc/vm0/rootfs/usr/lib' and of course, If I ls these directories, I have nothing inside. Bind mounting the root fs is fine, but it will not bind mount file systems under this, so you will need to add these to your fstab too. It looks like you have /lib and /usr/lib mounted on separate file systems and need to bind mount these too? Bind mounts work but, iirc, there was (in the past) a problem that if the container did a remount, the remount would propagate to the parent device. That caused all sorts of headaches (and I know, I was suppose to retest that scenario ages ago and I haven't) like when a container remounted its rootfs ro during a shutdown it made partitions ro to the host. Very bad. This was also at the heart of the problem with shutdowns causing ptty failures for any subsequent connections an container starts (it made that fs ro). If you try to do this, you may have to prohibit mounts inside the containers to prohibit the remount problems. It would probably be a good idea to test it and see if the container can remount an ro mount point as rw and what the impact would be. Justin Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] read only rootfs
On Mon, 2011-06-27 at 12:33 -0500, C Anthony Risinger wrote: On Mon, Jun 27, 2011 at 12:06 PM, Michael H. Warfield m...@wittsend.com wrote: On Mon, 2011-06-27 at 17:20 +0100, Justin Cormack wrote: On Mon, 2011-06-27 at 18:05 +0200, Samuel Maftoul wrote: I tried several ways to have the rootfs mounted RO. First I removed the lxc.rootfs from my config file and the tried: - lxc-start -n vm0 -o /tmp/lxc-vm0.log -l DEBUG -s lxc.mount.entry=/ /var/lib/lxc/vm0/rootfs none ro,bind 0 0 Then I tried: - echo / /var/lib/lxc/vm0/rootfs none ro,bind 0 0 /var/lib/lxc/vm0/fstab ; lxc-start -n vm0 -o /tmp/lxc-vm0.log -l DEBUG -s lxc.mount = /var/lib/lxc/vm0/fstab Finally I tried to boot with lxc.rootfs pointing to the same content, but on it's block device, mounted read-only The system starts, I have a console, but in the logs I get: lxc_conf - ignoring mount point '/var/lib/lxc/vm0/rootfs/lib' lxc_conf - ignoring mount point '/var/lib/lxc/vm0/rootfs/usr/lib' and of course, If I ls these directories, I have nothing inside. Bind mounting the root fs is fine, but it will not bind mount file systems under this, so you will need to add these to your fstab too. It looks like you have /lib and /usr/lib mounted on separate file systems and need to bind mount these too? Bind mounts work but, iirc, there was (in the past) a problem that if the container did a remount, the remount would propagate to the parent device. That caused all sorts of headaches (and I know, I was suppose to retest that scenario ages ago and I haven't) like when a container remounted its rootfs ro during a shutdown it made partitions ro to the host. Very bad. This was also at the heart of the problem with shutdowns causing ptty failures for any subsequent connections an container starts (it made that fs ro). If you try to do this, you may have to prohibit mounts inside the containers to prohibit the remount problems. It would probably be a good idea to test it and see if the container can remount an ro mount point as rw and what the impact would be. does this happen when the container rootfs is marked as a slave/private mount? slaves et al should not propagate changes to the master/host. That's exactly the thing that needs to be tested. I don't know at this point but I do know at one point it did not work properly and it did propagate. Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
