Re: resizing rootfs to fit the disk
Hi, if we (or someone else :-) can convince ourselves that the online resize is really no less risky than simply using the filesystem normally (i.e., if power fails, will the ext3 journal fix things as well as it ever does?), then there's no real reason to do the resize while the laptop is still booting. we can just as easily do it _after_ it has booted, while it's in use. I asked some kernel folks on IRC about this: cjb sandeen, etc: Do you know whether ext3 online-resize (in the kernel) is power-loss safe? cjb as in journal-recoverable or something. josef cjb: yeah its all journaled sandeen cjb, it probably is, yeah sandeen nothing really happens that's critical until the end sandeen oh, but actually it does log each piece sandeen in fact there is a crapton of journal resizing and restarting as it goes * sandeen remembers that bug ... cjb does that make it less safe? josef cjb: no just a pain sandeen i'm not sure why it is so slow, I need to look into that some day :( sandeen it does have to initialize inode block tables So, I'd say go ahead with this plan -- resize on first boot, but after userspace is up and running (and if it fails, try it again until it succeeds?). And, of course, we should test power loss during resize some more. Thanks, - Chris. -- Chris Ball c...@laptop.org One Laptop Per Child ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
On Wed, Mar 10, 2010 at 10:52 PM, Chris Ball c...@laptop.org wrote: So, I'd say go ahead with this plan -- resize on first boot, but after userspace is up and running (and if it fails, try it again until it succeeds?). And, of course, we should test power loss during resize some more. Sounds reasonable to me. - There should be a flag for devs / deployment teams to disable it. - With some added bundles, there is the risk that our Journal-is-full warning might appear on first boot; maybe it should not appear or use a lower threshold until the resize complete flag is set. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
martin wrote: On Wed, Mar 10, 2010 at 10:52 PM, Chris Ball c...@laptop.org wrote: So, I'd say go ahead with this plan -- resize on first boot, but after userspace is up and running (and if it fails, try it again until it succeeds?). And, of course, we should test power loss during resize some more. Sounds reasonable to me. - There should be a flag for devs / deployment teams to disable it. martin -- can you remind me of the use case for this flag? and, since i'm confident you'll make a very compelling case, can you tell me where such a flag should appear, and/or what it should look like? - With some added bundles, there is the risk that our Journal-is-full warning might appear on first boot; maybe it should not appear or use a lower threshold until the resize complete flag is set. i'd say that if they're that close to filling the disk, the filesystem size should be made larger at build time. the only reason not to do that is if the target disk is very small, and if it's that small, then they've added too many bundles. paul =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
On Wed, Mar 10, 2010 at 5:55 PM, Paul Fox p...@laptop.org wrote: martin -- can you remind me of the use case for this flag? I've right-sized my OS image and would like to have one less moving thing. Deployment teams can probably just set the we're resized already -- so it doesn't actually require extra work... - With some added bundles, there is the risk that our Journal-is-full warning might appear on first boot; maybe it should not appear or use a lower threshold until the resize complete flag is set. i'd say that if they're that close to filling the disk, the filesystem size should be made larger at build time. the only reason not to do that is if the target disk is very small, and if it's that small, then they've added too many bundles. Chris wants to generate a single image that is chock-full with as many sample activities as we can fit. It is a veritable squeeze for the sub-2GB image and will probably trigger the warning. And I forgot another To Do: add a big note in the release notes explaining why the output of df is unreliable and variable until the resize completes. It's bound to surprise a few people ;-) m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
martin wrote: On Wed, Mar 10, 2010 at 5:55 PM, Paul Fox p...@laptop.org wrote: martin -- can you remind me of the use case for this flag? I've right-sized my OS image and would like to have one less moving thing. Deployment teams can probably just set the we're resized already -- so it doesn't actually require extra work... you're assuming there needs to be such a flag. attempting to resize an already right-sized filesystem probably takes less time than checking a flag, so i wasn't going to add one if i didn't need to. but you're saying i need to, so i guess i will. but again: where should such a flag go? is there a precedent for such things? - With some added bundles, there is the risk that our Journal-is-full warning might appear on first boot; maybe it should not appear or use a lower threshold until the resize complete flag is set. i'd say that if they're that close to filling the disk, the filesystem size should be made larger at build time. the only reason not to do that is if the target disk is very small, and if it's that small, then they've added too many bundles. Chris wants to generate a single image that is chock-full with as many sample activities as we can fit. It is a veritable squeeze for the sub-2GB image and will probably trigger the warning. i'm not sure you've changed my argument. you've just pointed out that i should have it with chris, too. ;-) And I forgot another To Do: add a big note in the release notes explaining why the output of df is unreliable and variable until the resize completes. It's bound to surprise a few people ;-) bah. it's a learning experience. paul =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
On Wed, Mar 10, 2010 at 6:27 PM, Paul Fox p...@laptop.org wrote: you're assuming there needs to be such a flag. attempting to resize an already right-sized filesystem probably takes less time than checking a flag Not only time / cpu burn but... I'd say it's less risky... once resize completed with a sane exit code, mark it as done and stop running resize every boot. but again: where should such a flag go? is there a precedent for such things? /.i-am-a-hidden-flag ? Chris wants to generate a single image that is chock-full with as many sample activities as we can fit. It is a veritable squeeze for the sub-2GB image and will probably trigger the warning. i'm not sure you've changed my argument. you've just pointed out that i should have it with chris, too. ;-) No... I think that what Chris needs to do -- a stock image that fits in slightly less than 2GB -- is a reasonable use case that is not the mainstream case. It is good for developers, and for limited techteam deployments. But it barely fits on a 2GB disk and the partition size we ship it in will have it pretty tight. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
but again: where should such a flag go? is there a precedent for such things? /.i-am-a-hidden-flag ? Safety would argue for doing the opposite: * Build the installation images to contain a file like /.resize-root-once. * If that file is present ** Remove the file ** Sync the disk ** Attempt the online resize ** Sync the disk again * If that file is not present ** Do nothing. Leave the filesystem alone! The idea of a Linux system that, when it boots, resizes the root filesystem every time to try to fill up the entire drive it's on, fills me with horror. I prefer to partition my disks myself, thank you. John ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
john wrote: but again: where should such a flag go? is there a precedent for such things? /.i-am-a-hidden-flag ? Safety would argue for doing the opposite: * Build the installation images to contain a file like /.resize-root-once. * If that file is present ** Remove the file ** Sync the disk ** Attempt the online resize ** Sync the disk again * If that file is not present ** Do nothing. Leave the filesystem alone! The idea of a Linux system that, when it boots, resizes the root filesystem every time to try to fill up the entire drive it's on, fills me with horror. :-) well, that was certainly never my intention. I prefer to partition my disks myself, thank you. i take your point, and martin's. a flag file of some sort it is, then. paul =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
james wrote: On Fri, Mar 05, 2010 at 01:39:23AM -0500, Paul Fox wrote: resize drags performance way down. but nice -19 resize2fs /dev/mmcblk0p2 only adds about 25% to the runtime (~50sec vs. ~40sec), Consider ionice for reducing the I/O priority of the task. e.g. ionice -c3 resize2fs ... Also, did you get far with the kernel resize code? Doing it in the kernel seems tidier, and the code seems to be there. One remounts with resize option. i've not been able to get mount -o remount,resize to work: either i'm doing something wrong, or something's misconfigured, or, well, or something. however when resize2fs operates in online mode, i believe it's using the kernel code -- it sort of has to. looking at resize/online.c in the e2fsprogs source tree, it seems most of the work is done by a few ioctls, which call the same code that would used by mount (via fs/ext3/super.c and fs/ext3/resize.c). looking at the e2fsprogs changelog, there have been some bugs fixed since the 1.41.4 version we're using, though not surprisingly they mostly seem to be related to ext4 support, and to the offline version of the code. i haven't looked at the kernel tree to see what fixes might have gone into the kernel resize code since 2.6.31.6. paul =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
On Fri, Mar 5, 2010 at 1:39 AM, Paul Fox p...@laptop.org wrote: i.e., if power fails, will the ext3 journal fix things as well as it ever does? That would be my question... time to... yank that powercord! m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
martin wrote: On Fri, Mar 5, 2010 at 1:39 AM, Paul Fox p...@laptop.org wrote: i.e., if power fails, will the ext3 journal fix things as well as it ever does? That would be my question... time to... yank that powercord! i've only tried it once. when the machine rebooted, i was left with a filesystem that was bigger than it started, but not as big as it could be. re-running the resize finished the job. more testing needed, clearly. paul =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
On Fri, Mar 05, 2010 at 08:55:26AM -0500, Martin Langhoff wrote: On Fri, Mar 5, 2010 at 1:39 AM, Paul Fox p...@laptop.org wrote: i.e., if power fails, will the ext3 journal fix things as well as it ever does? That would be my question... time to... yank that powercord! I've got a desktop controlled power relay, and with the AP tag, and a careful serial script, I could try to automate the sequence; - fs-update - bye - resize in a root shell, - turn power off at certain time, ... and then vary the certain time. -- James Cameron http://quozl.linux.org.au/ ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
i'm moving a discussion-in-progress to the devel list. as background: because XO-1.5 uses an internal micro-SD for mass storage, we don't know the exact size filesystem to create at image build time. not only do we ship laptops with (currently) 2G and 4G cards, but cards of a quoted size from different vendors vary in useable size (by quite a bit). so we currently build otherwise-identical 2G and 4G images, both undersized to accomodate vendor variations. we'd prefer not to have to do this -- it would make the builds easier, and would let users use any size card they'd like. thus far in our story: - several people think we should be trying to do a full filesystem resize at boot -- advantage is that we can ship a single image, and have it work on all SD cards. it would require a new 'please wait' splash screen (which should be a good one, with seamless transition -- this may be the first user experience), and adds some risk to the boot process. first linux boot happens at the factory for new laptops, but in the child's hands in most deployment scenarios (where nandblaster or usb sticks are used). - several people think we should separate the base OS from the user data, by giving /home a separate, variable, partition. we would then build images to match a fixed root fs size. would we size the root fs? having a hard constraint that we must fit into wouldn't be bad thing, as long as we don't pick an unreasonably small value, and as long as there's a fallback plan when we blow it. olpc-update may double the space requirement in the worst case, right? i've never used LVM -- could it help make the boundary moveable? - we could put resize-at-boot code in place (with or without a splash screen), and continue shipping multiple, somewhat undersized images. any image could be used on any bigger SD card, and the right thing would happen, but in the usual case where an image is correctly sized, the resize time wouldn't take too long. perhaps we could suppress the splash screen for resizes that we think won't take too long? - we could make the resizing a manual step, from the control panel or elsewhere. we sort of need a simple disk management screen anyway. (btw, regardless of our decision, this is clearly more than a simple bug fix.) paul =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
[ some of these are repeats from the private discussion] On Thu, Mar 4, 2010 at 2:06 PM, Chris Ball c...@laptop.org wrote: I still like the flexibility here, but I suspect you're right that the support cost from doing a long resize at the first boot in front of a user (and having corruption result from a poweroff) will be too high. From a deployment-support point of view, *any* boot-time resize we do must be complemented by a utility to resize the .zd image . A deployment team knows what disk size they have (even if they have miniSD disks from various vendors, they can find out the lowest common denominator), and the advantage of skipping the resize step can be important... specially if we don't do partitioning. - several people think we should separate the base OS from the user data, by giving /home a separate partition. how would I don't think we should add this feature into the mix yet I don't like partitioning either -- cons: - We don't have a good resize plan: LVM+ext3 resizes won't work online, so we'll have to do it from OFW or from the initramfs. And LVM+ext3 resizes are fragile as hell, specially in the face of powerloss. - Will break systems when one partition is full, with little clarity as to what's happening. For example, yum can consume a ton of diskspace in /var/cache/yum to run an upgrade that leads to a similar-sized OS footprint. - Having a tight / breaks _both_ olpc-update and yum. It effectively cuts all upgrade options except fs-update / nandblast. - Our OSs are effectively OS+Apps -- and the Activities part of it lands in /home, so the desired protect /home/ when upgrading via fs-updatte / nandblast isn't actually achieved. Or will at least need extra elbow grease and storage in / (and some bundles can be huge - ie:Wikipedia.xo). pros: - makes first-boot resize super fast and resilient -- because it becomes firstboot mkfs /home - it could protect /home during upgrades if... am... lots of other things were solved :-) -- _if they can be solved so that they work with a tight / partition_. In summary, I suspect the main desired benefit of a partitioned /home is too hard to reach. - we could put resize-at-boot code in place (with or without a splash screen), and ship multiple, somewhat undersized images. ... ... - we could make the resizing a manual step, from the control panel or elsewhere. we sort of need a simple disk management screen anyway. Valid ideas. I think it would be smart to separate our use cases -- - Medium-to-large deployments -- give them the ability to make nearly-right-sized images easily, so nandblasting and usb-based reflashing don't require a slow brittle resize-on-first-boot. - Small deployments, developers, testers, enthusiasts, demo machines -- can handle slightly undersized images without ever worrying about it. For them, provide a commandline (`touch /resizeme`?) that on boot triggers the on-boot-resize codepath -- to be used by advanced enthusiasts/testers and smallish deployments (that often have some geek wisdom but aren't hardcore enough to make their own OS images). For completeness, an undercurrent here is mid/large deployments must have an XS performing backup of the Journal data. Yes they should, I'll add my signature to the petition! But fs-update-and-restore-your-journal-data can't be the only way to upgrade your XOs. Removing options (yum, olpc-update) won't be popular with people in the field. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
i wrote: i'm moving a discussion-in-progress to the devel list. as background: because XO-1.5 uses an internal micro-SD for mass storage, we don't know the exact size filesystem to create at image build time. not only do we ship laptops with (currently) 2G and 4G cards, but cards of a quoted size from different vendors vary in useable size (by quite a bit). so we currently build otherwise-identical 2G and 4G images, both undersized to accomodate vendor variations. we'd prefer not to have to do this -- it would make the builds easier, and would let users use any size card they'd like. thus far in our story: i thought of one more possibility, later this afternoon, which is pretty interesting. if we (or someone else :-) can convince ourselves that the online resize is really no less risky than simply using the filesystem normally (i.e., if power fails, will the ext3 journal fix things as well as it ever does?), then there's no real reason to do the resize while the laptop is still booting. we can just as easily do it _after_ it has booted, while it's in use. i've tried this -- left on its own, the resize drags performance way down. but nice -19 resize2fs /dev/mmcblk0p2 only adds about 25% to the runtime (~50sec vs. ~40sec), and keeps the system reasonably responsive in the meantime. more testing, and conversations with ext3 experts, would clearly be needed before thinking of commiting to this. paul - several people think we should be trying to do a full filesystem resize at boot -- advantage is that we can ship a single image, and have it work on all SD cards. it would require a new 'please wait' splash screen (which should be a good one, with seamless transition -- this may be the first user experience), and adds some risk to the boot process. first linux boot happens at the factory for new laptops, but in the child's hands in most deployment scenarios (where nandblaster or usb sticks are used). - several people think we should separate the base OS from the user data, by giving /home a separate, variable, partition. we would then build images to match a fixed root fs size. would we size the root fs? having a hard constraint that we must fit into wouldn't be bad thing, as long as we don't pick an unreasonably small value, and as long as there's a fallback plan when we blow it. olpc-update may double the space requirement in the worst case, right? i've never used LVM -- could it help make the boundary moveable? - we could put resize-at-boot code in place (with or without a splash screen), and continue shipping multiple, somewhat undersized images. any image could be used on any bigger SD card, and the right thing would happen, but in the usual case where an image is correctly sized, the resize time wouldn't take too long. perhaps we could suppress the splash screen for resizes that we think won't take too long? - we could make the resizing a manual step, from the control panel or elsewhere. we sort of need a simple disk management screen anyway. (btw, regardless of our decision, this is clearly more than a simple bug fix.) paul =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel =- paul fox, p...@laptop.org ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: resizing rootfs to fit the disk
On Fri, Mar 05, 2010 at 01:39:23AM -0500, Paul Fox wrote: resize drags performance way down. but nice -19 resize2fs /dev/mmcblk0p2 only adds about 25% to the runtime (~50sec vs. ~40sec), Consider ionice for reducing the I/O priority of the task. e.g. ionice -c3 resize2fs ... Also, did you get far with the kernel resize code? Doing it in the kernel seems tidier, and the code seems to be there. One remounts with resize option. -- James Cameron http://quozl.linux.org.au/ ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
