Re: Deployment image customization
Michael Stone wrote: > On Tue, Dec 23, 2008 at 03:29:08PM -0500, p...@laptop.org wrote: > >> daniel wrote: >> >>> On Tue, Dec 23, 2008 at 8:08 PM, wrote: >>> ssh host keys are probably generated on first boot as well. with partitioning support, it should be possible to have a r.o. root overlaid by a unionfs writeable mount, so machine-specific changes don't modify the released partition. this would make cloning quite a bit easier, i'd think. i have no idea what the performance hit of a unionfs setup would be, nor how such a partitioning would fit into the rest of the update strategy (e.g. olpc-update). >>> unionfs isn't upstream and was quite unreliable last time I use it. >>> And it adds the challenge of differentiating state that must be >>> discarded for the cloned image, and state that must not be. >>> >>> For example, we would want to ssh keys generated during first boot to >>> *not* be included in the clonable image, that's obvious. But if the >>> user boots the OLPC image, goes into the control panel and sets a >>> language, then we *do* want that language change to be included in the >>> clonable image that is the output of the process. >>> >>> How would the system differentiate between those two? >>> >> i dunno. i guess the lead engineer on the project would have to >> decide. :-) >> > > In my opinion, the simplest way to approach this is to add a > "hard-reset" script (perhaps named "olpc-hard-reset") which cleans up > the image and then prints out a diff from the starting image to the > result for manual review. > > Michael > > > See the following for a similar facility that Sun has used for longer than I can remember: http://docs.sun.com/app/docs/doc/802-1930-1M/6i5u98eaf?a=view ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 03:29:08PM -0500, p...@laptop.org wrote: >daniel wrote: > > On Tue, Dec 23, 2008 at 8:08 PM, wrote: > > > ssh host keys are probably generated on first boot as well. > > > > > > with partitioning support, it should be possible to have a r.o. root > > > overlaid by a unionfs writeable mount, so machine-specific changes > > > don't modify the released partition. this would make cloning quite a > > > bit easier, i'd think. i have no idea what the performance hit of > > > a unionfs setup would be, nor how such a partitioning would fit > > > into the rest of the update strategy (e.g. olpc-update). > > > > unionfs isn't upstream and was quite unreliable last time I use it. > > And it adds the challenge of differentiating state that must be > > discarded for the cloned image, and state that must not be. > > > > For example, we would want to ssh keys generated during first boot to > > *not* be included in the clonable image, that's obvious. But if the > > user boots the OLPC image, goes into the control panel and sets a > > language, then we *do* want that language change to be included in the > > clonable image that is the output of the process. > > > > How would the system differentiate between those two? > >i dunno. i guess the lead engineer on the project would have to >decide. :-) In my opinion, the simplest way to approach this is to add a "hard-reset" script (perhaps named "olpc-hard-reset") which cleans up the image and then prints out a diff from the starting image to the result for manual review. Michael ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
daniel wrote: > On Tue, Dec 23, 2008 at 8:08 PM, wrote: > > ssh host keys are probably generated on first boot as well. > > > > with partitioning support, it should be possible to have a r.o. root > > overlaid by a unionfs writeable mount, so machine-specific changes > > don't modify the released partition. this would make cloning quite a > > bit easier, i'd think. i have no idea what the performance hit of > > a unionfs setup would be, nor how such a partitioning would fit > > into the rest of the update strategy (e.g. olpc-update). > > unionfs isn't upstream and was quite unreliable last time I use it. > And it adds the challenge of differentiating state that must be > discarded for the cloned image, and state that must not be. > > For example, we would want to ssh keys generated during first boot to > *not* be included in the clonable image, that's obvious. But if the > user boots the OLPC image, goes into the control panel and sets a > language, then we *do* want that language change to be included in the > clonable image that is the output of the process. > > How would the system differentiate between those two? i dunno. i guess the lead engineer on the project would have to decide. :-) paul =- paul fox, p...@laptop.org give one laptop, get one laptop --- http://www.laptop.com/xo ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 8:08 PM, wrote: > ssh host keys are probably generated on first boot as well. > > with partitioning support, it should be possible to have a r.o. root > overlaid by a unionfs writeable mount, so machine-specific changes > don't modify the released partition. this would make cloning quite a > bit easier, i'd think. i have no idea what the performance hit of > a unionfs setup would be, nor how such a partitioning would fit > into the rest of the update strategy (e.g. olpc-update). unionfs isn't upstream and was quite unreliable last time I use it. And it adds the challenge of differentiating state that must be discarded for the cloned image, and state that must not be. For example, we would want to ssh keys generated during first boot to *not* be included in the clonable image, that's obvious. But if the user boots the OLPC image, goes into the control panel and sets a language, then we *do* want that language change to be included in the clonable image that is the output of the process. How would the system differentiate between those two? Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
morgan wrote: > On Tue, Dec 23, 2008 at 18:29, Daniel Drake wrote: > > On Tue, Dec 23, 2008 at 2:19 PM, Greg Smith > > wrote: ... > >> The biggest challenge I see is to find those things which you do not want > >> to > >> "clone" from the source XO. The only things that come to mind are Name and > >> Color. We could even pre-fill them as long as those dialog boxes come up > >> at > >> start up. > > > > There is a lot more than that - it's things that are invisible to the > > user, technical details of the system, which are the bits we don't > > have a good answer for. For example (an easy one), keys are generated > > on first boot, but it is potentially bad news down the line if > > multiple XOs have the same keys. The hard part is tracking these > > things, which are not specified anywhere and there's no one place you ... > I believe the mail from Michael that you were looking for is > http://lists.laptop.org/pipermail/devel/2008-March/012200.html - and > http://lists.laptop.org/pipermail/devel/2008-April/012957.html is > probably also relevant. > > The keys generated in ~/.sugar/default/ are AFAIK not used for crypto, > but are used to generate the unique Jabber ID (JID). If two XOs (or > Sugar clients) have the same keys, Strange Bad Things happen to > presence and collaboration. ssh host keys are probably generated on first boot as well. with partitioning support, it should be possible to have a r.o. root overlaid by a unionfs writeable mount, so machine-specific changes don't modify the released partition. this would make cloning quite a bit easier, i'd think. i have no idea what the performance hit of a unionfs setup would be, nor how such a partitioning would fit into the rest of the update strategy (e.g. olpc-update). paul =- paul fox, p...@laptop.org give one laptop, get one laptop --- http://www.laptop.com/xo ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 11:46:09AM -0800, Edward Cherlin wrote: >On Tue, Dec 23, 2008 at 11:26 AM, Michael Stone wrote: >> On Tue, Dec 23, 2008 at 09:18:51PM +0200, Morgan Collett wrote: >>>On Tue, Dec 23, 2008 at 18:29, Daniel Drake wrote: On Tue, Dec 23, 2008 at 2:19 PM, Greg Smith wrote: > Your suggestion that we allow > addition of RPMs and get those built into a signed image via "pilgrim or > puritan" is certainly valuable and part of the requirement. > > However, it doesn't cover a few added things (language settings was > specifically requested by Mongolia and others): > > - Updated language packs (I believe we are trying to make this an RPM > which > may solve it) > - Starting language > - Date, time and timezone > - Network settings >> >> Both puritan and pilgrim install many unpackaged hacks; that's actually >> the major reason why they exist. >> >> (Some special indirection needs to be taken if you want to deploy hacks >> to /home via olpc-update, since it doesn't touch /home, but for >> whole-NAND-reflash tasks, either is certainly adequate.) >> >> It's also possible to combine a compose-tool like puritan or pilgrim >> with our existing image-builder technology (which generates mfg-ready >> images from customization-stick data.) > >Would it be possible to apply these tools to creating LiveCDs and >qemu-ready image files? I certainly don't want to add to the burdens >of over-burdened staff, but can some of us volunteers do that part? I >ask in large part because recent LiveCDs and image files don't work on >my computer. With some work, yes. (The QEMU images are much easier than the livecds since you only have to make it work on one kind of hardware and since both programs already have support for generating QEMU images.) Whether it is wise to use them for this purpose is a separate question -- and, in a sense, a religious war. Michael ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 5:31 PM, Greg Smith wrote: > Both have challenges. My preference is clone because I think its easier for > the end user (create an XO the way you like it then click "clone"). However, > we need to figure out the list of things that should not be cloned as you > mention. It's not just creating a list. You also have to figure out how the list can be made automatically. And then finally (the easy part, but may bring some unexpected challenges) is acting upon the list. The actual imaging may also present challenges, e.g. do you clean up before or after? > Extending the customization key has two main challenges as I see it. The > first is that we may miss some customization that people want. The second is > that its a little cumbersome to collect all the files, put them on a USB > stick, add an fs.zip etc. I believe it also requires a second boot of the XO > and possibly 2 USB sticks. One to update OS and a second to add > "customizations". It can be done like that, or the deployments can do as they already do with the customization stick: they make the customization stick, test it, then turn it into an image (including OS). http://wiki.laptop.org/go/Image_builder then OLPC signs the image, as has been done for Peru builds. > In terms of customization stick or clone, this is where the lead engineer or > whoever does the work gets to make the final call. Just make sure you > address all the requirements and that we get it done in time for a QA cycle > in February and we'll be OK. OK, glad to hear that engineering will have input into the decision ;) I do not have time to work on this myself but may chip in here and there. Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 11:26 AM, Michael Stone wrote: > On Tue, Dec 23, 2008 at 09:18:51PM +0200, Morgan Collett wrote: >>On Tue, Dec 23, 2008 at 18:29, Daniel Drake wrote: >>> On Tue, Dec 23, 2008 at 2:19 PM, Greg Smith wrote: Your suggestion that we allow addition of RPMs and get those built into a signed image via "pilgrim or puritan" is certainly valuable and part of the requirement. However, it doesn't cover a few added things (language settings was specifically requested by Mongolia and others): - Updated language packs (I believe we are trying to make this an RPM which may solve it) - Starting language - Date, time and timezone - Network settings > > Both puritan and pilgrim install many unpackaged hacks; that's actually > the major reason why they exist. > > (Some special indirection needs to be taken if you want to deploy hacks > to /home via olpc-update, since it doesn't touch /home, but for > whole-NAND-reflash tasks, either is certainly adequate.) > > It's also possible to combine a compose-tool like puritan or pilgrim > with our existing image-builder technology (which generates mfg-ready > images from customization-stick data.) Would it be possible to apply these tools to creating LiveCDs and qemu-ready image files? I certainly don't want to add to the burdens of over-burdened staff, but can some of us volunteers do that part? I ask in large part because recent LiveCDs and image files don't work on my computer. > Michael > ___ > Devel mailing list > Devel@lists.laptop.org > http://lists.laptop.org/listinfo/devel > -- Silent Thunder (默雷/धर्ममेघशब्दगर्ज/دھرممیگھشبدگر ج) is my name And Children are my nation. The Cosmos is my dwelling place, The Truth my destination. http://wiki.sugarlabs.org/go/User:Mokurai ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 09:18:51PM +0200, Morgan Collett wrote: >On Tue, Dec 23, 2008 at 18:29, Daniel Drake wrote: >> On Tue, Dec 23, 2008 at 2:19 PM, Greg Smith wrote: >>> Your suggestion that we allow >>> addition of RPMs and get those built into a signed image via "pilgrim or >>> puritan" is certainly valuable and part of the requirement. >>> >>> However, it doesn't cover a few added things (language settings was >>> specifically requested by Mongolia and others): >>> >>> - Updated language packs (I believe we are trying to make this an RPM which >>> may solve it) >>> - Starting language >>> - Date, time and timezone >>> - Network settings Both puritan and pilgrim install many unpackaged hacks; that's actually the major reason why they exist. (Some special indirection needs to be taken if you want to deploy hacks to /home via olpc-update, since it doesn't touch /home, but for whole-NAND-reflash tasks, either is certainly adequate.) It's also possible to combine a compose-tool like puritan or pilgrim with our existing image-builder technology (which generates mfg-ready images from customization-stick data.) Michael ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 18:29, Daniel Drake wrote: > On Tue, Dec 23, 2008 at 2:19 PM, Greg Smith wrote: >> Your suggestion that we allow >> addition of RPMs and get those built into a signed image via "pilgrim or >> puritan" is certainly valuable and part of the requirement. >> >> However, it doesn't cover a few added things (language settings was >> specifically requested by Mongolia and others): >> >> - Updated language packs (I believe we are trying to make this an RPM which >> may solve it) >> - Starting language >> - Date, time and timezone >> - Network settings > > Well, my suggestion of pilgrim and puritan was only for customising > RPM packages. For the other things, I wrote: > "As for other customisations, the current method (customization key) > works fine for the limited customisations that it allows, so that > simply needs to be expanded. " > > I could have probably worded that better. In other words, my opinion > is that the existing customisation stick system should be expanded to > also allow customisation of timezone, language settings, translation > installation, etc. > >> The biggest challenge I see is to find those things which you do not want to >> "clone" from the source XO. The only things that come to mind are Name and >> Color. We could even pre-fill them as long as those dialog boxes come up at >> start up. > > There is a lot more than that - it's things that are invisible to the > user, technical details of the system, which are the bits we don't > have a good answer for. For example (an easy one), keys are generated > on first boot, but it is potentially bad news down the line if > multiple XOs have the same keys. The hard part is tracking these > things, which are not specified anywhere and there's no one place you > can look to find them. I wish I could find a link to Michael's mail, > where he investigated some of these things for an old build, and some > of the findings were surprising even to us who hack on the system > level all day... I believe the mail from Michael that you were looking for is http://lists.laptop.org/pipermail/devel/2008-March/012200.html - and http://lists.laptop.org/pipermail/devel/2008-April/012957.html is probably also relevant. The keys generated in ~/.sugar/default/ are AFAIK not used for crypto, but are used to generate the unique Jabber ID (JID). If two XOs (or Sugar clients) have the same keys, Strange Bad Things happen to presence and collaboration. >> In short, I like your proposal but I still want a little more :-) > > If you're looking for initial high-level action items: > 1. The discussions on lease delegation / allowing countries to sign > their builds / providing customised builds for countries need to be > finished. The outcome would be someone implementing whatever method > allows country A to say "we want OLPC build B with added RPMs C,D,E" > 2. Someone needs to implement the customization stick enhancements > (and surrounding projects, such as xot bundles). Will require some > modifications on the sugar-level too. > > Daniel Regards Morgan ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
Hi Dan, Those sound like two good steps. I think we should make a design decision here to either: 1 - "clone" minus a list of configurations or 2 - Extend customization to include everything relevant for a deployment. Both have challenges. My preference is clone because I think its easier for the end user (create an XO the way you like it then click "clone"). However, we need to figure out the list of things that should not be cloned as you mention. Extending the customization key has two main challenges as I see it. The first is that we may miss some customization that people want. The second is that its a little cumbersome to collect all the files, put them on a USB stick, add an fs.zip etc. I believe it also requires a second boot of the XO and possibly 2 USB sticks. One to update OS and a second to add "customizations". The final image must be deliverable as an upgrade or a clean install via and of the following: olpc-update to internet, via olpc-update to XS, via NAND Blaster, via USB stick. I updated the requirements to bullet those out more clearly. In terms of customization stick or clone, this is where the lead engineer or whoever does the work gets to make the final call. Just make sure you address all the requirements and that we get it done in time for a QA cycle in February and we'll be OK. If some requirement cannot be met (e.g. set the language to Mongolian) we can consider living without it. I prefer to nail them all but hitting the date and improving the product is more important than the perfect solution which never comes. I agree that its closely related to the lease delegation, signing, and building requirements (your next step #1 below). A well architected approach that addresses all the points (section 2 here: http://wiki.laptop.org/go/9.1.0#Top_Priority) would be great! Thanks, Greg S Daniel Drake wrote: > On Tue, Dec 23, 2008 at 2:19 PM, Greg Smith wrote: >> Your suggestion that we allow >> addition of RPMs and get those built into a signed image via "pilgrim or >> puritan" is certainly valuable and part of the requirement. >> >> However, it doesn't cover a few added things (language settings was >> specifically requested by Mongolia and others): >> >> - Updated language packs (I believe we are trying to make this an RPM which >> may solve it) >> - Starting language >> - Date, time and timezone >> - Network settings > > Well, my suggestion of pilgrim and puritan was only for customising > RPM packages. For the other things, I wrote: > "As for other customisations, the current method (customization key) > works fine for the limited customisations that it allows, so that > simply needs to be expanded. " > > I could have probably worded that better. In other words, my opinion > is that the existing customisation stick system should be expanded to > also allow customisation of timezone, language settings, translation > installation, etc. > >> The biggest challenge I see is to find those things which you do not want to >> "clone" from the source XO. The only things that come to mind are Name and >> Color. We could even pre-fill them as long as those dialog boxes come up at >> start up. > > There is a lot more than that - it's things that are invisible to the > user, technical details of the system, which are the bits we don't > have a good answer for. For example (an easy one), keys are generated > on first boot, but it is potentially bad news down the line if > multiple XOs have the same keys. The hard part is tracking these > things, which are not specified anywhere and there's no one place you > can look to find them. I wish I could find a link to Michael's mail, > where he investigated some of these things for an old build, and some > of the findings were surprising even to us who hack on the system > level all day... > >> In short, I like your proposal but I still want a little more :-) > > If you're looking for initial high-level action items: > 1. The discussions on lease delegation / allowing countries to sign > their builds / providing customised builds for countries need to be > finished. The outcome would be someone implementing whatever method > allows country A to say "we want OLPC build B with added RPMs C,D,E" > 2. Someone needs to implement the customization stick enhancements > (and surrounding projects, such as xot bundles). Will require some > modifications on the sugar-level too. > > Daniel > ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Deployment image customization
On Tue, Dec 23, 2008 at 2:19 PM, Greg Smith wrote: > Your suggestion that we allow > addition of RPMs and get those built into a signed image via "pilgrim or > puritan" is certainly valuable and part of the requirement. > > However, it doesn't cover a few added things (language settings was > specifically requested by Mongolia and others): > > - Updated language packs (I believe we are trying to make this an RPM which > may solve it) > - Starting language > - Date, time and timezone > - Network settings Well, my suggestion of pilgrim and puritan was only for customising RPM packages. For the other things, I wrote: "As for other customisations, the current method (customization key) works fine for the limited customisations that it allows, so that simply needs to be expanded. " I could have probably worded that better. In other words, my opinion is that the existing customisation stick system should be expanded to also allow customisation of timezone, language settings, translation installation, etc. > The biggest challenge I see is to find those things which you do not want to > "clone" from the source XO. The only things that come to mind are Name and > Color. We could even pre-fill them as long as those dialog boxes come up at > start up. There is a lot more than that - it's things that are invisible to the user, technical details of the system, which are the bits we don't have a good answer for. For example (an easy one), keys are generated on first boot, but it is potentially bad news down the line if multiple XOs have the same keys. The hard part is tracking these things, which are not specified anywhere and there's no one place you can look to find them. I wish I could find a link to Michael's mail, where he investigated some of these things for an old build, and some of the findings were surprising even to us who hack on the system level all day... > In short, I like your proposal but I still want a little more :-) If you're looking for initial high-level action items: 1. The discussions on lease delegation / allowing countries to sign their builds / providing customised builds for countries need to be finished. The outcome would be someone implementing whatever method allows country A to say "we want OLPC build B with added RPMs C,D,E" 2. Someone needs to implement the customization stick enhancements (and surrounding projects, such as xot bundles). Will require some modifications on the sugar-level too. Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Deployment image customization
Hi Dan, Thanks for the comments on the image customization feature: http://wiki.laptop.org/go/Feature_roadmap/Image_customization I moved them from the requirements to the specification section because I think you are proposing a possible solution. Your suggestion that we allow addition of RPMs and get those built into a signed image via "pilgrim or puritan" is certainly valuable and part of the requirement. However, it doesn't cover a few added things (language settings was specifically requested by Mongolia and others): - Updated language packs (I believe we are trying to make this an RPM which may solve it) - Starting language - Date, time and timezone - Network settings Essentially the things in the control panel, but other settings may be relevant too and even more important to pre-configure if they are only accessible via CLI. The biggest challenge I see is to find those things which you do not want to "clone" from the source XO. The only things that come to mind are Name and Color. We could even pre-fill them as long as those dialog boxes come up at start up. In short, I like your proposal but I still want a little more :-) Let me know what you think. Thanks, Greg S ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
