Re: SD/MMC cards, a year later
On Thu, Aug 19, 2010 at 11:24 AM, John Watlington w...@laptop.org wrote: Our experiment with SD/MMC cards as main storage continues. FWIW, I'm about two weeks into failure testing of 4G MLC compact flash from a couple of vendors. I'll update the list when one or both of them kick the bucket. Is OLPC looking at SD rather than CF for price, speed, form factor, or other reasons? --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SD/MMC cards, a year later
On Aug 19, 2010, at 2:23 AM, C. Scott Ananian wrote: On Thu, Aug 19, 2010 at 11:24 AM, John Watlington w...@laptop.org wrote: Our experiment with SD/MMC cards as main storage continues. FWIW, I'm about two weeks into failure testing of 4G MLC compact flash from a couple of vendors. I'll update the list when one or both of them kick the bucket. Is OLPC looking at SD rather than CF for price, Price (and popularity of the SD interface.) speed, form factor, or other reasons? --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: git help needed for 2.6.34 kernel branch revival
On Tue, Aug 17, 2010 at 5:01 PM, Daniel Drake d...@laptop.org wrote: It would be nice to be able to use his work, given that it appears a big chunk of the work is already done. But there are 2 complications: Hi Daniel I think I know what you want -- and I'm an old git hand. Should be able to help You have an oldish 2.6.31 + patches and you want to rebase those onto 2.6.34, without getting tangled with. So the first step would be get the latest of 2.6.31 # git fetch extract the not upstream patches # git format-patch -o mypatches 2.6.31..2.6.31custom checkout a rebasing branch # git checkout -b testrebase 2.6.34 git am mypatches And now you get to go through the pain of applying each patch, dealing with conflicts, etc. git-rebase is actually a trivial wrapper around git-format-patch and git-am -- in this case you know better how to choose the patches to try to rebase. On a separate note, I guess it would make sense for OLPC to define a policy where -stable updates are not brought into the OLPC kernel tree. This would avoid this headache in future, as the history would flow consistent with Linus' tree. I would rule out unfounded merges, but if there is a good reason, we should just do it. Probably that's how we've been rolling anyway. We _always_ have some custom patches for our new/current HW, so we are always playing the rebase game when we move to a new kernel branch. It's not like we'll be able to avoid this (as long as we keep innovating). cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: git help needed for 2.6.34 kernel branch revival
On 19 August 2010 09:43, Martin Langhoff martin.langh...@gmail.com wrote: I think I know what you want -- and I'm an old git hand. Should be able to help You have an oldish 2.6.31 + patches and you want to rebase those onto 2.6.34, without getting tangled with. So the first step would be get the latest of 2.6.31 # git fetch extract the not upstream patches # git format-patch -o mypatches 2.6.31..2.6.31custom checkout a rebasing branch # git checkout -b testrebase 2.6.34 git am mypatches And now you get to go through the pain of applying each patch, dealing with conflicts, etc. Thanks for the advice. This is effectively the 399 patches approach I mentioned. I would rule out unfounded merges, but if there is a good reason, we should just do it. Probably that's how we've been rolling anyway. We _always_ have some custom patches for our new/current HW, so we are always playing the rebase game when we move to a new kernel branch. It's not like we'll be able to avoid this (as long as we keep innovating). For the 2.6.31 to 2.6.34 move, no rebasing happened. The 2.6.34 kernel tree was simply merged into the 2.6.31.6 + olpc tree. I think this was a mistake, given that 2.6.34 doesn't logically follow on from 2.6.31.6. I'm in agreement that the right thing to do is rebase (as you outlined), especially after starting the process of doing this. This also means that merging linux-stable is OK (which is not a bad idea at all). And we should put more effort into upstreaming so that the amount of rebasing work is not so great each time. Thanks, Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: git help needed for 2.6.34 kernel branch revival
On Thu, Aug 19, 2010 at 11:53 AM, Daniel Drake d...@laptop.org wrote: Thanks for the advice. This is effectively the 399 patches approach I mentioned. Yes indeed. I couldn't tell if you knew about using format-patch and am separately - For the 2.6.31 to 2.6.34 move, no rebasing happened. The 2.6.34 kernel tree was simply merged into the 2.6.31.6 + olpc tree. I think this was a mistake, given that 2.6.34 doesn't logically follow on from 2.6.31.6. Yeah. Ugly mistake. And we should put more effort into upstreaming so that the amount of rebasing work is not so great each time. Always true. Helps when you have a kernel hacker around. We've lost ours :-/ Any kernel hacker reading this should look at: http://www.laptop.org/en/utility/people/opportunities.shtml ... m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: git help needed for 2.6.34 kernel branch revival
I'm in agreement that the right thing to do is rebase (as you outlined), especially after starting the process of doing this. This also means that merging linux-stable is OK (which is not a bad idea at all). And we should put more effort into upstreaming so that the amount of rebasing work is not so great each time. A quick question here. Now that fedora has moved to git does it make any sense to follow their repo as the OLPC master and then have the XO customizations exist as a branch off of that? This should help leverage a lot of the work the Fedora kernel maintainers are already doing. We might even want to go as far as to break out XO specific work into individual branches like ofw, wireless, etc so it is easier to merge our changes with newer kernel releases. -Jon ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: git help needed for 2.6.34 kernel branch revival
On Thu, Aug 19, 2010 at 18:01, Martin Langhoff martin.langh...@gmail.com wrote: On Thu, Aug 19, 2010 at 11:53 AM, Daniel Drake d...@laptop.org wrote: Thanks for the advice. This is effectively the 399 patches approach I mentioned. Yes indeed. I couldn't tell if you knew about using format-patch and am separately - For the 2.6.31 to 2.6.34 move, no rebasing happened. The 2.6.34 kernel tree was simply merged into the 2.6.31.6 + olpc tree. I think this was a mistake, given that 2.6.34 doesn't logically follow on from 2.6.31.6. Yeah. Ugly mistake. And we should put more effort into upstreaming so that the amount of rebasing work is not so great each time. Always true. Helps when you have a kernel hacker around. We've lost ours :-/ Any kernel hacker reading this should look at: http://www.laptop.org/en/utility/people/opportunities.shtml ... Considered asking Corbet to put some kind of plug in LWN? May also help attending any of the several kernel conf/summit/gettogethers around Boston. Regards, Tomeu m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: git help needed for 2.6.34 kernel branch revival
On Thu, Aug 19, 2010 at 12:07 PM, Jon Nettleton jon.nettle...@gmail.com wrote: A quick question here. Now that fedora has moved to git does it make any sense to follow their repo as the OLPC master and then have the XO customizations exist as a branch off of that? This should help leverage a lot of the work the Fedora kernel maintainers are already doing. We might even want to go as far as to break out XO specific work into individual branches like ofw, wireless, etc so it is easier to merge our changes with newer kernel releases. I've been followign the migration with interest, with exactly that idea in mind. Based on their git-based framework, there are probably a few smart strategies we can use to make the work of a Fedora derivative much easier. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SD/MMC cards, a year later
Thanks for the update. Can you disclose the brands of the mentioned SD cards? It's always useful to know what not to buy. I'm assuming none of these cards don't have static wear levelling. Did any manufacturer provide you details on that or are they only using spare blocks for repairability? Any plans for the use of NILFS? It seems to help with wear levelling, although it is not a completely circular log FS. Best regards, Tiago On Thu, Aug 19, 2010 at 4:24 AM, John Watlington w...@laptop.org wrote: Our experiment with SD/MMC cards as main storage continues. One frustration has been the rate of change in the SD industry. The dominant model from a vendor in a particular size and speed may only be in production for three or four months before being superseded. Vendors are reluctant to properly inform Quanta of changes which might require retesting. The result is that SD card certification is an ongoing process. A disturbing trend has been the increased error rates and decreased device lifetimes brought by higher density devices. And these are occuring throughout the industry. A batch of 2GB class 2 microSD cards obtained a year ago from a particular manufacturer averaged around 10 TB written before failing, with few transient errors. A batch of 2GB class 2 microSD cards from the same manufacturer today failed with more than half corrupting their FS after only 1TB of writes. The devices wear out around 2-4 TB of writes. I'm seeing the same error distribution on 4GB parts from the same manufacturer, and similar problems of early filesystem corruption from other manufacturers as well. Of the last five SD card models we've tested, we rejected three of them for failing to survive 3 TB of writes. In some size/speeds, we only have a single vendor/model qualified --- always on the brink of being end-of-life'd. While the Armada 610 SOC being used in XO-1.75 does have a raw NAND Flash interface, it's design faces the same problem as the CaFE in XO-1 --- it will only be useful for today's (yesterday's) NAND Flash parts. Each new generation tends to introduce higher demands on the (hardware implemented) error correction algorithms. Nonetheless, we are going to try testing it with UbiFS and considering whether any durability improvement justify the increased price (now up to 2x) and built-in obsolescence of using raw NAND chips. On the positive side, the SD interfaces on XO-1.5 have been exercised extensively and we have eliminated the bit errors that plagued some XO-1 motherboards. Overall, the industry is even more behind microSD than a year ago, although eMMC (roughly the same protocol and all the same automated wear levelling issues but 8 bits wide with multichip modules soldered directly to the PCB) is gaining acceptance in the cell-phone industry. Cheers, wad ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SD/MMC cards, a year later
On Aug 19, 2010, at 1:59 PM, Tiago Marques wrote: Thanks for the update. Can you disclose the brands of the mentioned SD cards? It's always useful to know what not to buy. I probably could, but variations between models from one manufacturer may be greater than variations between manufacturer. I suggest that anyone thinking of purchasing a number of SD cards test the ones they are considering buying, instead of relying on outdated test data. I'm assuming none of these cards don't have static wear levelling. Did any manufacturer provide you details on that or are they only using spare blocks for repairability? They all have dynamic wear levelling, where blocks are actively moved in order to balance the wear across all block in the device. Any plans for the use of NILFS? It seems to help with wear levelling, although it is not a completely circular log FS. We are starting to look at it. Does anybody have experience shipping it ? even using it ? Cheers, wad ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SD/MMC cards, a year later
On Aug 19, 2010, at 4:21 PM, John Watlington wrote: Thanks for the update. Can you disclose the brands of the mentioned SD cards? It's always useful to know what not to buy. I probably could, but variations between models from one manufacturer may be greater than variations between manufacturer. That variation works both ways. You will find identically-branded SD cards that are different internally, and you will find SD cards that are identical internally but branded differently. - Ed ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SD/MMC cards, a year later
On Thu, Aug 19, 2010 at 9:29 PM, Ed McNierney e...@laptop.org wrote: On Aug 19, 2010, at 4:21 PM, John Watlington wrote: Thanks for the update. Can you disclose the brands of the mentioned SD cards? It's always useful to know what not to buy. I probably could, but variations between models from one manufacturer may be greater than variations between manufacturer. That variation works both ways. You will find identically-branded SD cards that are different internally, and you will find SD cards that are identical internally but branded differently. - Ed Directly from the OEM or on the open market? http://www.bunniestudios.com/blog/?page_id=1022 Best regards, Tiago ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SD/MMC cards, a year later
On Thu, Aug 19, 2010 at 9:21 PM, John Watlington w...@laptop.org wrote: On Aug 19, 2010, at 1:59 PM, Tiago Marques wrote: Thanks for the update. Can you disclose the brands of the mentioned SD cards? It's always useful to know what not to buy. I probably could, but variations between models from one manufacturer may be greater than variations between manufacturer. I suggest that anyone thinking of purchasing a number of SD cards test the ones they are considering buying, instead of relying on outdated test data. Ok, thanks. I'm half way down that path already, wanted to save some work. I'm assuming none of these cards don't have static wear levelling. Did any manufacturer provide you details on that or are they only using spare blocks for repairability? They all have dynamic wear levelling, where blocks are actively moved in order to balance the wear across all block in the device. Nice to know. Unfortunately I've come across some flash devices that seem to have none. Correct me if I'm wrong. Dynamic wear levelling is the one done on the free blocks and not the ones that have data, right? From your tests, 10TB on 2GB cards would imply 5000 write/erase cycles with static wear levelling, which is pretty good. An SSD I bought a while ago also had 5000 W/E cycles and that number seems to be going down as they migrate to smaller process nodes. Perhaps that's the case? 3 bits per cell MLC also doesn't spell good things IMHO, especially as they're only reducing the die size in 20% for the same capacity(as per Intel/Micron 25nm 3bpc flash). Any plans for the use of NILFS? It seems to help with wear levelling, although it is not a completely circular log FS. We are starting to look at it. Does anybody have experience shipping it ? even using it ? I've stuck with EXT2 with noatime for now but was looking into it. The only thing I didn't like was it being not fully circular and being arranged in blocks. Other than that it seems fine and should help a lot with the random write latencies, as you may know, which was my motivation for looking into it. Best regards, Tiago Cheers, wad ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: SD/MMC cards, a year later
Directly from the OEM or on the open market? Both. - Ed ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [PATCH] Disable some useless cron jobs
Thanks for the patch, but these are already disabled within the very same file. Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Server-devel] Schoolserver development in Uruguay
El Thu, 19-08-2010 a las 20:56 -0600, Daniel Drake escribió: XS-0.6 and some of the package updates that come later fix a few bugs related to ejabberd CPU/DB. I guess in Paraguay they are still on 0.5. Indeed. Three schools moved to 0.6 due to an HD crash and the 27 new schools which are receiving th next wave of laptops will have 0.6 too. Is there a documented upgrade procedure for the remaining 7 schools? If not, can we hope yum upgrade to be sufficiently smart? But it's not a huge issue because the XOs also have a copy of the journal. So, if technical resources are available for a quick XS repair, disruption should be minimal. Also my thinking. There's however the problem of loosing registrations to the schoolserver. With Sugar 0.84, all the laptops in the school would be stuck without the Register menu item. In Dextrose-1, we've worked around this and similar cases by providing a Register Again function. Admittedly, this solution sucks. Ideally we'd want laptops to probe for a schoolserver in the background and attempt to register automatically until they gain access. Something like the patch for Sugar 0.82 which you wrote. I wanted to rework it for 0.88 so we could merge it in Dextrose, but there wasn't enough time. You're giving numbers but missing an important consideration - the XS backup system makes multiple backups. And it'll continue to do make more and more copes until it meets a certain threshold based on disk size (likely to be 238GB in your case). At this point, it will purge the oldest backups before making new ones. Oops. Actually, no cleanup was being done on that particular schoolserver. There was a /var/lib/ds-backup/ds-cleanup.run left there from 2009 :-( [...] It's possible that within that space you have 10 backups of every journal. So you could possibly get away with a disk half the size, and only retain 5 copies. I'm inventing numbers (and they aren't strictly copies either), but you can provide real ones - how many backups (on average) are there of a journal in this server? What's the disk space used if you only total the space used by the most recent backup of each journal? Also, is it possible that your space-measuring script is counting a 5mb file with 2 hardlinks as 10mb of used disk space? Heh, these are good questions, but answering them all would take quite some time, and it's 1AM over here :-) You still have shell accounts on our schoolservers. Feel free to perform any forensic analysis of this kind. Excellent is a bit subjective, but yes, the fact that it requires any form of connectivity is a roadblock in many cases. However, we came up with a way around this (ideas only, for now, but wouldn't be hard to implement) for puppet: - clone all the puppet repositories and the config files and put them on a USB disk (and do this periodically) - install puppet-server on all the XSs (but dont run it by default) - go to a school with said USB disk, plug it in and run puppet-server - run puppet-client, connecting to localhost - stop puppet-server, unplug USB disk, go home Excellent idea, although the complexity of puppet is hard to justify if the only thing it provides over a mere shell script is some sophisticated dependency checks. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Server-devel mailing list server-de...@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Schoolserver development in Uruguay
On 19 August 2010 21:51, Bernie Innocenti ber...@codewiz.org wrote: El Thu, 19-08-2010 a las 20:56 -0600, Daniel Drake escribió: XS-0.6 and some of the package updates that come later fix a few bugs related to ejabberd CPU/DB. I guess in Paraguay they are still on 0.5. Indeed. Three schools moved to 0.6 due to an HD crash and the 27 new schools which are receiving th next wave of laptops will have 0.6 too. Is there a documented upgrade procedure for the remaining 7 schools? If not, can we hope yum upgrade to be sufficiently smart? Yes, it's on the wiki in the school server section. (forgive my lazyness and atrocious connection for not finding you the link) There's however the problem of loosing registrations to the schoolserver. Yeah, that's a big issue on the XO side. Oops. Actually, no cleanup was being done on that particular schoolserver. Are you sure? There was a /var/lib/ds-backup/ds-cleanup.run left there from 2009 :-( That file is executed automatically. And I think that was even the case in XS-0.5. Excellent idea, although the complexity of puppet is hard to justify if the only thing it provides over a mere shell script is some sophisticated dependency checks. There's more than that. It's a tool that makes you really think through the changes that your making. It helps you centralize everything. It also results in a configuration that results in the ability to upgrade from any point in time to the latest version. It would be less error prone in many ways. And if you have a mix of offline and online servers, its a no-brainer. The puppet benefits (vs shell script) for connected servers are very significant. And if you can just take a few easy steps to share the configuration with your offline servers, you save a lot of time. Daniel ___ Server-devel mailing list server-de...@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
[PATCH] Disable some useless cron jobs
Signed-off-by: Bernie Innocenti ber...@codewiz.org --- modules/base/kspost.10.core.inc |8 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/modules/base/kspost.10.core.inc b/modules/base/kspost.10.core.inc index 431ca3a..d7cf96b 100644 --- a/modules/base/kspost.10.core.inc +++ b/modules/base/kspost.10.core.inc @@ -165,3 +165,11 @@ sed -i -e 's%\t\tdo_dsa%#\t\tdo_dsa%g' \ # disable sshd, saves memory and speeds up startup time /sbin/chkconfig sshd off + +# disable mdadm's cron job (sl#2185), plus a few more useless cron jobs +rm -f /etc/cron.weekly/99-raid-check +rm -f /etc/cron.daily/logrotate +rm -f /etc/cron.daily/makewhatis.cron +rm -f /etc/cron.daily/prelink +rm -f /etc/cron.daily/rpm +rm -f /etc/cron.daily/tmpwatch -- 1.7.2.1 ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
[Server-devel] Looking for additional hands on XS
http://www.laptop.org/en/utility/people/opportunities.shtml cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Looking for additional hands on XS
El Thu, 19-08-2010 a las 15:49 -0400, Martin Langhoff escribió: http://www.laptop.org/en/utility/people/opportunities.shtml Ha! I was just about to write you about some XS requirements for Uruguay :-) -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Looking for additional hands on XS
Whoa.. .found my dreamjob.. applying... 2010/8/19 Bernie Innocenti ber...@codewiz.org El Thu, 19-08-2010 a las 15:49 -0400, Martin Langhoff escribió: http://www.laptop.org/en/utility/people/opportunities.shtml Ha! I was just about to write you about some XS requirements for Uruguay :-) -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Cloning
Hi Martin We had some results doing this , but it is actually much more time consuming than installing and configuring the servers individually. What would be really useful is (a) some way of automatic this or (b) the ability to export the XS configuration especially the Moodle settings. David Leeming -Original Message- From: Martin Langhoff [mailto:martin.langh...@gmail.com] Sent: Friday, 13 August 2010 11:46 p.m. To: David Leeming Cc: XS Devel Subject: Re: [Server-devel] Cloning Hi David, super brief - - get the machine in 'single user mode' (init 1 on the commandline will work) - plug new disk in, setup the partition table, and the filesystems - rsync / and /library to the temp mountpoints - remove hardware-specific files: /etc/udev/rules.d/70-persisntent-net.rules - fixup /etc/fstab to match the UID of the new filesystems - run grub targetting the new disk - move new disk to new machine that will _clone_ the XS so it will keep a lot of identifiers (and all the data) from the original XS. Do NOT use this technique for cloning a base install. Use kickstart/anaconda methods to automate many-machine rollouts. cheers, martin On Fri, Aug 13, 2010 at 6:22 AM, David Leeming da...@leeming-consulting.com wrote: Are there any recommended ways of cloning an XS installation? We have tried Ping and Reflect but with difficulty. David Leeming ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
[Server-devel] Schoolserver development in Uruguay
I'm currently at Plan Ceibal. As you may know, Uruguay developed its own schoolserver based on Debian, running software developed in-house and managed with CFengine. Yesterday we briefly discussed their future plans for the school server. == Debian vs Fedora == First of all, there's no way they're going to reinstall 2500 schoolservers with Fedora or even a newer release of Debian. Online upgrades would be possible, though. There's some interest in repackaging in Debian the datastore backup server and other components of the OLPC XS. This work could be contributed back to you or whoever will become the next schoolserver architect. Perhaps we could get one of the Debian maintainers in our community to get these packages accepted. I could do the same for Fedora. As you said, recommending or supporting multiple schoolserver configurations in parallel doesn't make sense, but it wouldn't hurt if some of the underlying components were shared horizontally, especially for the configurations that are already widely deployed. == Jabber == There are two people working on Jabber. They have been using ejabberd and, quite surprisingly, they've not seen any issues of high CPU load and database corruption. Tomorrow I'll get to work more with them. I still had no time to review Prosody, the Jabber implementation recommended by Collabora. My hacker senses are telling me that switching from Erlang to Lua is a small step in the direction of sanity and simplicity. The Sugar Labs Infrastructure Team has setup new dedicated VM for collaboration, but at this time nobody has been working on it. It's an Ubuntu Lucid machine, but we could reinstall it if needed. Tomeu and Collabora overwhelmed the collaboration stack in Sugar 0.90 and seem to have plans to further evolve it. They should be consulted prior to making any long-term decision on the server side. == Backups == This is a black hole in all deployments I visited. Redundant storage is too expensive. One cheap 500GB hard-drive is typical. In one year, 3 of the 10 schoolservers in Caacupé developed a hard drive failure. Loosing all data is sadly the status quo in both Uruguay and Paraguay. I worked on implementing remote backups for a subset of /library using rsync, but 2Mbit per school and 70Mbit on the backup server are insufficient for the initial sync and probably also for nightly updates. What numbers are we talking about, in terms of size? Here are some numbers from an actual school which has been operating for over one year with 530 registered laptops: 262M backup 19Gcache 3.4M games 1.7M orug 62Mpgsql-xs 67Muploads 238G users 20Kwebcontenido 17Mxs-activation 516M xs-activity-server 827M xs-rsync 2.7G zope-var The feasibility of remote backups varies depending on how much we care to backup. In Paraguay, it was decided that the journal backups are to be considered a valuable if we are to instill the idea in teachers that the laptop is the same of a notebook with homework on it. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. Yesterday Daniel Castelo and I discussed the idea of performing cross-backups between nearby schools. This solution would probably work well in terms of bandwidth distribution, but it would bring some logistic complexity. Probably an acceptable trade-off. == Content management == Paraguay seems quite happy with Plone, but frankly I can't understand why. Teachers heavily use a really simple php tool called PAFM, which provides basic hierarchical file management with no access control or versioning. Oddly, I've not yet may anyone using Moodle. When I ask why, I always hear some vague comment about it being designed for higher education. Same goes for Schooltool. These more structured tools probably present an steeper learning curve and a bad fit for unsophisticated requirements of users who are being exposed to information systems for the first time. After they have functioning backups, Uruguay would like to provide a wiki. They have already looked at Dokuwiki, with which I'm not familiar. It seems to have a readable and easy to learn Creole-like syntax. I would personally recommend going either for the simplest possible wiki in this category, or straight to Mediawiki--the most feature-complete out there. Any mid-size solution such as MoninMoin is likely to bring the worst of both worlds. Having written my own minimalist wiki, perhaps I'm slightly biased on this topic. Just slightly, yeah :-) Seriously, the choice of wiki would depend on what other tools would complement it. If you already have Moodle or Schooltool, you probably need just a basic wiki for taking notes on the side. With Mediawiki, one would probably install a bunch of powerful extensions to build
Re: [Server-devel] Schoolserver development in Uruguay
Bernie, Guys: A few of my ideas are below: 2010/8/19 Bernie Innocenti ber...@codewiz.org I'm currently at Plan Ceibal. As you may know, Uruguay developed its own schoolserver based on Debian, running software developed in-house and managed with CFengine. Yesterday we briefly discussed their future plans for the school server. == Debian vs Fedora == First of all, there's no way they're going to reinstall 2500 schoolservers with Fedora or even a newer release of Debian. Online upgrades would be possible, though. There's some interest in repackaging in Debian the datastore backup server and other components of the OLPC XS. This work could be contributed back to you or whoever will become the next schoolserver architect. Whichever the distro... It should be easily maintainable and deployable. I've no problem in building the dpkg's of the XS components. I think its justified, since there is a 2500 servers base. Let me tell you that I've only worked with rpm before, but I've no problem in learning the Debian guidelines, and maybe, try to push the packages into debian testing. Perhaps we could get one of the Debian maintainers in our community to get these packages accepted. I could do the same for Fedora. As you said, recommending or supporting multiple schoolserver configurations in parallel doesn't make sense, but it wouldn't hurt if some of the underlying components were shared horizontally, especially for the configurations that are already widely deployed. == Jabber == There are two people working on Jabber. They have been using ejabberd and, quite surprisingly, they've not seen any issues of high CPU load and database corruption. Tomorrow I'll get to work more with them. I still had no time to review Prosody, the Jabber implementation recommended by Collabora. My hacker senses are telling me that switching from Erlang to Lua is a small step in the direction of sanity and simplicity. The Sugar Labs Infrastructure Team has setup new dedicated VM for collaboration, but at this time nobody has been working on it. It's an Ubuntu Lucid machine, but we could reinstall it if needed. Tomeu and Collabora overwhelmed the collaboration stack in Sugar 0.90 and seem to have plans to further evolve it. They should be consulted prior to making any long-term decision on the server side. == Backups == This is a black hole in all deployments I visited. Redundant storage is too expensive. One cheap 500GB hard-drive is typical. In one year, 3 of the 10 schoolservers in Caacupé developed a hard drive failure. Loosing all data is sadly the status quo in both Uruguay and Paraguay. I worked on implementing remote backups for a subset of /library using rsync, but 2Mbit per school and 70Mbit on the backup server are insufficient for the initial sync and probably also for nightly updates. What numbers are we talking about, in terms of size? Here are some numbers from an actual school which has been operating for over one year with 530 registered laptops: 262M backup 19Gcache 3.4M games 1.7M orug 62Mpgsql-xs 67Muploads 238G users 20Kwebcontenido 17Mxs-activation 516M xs-activity-server 827M xs-rsync 2.7G zope-var The feasibility of remote backups varies depending on how much we care to backup. In Paraguay, it was decided that the journal backups are to be considered a valuable if we are to instill the idea in teachers that the laptop is the same of a notebook with homework on it. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. Yesterday Daniel Castelo and I discussed the idea of performing cross-backups between nearby schools. This solution would probably work well in terms of bandwidth distribution, but it would bring some logistic complexity. Probably an acceptable trade-off. How about 2 500GB in RAID-1? I mean, specially in Paraguay, bandwidth is scarce. == Content management == Paraguay seems quite happy with Plone, but frankly I can't understand why. Teachers heavily use a really simple php tool called PAFM, which provides basic hierarchical file management with no access control or versioning. Oddly, I've not yet may anyone using Moodle. When I ask why, I always hear some vague comment about it being designed for higher education. Same goes for Schooltool. These more structured tools probably present an steeper learning curve and a bad fit for unsophisticated requirements of users who are being exposed to information systems for the first time. After they have functioning backups, Uruguay would like to provide a wiki. They have already looked at Dokuwiki, with which I'm not familiar. It seems to have a readable and easy to learn Creole-like syntax. I would
[Server-devel] Schoolserver development in Uruguay
Comments regarding the initial paraguayian deployment, i'm not very familiar with the current status Regarding distros, when the initial setup was made, the XS (fedora based) schoolserver was the only straightforward instalation that could have anything working with not so much tampering, and was pretty automatic, so XS was chosen, I'm a regular sysadmin, and i got the thing working given enough research time, and martin's help.. is not always like that.. we must take into account that there are not many people that could get an XS working on any given distro, and although there are many volunteers (like bernie) who go around the world doing this things, sustainability is very far away. If there is a distro, or many distros, is not the real problem, the real problem is that there needs to be a simple straightforward and automatic way to deploy a schoolserver without needing a masters in computer science, or even a deegree at all. It has to be fast and it has to be simple. In Plan Ceibal worked fine using debian, because they have specialized people that can do the develpment and can do the maintenance, not because the chose this or that distro In Paraguay, Fedora was chosen for the same reasons, it was the fastest way to get things done, and the simple way to sustain it in the long term, with XS and Fedora patches, which I don't know if were made later on. == Jabber == I don't really understand much how collaboration works, so, no comments == Backups == This numbers make sense to paraguay deployment but may not make sense to other deployments, so I'll explain the folders that I remember 262M backup Backup folder, where all data that was going to be rsynced to the datacentes was stored, it would amount to a backup of the plone, the databases, some configs, and other stuff 3.4M games This was a folder where a web based game was going to be stored, this would be published by the apache web server 1.7M orug Same as before, it was a game developed by a paraguayian legal team to help kids learn about thir rights 62M pgsql-xs I don't remembre 67M uploads The PAFM web folder discussed leter 238G users The datastores folder 20K webcontenido The apache default webpage, with specialized links for games, activities and others 17M xs-activation 516M xs-activity-server 827M xs-rsync I don't remember 2.7G zope-var Since plone works with a selfcontained filesystem for its webpage, this _single_ file was going to be backed up to the datacenter as well, i think this is the folder, but i remember that it had to go to the backup folder anyways The feasibility of remote backups varies depending on how much we care to backup. In Paraguay, it was decided that the journal backups are to be considered a valuable if we are to instill the idea in teachers that the laptop is the same of a notebook with homework on it. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. This problem is more related to the way the journal stores the files and the metadata, I remember little about it, but the main problem with backing up a laptop is no just about taking files, any single file in the datastore doesn't get you a back up, you have to take the whole datastore folder. Incremental, or differential backups could be made if the datastore treated the files differently, I'm sorry if I hurt some suceptibilities, but is the truth, there's no simple way to back up _just the data_ from the journal, you back it all or nothing, because _part of it_ is useless. I don't know if that improved in the newer version, but 0.82 (i think) is the one that was used in Paraguay, is like that Yesterday Daniel Castelo and I discussed the idea of performing cross-backups between nearby schools. This solution would probably work well in terms of bandwidth distribution, but it would bring some logistic complexity. Probably an acceptable trade-off. This an interesting idea, and is related to the sustainability part of the deployment, and XS Deploying a schoolserver should be made easy, this would help small deployments and big deployments, the faster the server gets to the school the better, we all know that, but the real advantage is when maintenance can be made from remote or with simple and fast solutions, like puppet, CFEngine, or even self conained rpm/deb packages, becuase this is the way that we get the masters in computer scince in every schoolserver we want. The faster that a schoolserver can be install, and for the matter reinstalled and restore the better, because then you would only need to send a guy (not a sysadmin) to go, insert a CD, next, next, next, voila.. even when changes need to be done.. they should be done in the way that can be applyed to the
Re: [Server-devel] [Tecnologia] Schoolserver development in Uruguay
El Thu, 19-08-2010 a las 21:57 -0400, Rodolfo D. Arce S. escribió: If there is a distro, or many distros, is not the real problem, the real problem is that there needs to be a simple straightforward and automatic way to deploy a schoolserver without needing a masters in computer science, or even a deegree at all. It has to be fast and it has to be simple. Yes, this is true. Basic IPv4 setup and swapping a broken network card currently require too much command line wizardry. This is probably due to lack of time to develop a nicer interface, not a sadistic design choice. Also, once you have more servers than you can count on the palm of your hand, you really need advanced management tools such as CFengine, Puppet and BCFG2 along with advanced monitoring tools. With such advanced tools come the need for advanced UNIX sysadmins to install and manage them. I don't see any practical way to sweeten this pill. In Plan Ceibal worked fine using debian, because they have specialized people that can do the develpment and can do the maintenance, not because the chose this or that distro I agree, one distro is more or less equivalent to another. What really matters is how much experience you have acquired. 20Kwebcontenido The apache default webpage, with specialized links for games, activities and others This does not seem to be visible in the schools. If I visit http://schoolserver , I get redirected to Plone. 17Mxs-activation 516M xs-activity-server 827M xs-rsync I don't remember I know about these... they don't need to be backed up. 2.7G zope-var Since plone works with a selfcontained filesystem for its webpage, this _single_ file was going to be backed up to the datacenter as well, i think this is the folder, but i remember that it had to go to the backup folder anyways Isn't there a way to perform garbage collection on this file? Or do you think that school is really storing 2.7GB of data in Plone? This problem is more related to the way the journal stores the files and the metadata, I remember little about it, but the main problem with backing up a laptop is no just about taking files, any single file in the datastore doesn't get you a back up, you have to take the whole datastore folder. Incremental, or differential backups could be made if the datastore treated the files differently, I'm sorry if I hurt some suceptibilities, but is the truth, there's no simple way to back up _just the data_ from the journal, you back it all or nothing, because _part of it_ is useless. Yes, but metadata consists of very tiny files compared to actual data files, which is often multimedia content. Sascha designed some kind of datastore object bundle which encapsulates data and metadata in a single archive, to be used for backups and data transfers. Even if it were already available, this new backup format wouldn't solve the fundamental problem that backups of all journals in a school amount to 260GB, though. Backing up the datastore with a simple rsync is simple and fast. Note that hardlinks are already being used to reduce the size of incremental backups as much as possible and the Sugar datastore does not modify or move around application data once it has been created. I don't know if that improved in the newer version, but 0.82 (i think) is the one that was used in Paraguay, is like that The datastore layout changed radically from 0.82 to 0.84. In Dextrose-1, we force an index rebuild after restoring a backup, so even a partial backup would be useful. But why would we not want to backup everything? [...] Paraguay in that sense did something really cool.. install the schoolserver _in house_, update it and send it to the field, but in the fild had to go somebody that could make the adjustments for the network to work, and that simple task was very difficult sometimes, and updating the leases and content when no conection is available is not fun either. After over one year, we're still sending out a sysadmin from Asuncion to Caacupe for every non-trivial operation that needs to be performed from the console. This would obviously not work with a nation-wide deployment. Tomorrow I'll try to interview the people of Plan Ceibal to figure out whether they managed to train field technicians even in the most remote areas or if they've got a sysadmin taskforce that gets parachuted on demand. The plone deployment was based on a decision to give the schools a simple CMS, not simple in deployment, but simple in using, using a plone CMS to create content is super friendly, but in time, none of the teachers or students got to used it (simple or not) The per-school Plone instances have been superseded by a centralized version which the education team seems to be using for uploading documents: http://biblioteca.paraguayeduca.org/ They used this tool instead because they learned how to use it during teachyer training, and they like it, it had
Re: [Server-devel] Schoolserver development in Uruguay
On 19 August 2010 18:25, Bernie Innocenti ber...@codewiz.org wrote: == Jabber == There are two people working on Jabber. They have been using ejabberd and, quite surprisingly, they've not seen any issues of high CPU load and database corruption. Tomorrow I'll get to work more with them. XS-0.6 and some of the package updates that come later fix a few bugs related to ejabberd CPU/DB. I guess in Paraguay they are still on 0.5. This is a black hole in all deployments I visited. Redundant storage is too expensive. One cheap 500GB hard-drive is typical. In one year, 3 of the 10 schoolservers in Caacupé developed a hard drive failure. But it's not a huge issue because the XOs also have a copy of the journal. So, if technical resources are available for a quick XS repair, disruption should be minimal. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. You're giving numbers but missing an important consideration - the XS backup system makes multiple backups. And it'll continue to do make more and more copes until it meets a certain threshold based on disk size (likely to be 238GB in your case). At this point, it will purge the oldest backups before making new ones. Saying that you've hit 238GB after a year isn't conclusive because its likely that you'll meet the threshold when you're measuring an active school over such a long time period. It's the design - use the available space. It's possible that within that space you have 10 backups of every journal. So you could possibly get away with a disk half the size, and only retain 5 copies. I'm inventing numbers (and they aren't strictly copies either), but you can provide real ones - how many backups (on average) are there of a journal in this server? What's the disk space used if you only total the space used by the most recent backup of each journal? Also, is it possible that your space-measuring script is counting a 5mb file with 2 hardlinks as 10mb of used disk space? Paraguay uses Puppet. We're very happy with it. Uruguay uses CFengine. They seem to be very happy with it as well. Both employ a flat hierarchy with one puppet master controlling all the schools, which is simple and straightforward, but requires excellent connectivity. Excellent is a bit subjective, but yes, the fact that it requires any form of connectivity is a roadblock in many cases. However, we came up with a way around this (ideas only, for now, but wouldn't be hard to implement) for puppet: - clone all the puppet repositories and the config files and put them on a USB disk (and do this periodically) - install puppet-server on all the XSs (but dont run it by default) - go to a school with said USB disk, plug it in and run puppet-server - run puppet-client, connecting to localhost - stop puppet-server, unplug USB disk, go home Daniel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Schoolserver development in Uruguay
El Fri, 20-08-2010 a las 00:51 -0300, Bernie Innocenti escribió: Heh, these are good questions, but answering them all would take quite some time, and it's 1AM over here :-) Meanwhile, my du run to find out the size of current backups completed: # du -sh --exclude datastore-200* /library/backup 92G/library/backup So, backing up the last versions of all journals would take just 92GB, which would take more that 4 days on a 2mbit link for the initial backup. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
