Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Wed, Jun 17, 2009 at 03:15, C. Scott Ananiancsc...@cscott.net wrote: When I wrote the sugar updater, I included all the code necessary to transfer only the changed portions of a large zip file. http://dev.laptop.org/git/users/cscott/sugar-update-control/tree/bitfrost/util/urlrange.py contains the interesting bits. The only missing piece was a proper manifest file format, which would allow you to download only the manifest to determine which bits have changed, so you could then download only the changed parts inside the zip file. What about reading the zip index (central directory?), then deciding which files to download by the timestamp and file size? Regards, Tomeu --scott -- ( http://cscott.net/ ) ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Wed, Jun 17, 2009 at 4:27 AM, Tomeu Vizosoto...@sugarlabs.org wrote: On Wed, Jun 17, 2009 at 03:15, C. Scott Ananiancsc...@cscott.net wrote: When I wrote the sugar updater, I included all the code necessary to transfer only the changed portions of a large zip file. http://dev.laptop.org/git/users/cscott/sugar-update-control/tree/bitfrost/util/urlrange.py contains the interesting bits. The only missing piece was a proper manifest file format, which would allow you to download only the manifest to determine which bits have changed, so you could then download only the changed parts inside the zip file. What about reading the zip index (central directory?), then deciding which files to download by the timestamp and file size? That would work fine; we can read the zip index without downloading the entire file. It might require a bit of care on the part of the bundle-preparer to ensure that the timestamps of unmodified files are kept constant. The zip index also has a CRC32 of the file (see http://docs.python.org/library/zipfile.html). It should be noted that trusting the CRC32 to validate the file contents is insecure -- but IIRC the security of the update scheme relies on other signatures (probably still unimplemented -- do you know anything about that, Michael?) so you shouldn't have to worry about it during download. --scott -- ( http://cscott.net/ ) ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Mon, Jun 15, 2009 at 10:47 PM, Bernie Innocentiber...@codewiz.org wrote: This means you'd have to change both the client-side (sugar-update-control, fairly simple) and the server side (in our case Mozilla Addons, fairly complex). The XS - which Bryan has - already has an rsync server using the bare rsync proto, and it is relatively easy to install new things that are offered to clients. Have a look at the xs-rsync package. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
C. Scott Ananian wrote: It should be noted that trusting the CRC32 to validate the file contents is insecure -- but IIRC the security of the update scheme relies on other signatures (probably still unimplemented -- do you know anything about that, Michael?) so you shouldn't have to worry about it during download. I have some preliminary thoughts written up on the subject about which I am quietly seeking advice from a variety of people including yourself and which I intend to publish when I am more satisfied with my writing and my thinking. Michael P.S. - Does anyone know of any other folks out there actively working on similar problems? ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Mon, 2009-06-15 at 22:15 +0100, Peter Robinson wrote: What about something similar to the deltarpm feature in F11 where there's a delta created between the previous release and it is served over standard http/ftp/whatever like the full package. In the initial set of updates released for F11 this reduced the update from something like 120 meg to around 18. I'm not sure what format the .xo packages take so I'm not sure what the differences/similarities are. xo bundles are just zip files. One could create file-wise deltas or even a collection of per-file xdeltas, but getting all corner cases right would start to get to tricky. The XO bundle format was invented because existing package formats with dependencies and all that were considered too hard to use for the majority of developers. If we are going to re-implement all these advanced features of real package managers, then why don't we just pick a popular one (rpm, dpkg, I don't care) along with its updaters (yum, aptitude... along with PackageKit)? The strongest objection I heard for this proposal is that we need to support local, unprivileged installation. Well, this can certainly be added to a package manager faster than than we can add deltas to our XO bundle format. Or, we could reconsider our requirements and perform system-wide instalations perhaps with --noscripts (which the xo bundles don't support anyway). -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Tue, 2009-06-16 at 06:00 +0545, Bryan Berry wrote: In Nepal, we are not updating against activities.sugarlabs.org but against the local XS. Would it be terribly complicated to change the code on the XS? No, because there's no code at all: it's just static HTML. You could set olpc-activity-url to point at an rsync:// URL, like this: table class=olpc-activity-info tr tdBrowse/td td class=olpc-activity-idorg.laptop.WebActivity/td td class=olpc-activity-version54/td td class=olpc-activity-url a href=rsync://schoolserver.local/activities/Browse-54.xo download!/a/td /tr /table -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Tue, 2009-06-16 at 06:00 +0545, Bryan Berry wrote: In Nepal, we are not updating against activities.sugarlabs.org but against the local XS. Would it be terribly complicated to change the code on the XS? No, because there's no code at all: it's just static HTML. You could set olpc-activity-url to point at an rsync:// URL, like this: table class=olpc-activity-info tr tdBrowse/td td class=olpc-activity-idorg.laptop.WebActivity/td td class=olpc-activity-version54/td td class=olpc-activity-url a href=rsync://schoolserver.local/activities/Browse-54.xo download!/a/td /tr /table -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Tue, 2009-06-16 at 06:00 +0545, Bryan Berry wrote: In Nepal, we are not updating against activities.sugarlabs.org but against the local XS. Would it be terribly complicated to change the code on the XS? No, because there's no code at all: it's just static HTML. You could set olpc-activity-url to point at an rsync:// URL, like this: table class=olpc-activity-info tr tdBrowse/td td class=olpc-activity-idorg.laptop.WebActivity/td td class=olpc-activity-version54/td td class=olpc-activity-url a href=rsync://schoolserver.local/activities/Browse-54.xo download!/a/td /tr /table -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 On Mon, Jun 15, 2009 at 10:47:57PM +0200, Bernie Innocenti wrote: On 06/15/09 16:37, Bryan Berry wrote: I want to use rsync but I need a mechanism that the users (kids) can initiate through a simple GUI, like the current Activity Update mechanism. Rsync's unique capability to transmit only changed blocks within large files requires using the real rsync:// protocol. It won't work over dumb protocols such as http:// . This means you'd have to change both the client-side (sugar-update-control, fairly simple) and the server side (in our case Mozilla Addons, fairly complex). We could easily export the files directory of Mozilla Addons with rsync://, and the activity updater could use it when available. Exporting all the activities.sugarlabs.org data in an easily mirrorable format would also be a useful service for deployments that want to create a local mirrors. In fact, all our public data should also be made available in raw formats for easy mirroring. I suspect that use of the rsync protocol alone does not solve the challenge described by Bryan. I have no experience with the using public unencrypted rsync: but when wrapped in an ssh session, there is *no* benefit from the clever rsync delta algorithm when comparing differently named files. So you would need to somehow educate rsync about which files it should treat as different versions of same object. And you should then be _very_ certain that newest version also has the newest timestamps. If e.g. the local XO has bogus timestamps, then those files might be seen as newer and thus ignored. Yes, it is possible to force always delta'ing all files, but then you won't see the dramatic speed boost, as rsync then needs to generate hashes of *all* files. - Jonas - -- * Jonas Smedegaard - idealist og Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEAREDAAYFAko3Zx8ACgkQn7DbMsAkQLhmKQCgg7M7Ep6nIZuRJKg8ZoK+nsxF zUYAn1GEcMDs6Unvyzwxgy+ozUyrqIFi =ice2 -END PGP SIGNATURE- ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
When I wrote the sugar updater, I included all the code necessary to transfer only the changed portions of a large zip file. http://dev.laptop.org/git/users/cscott/sugar-update-control/tree/bitfrost/util/urlrange.py contains the interesting bits. The only missing piece was a proper manifest file format, which would allow you to download only the manifest to determine which bits have changed, so you could then download only the changed parts inside the zip file. --scott -- ( http://cscott.net/ ) ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
We soon need to roll out updated version of Nepal's custom suite of activities, E-Paath. The updated bundle weighs in at a whopping 180 MB zipped and 300 MB unzipped. We have found it impractical to use the sugar-update-control mechanism to update the bundle. While E-Paath is large, the updated bundle is only 30% different than the currently deployed bundle. Updating E-Paath with sugar-control-update from Sugar 0.82 currently takes 17 minutes. And that is w/ XO updating against the XS. I figure it would be at least 3x longer w/ 30 XO's updating at once, if many of the updates don't completely stall out. 40 minutes is simply too long in a chaotic classroom environment. I would like to know if the newer version of sugar-control-update http://git.sugarlabs.org/projects/sugar-update-control/repos/mainline/blobs/master/src/model.py only transfers the differences between the new and old bundles, thus making the update process much faster. I have stared at the code myself for about 30 minutes and I can't tell. If the newer version of the sugar-update-control is much faster, how much trouble would it be to backport it to 0.82, if even possible? While E-Paath is a particularly large activity there are other activities that are of significant size such as the big GCompris bundle. wikipedia.xo, and hopefully more in the future. Regards, -- Bryan W. Berry Technology Director OLE Nepal, http://www.olenepal.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Mon, Jun 15, 2009 at 9:25 AM, Bryan Berrybr...@olenepal.org wrote: I would like to know if the newer version of sugar-control-update http://git.sugarlabs.org/projects/sugar-update-control/repos/mainline/blobs/master/src/model.py only transfers the differences between the new and old bundles, thus making the update process much faster. I have stared at the code myself for about 30 minutes and I can't tell. If the newer version of the sugar-update-control is much faster, how much trouble would it be to backport it to 0.82, if even possible? Bryan, Looking at the model.py code in git it seems the full bundle is downloaded since it uses urlretrieve. So nothing has changed w.r.t. downloading compared with 0.82. Please correct me if I am wrong. Could you use something like rsync to update the Epaati bundle? That only transfers the differences. Ton van Overbeek ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Mon, 2009-06-15 at 10:18 -0400, Ton van Overbeek wrote: On Mon, Jun 15, 2009 at 9:25 AM, Bryan Berrybr...@olenepal.org wrote: I would like to know if the newer version of sugar-control-update http://git.sugarlabs.org/projects/sugar-update-control/repos/mainline/blobs/master/src/model.py only transfers the differences between the new and old bundles, thus making the update process much faster. I have stared at the code myself for about 30 minutes and I can't tell. If the newer version of the sugar-update-control is much faster, how much trouble would it be to backport it to 0.82, if even possible? Bryan, Looking at the model.py code in git it seems the full bundle is downloaded since it uses urlretrieve. So nothing has changed w.r.t. downloading compared with 0.82. Please correct me if I am wrong. Could you use something like rsync to update the Epaati bundle? That only transfers the differences. Ton van Overbeek I want to use rsync but I need a mechanism that the users (kids) can initiate through a simple GUI, like the current Activity Update mechanism. -- Bryan W. Berry Technology Director OLE Nepal, http://www.olenepal.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On 06/15/09 16:37, Bryan Berry wrote: I want to use rsync but I need a mechanism that the users (kids) can initiate through a simple GUI, like the current Activity Update mechanism. Rsync's unique capability to transmit only changed blocks within large files requires using the real rsync:// protocol. It won't work over dumb protocols such as http:// . This means you'd have to change both the client-side (sugar-update-control, fairly simple) and the server side (in our case Mozilla Addons, fairly complex). We could easily export the files directory of Mozilla Addons with rsync://, and the activity updater could use it when available. Exporting all the activities.sugarlabs.org data in an easily mirrorable format would also be a useful service for deployments that want to create a local mirrors. In fact, all our public data should also be made available in raw formats for easy mirroring. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
I want to use rsync but I need a mechanism that the users (kids) can initiate through a simple GUI, like the current Activity Update mechanism. Rsync's unique capability to transmit only changed blocks within large files requires using the real rsync:// protocol. It won't work over dumb protocols such as http:// . This means you'd have to change both the client-side (sugar-update-control, fairly simple) and the server side (in our case Mozilla Addons, fairly complex). We could easily export the files directory of Mozilla Addons with rsync://, and the activity updater could use it when available. Exporting all the activities.sugarlabs.org data in an easily mirrorable format would also be a useful service for deployments that want to create a local mirrors. In fact, all our public data should also be made available in raw formats for easy mirroring. What about something similar to the deltarpm feature in F11 where there's a delta created between the previous release and it is served over standard http/ftp/whatever like the full package. In the initial set of updates released for F11 this reduced the update from something like 120 meg to around 18. I'm not sure what format the .xo packages take so I'm not sure what the differences/similarities are. Peter ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem using sugar-update-control to update large (100+ MB) activity bundles
On Mon, 2009-06-15 at 22:47 +0200, Bernie Innocenti wrote: On 06/15/09 16:37, Bryan Berry wrote: I want to use rsync but I need a mechanism that the users (kids) can initiate through a simple GUI, like the current Activity Update mechanism. Rsync's unique capability to transmit only changed blocks within large files requires using the real rsync:// protocol. It won't work over dumb protocols such as http:// . This means you'd have to change both the client-side (sugar-update-control, fairly simple) and the server side (in our case Mozilla Addons, fairly complex). In Nepal, we are not updating against activities.sugarlabs.org but against the local XS. Would it be terribly complicated to change the code on the XS? We could easily export the files directory of Mozilla Addons with rsync://, and the activity updater could use it when available. Exporting all the activities.sugarlabs.org data in an easily mirrorable format would also be a useful service for deployments that want to create a local mirrors. In fact, all our public data should also be made available in raw formats for easy mirroring. -- Bryan W. Berry Technology Director OLE Nepal, http://www.olenepal.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
