Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Robert Collins wrote: Huh? Doing what I suggest has nothing to do with the # of connections, the protocol supports it completely. It might, if you are spawning rsync locally each time, with the file list option, you could (easily) add a new file destination list, and that would provide the single connection, custom basis approach. Huh? Do yo mean using the new batch mode, with --read-batch and --write-batch? I don't think I understand in which way you mean to accomplish it... Probably I have overlooked the full usefulness of some command line option... -- Lapo Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796)
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Christopher Faylor wrote: It would be nice if setup.exe supported the rsync protocol. That would also give you the ability to suspend and resume. I don't know how easy it would be to get librsync working in a mingw environment, though. This may not be really up-to-date but rsync use is still your life is so much better if you spawn an actual rsync program to do the work, e.g. rsync authors themself suggest not to use librsync. I asked this very question some time ago because I was ionterested ni adding rsync support to Mozilla (it would be oh so great...), then I tries using protozailla (to actualyl spawn a rsync exe) but protozilla kinda died, it seems. Extract from http://librsync.sourceforge.net/ librsync is not wire-compatible with rsync 2.x, and is not likely to be in the future. This is a new codebase, designed to allow a fresh start on the problem and a cleaner design. It may be used in a future version of rsync or a successor project. Would it help if setup.exe was broken down into two different programs? One for accessing the network and another for manipulating packages? Updating local mirror with rsync would be certainly possible, but only after the base system is already set up, as there is no rsync-on-mingw visible even on the fartest horizon, right now.. -- Lapo Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796)
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Max Bowsher wrote: Brian Dessent wrote: Max Bowsher wrote: So, I'm looking for comments, and suggestions for candidate http/ftp client libraries to investigate. What's wrong with libcurl? It has extensive support for all kinds of extended http/ftp features, see for example http://curl.haxx.se/docs/comparison-table.html, and is MIT-licensed so should be no problem statically linking it into setup.exe. Indeed, libcurl was one which came to mind. I've yet to play with it, but one thing I'm a little worried about is it being *too* extensive - I don't want setup.exe to grow very much. If you are worried about the size of setup, would it help to move /cygwin.bat and /cygwin.ico into base-files? I've an update going out for this soon and it would be little/no trouble... (I could even rename them to have a captial 'C' so they don't clash with /cygdrive ;) J.
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Lapo Luchini wrote: It would be nice if setup.exe supported the rsync protocol. That would also give you the ability to suspend and resume. I don't know how easy it would be to get librsync working in a mingw environment, though. This may not be really up-to-date but rsync use is still your life is so much better if you spawn an actual rsync program to do the work, e.g. rsync authors themself suggest not to use librsync. Plus, with ftp resume and http byte-range support from libcurl (or equiv) you can achieve almost the same thing. If you have a local .tar.bz2 file of the same name but smaller size from a prior aborted download, just resume the download a few kbytes prior to the tail of the file that you have, and if that rollback amount matches up then just continue the download to the end. And in any case you've always got the md5sum so worst case you just redownload the package. The case that rsync would be nice is where the package files are updated on the mirror but the filename/version not bumped. However that seems to happen so infrequently (if ever) that it's probably not a big deal. (And of course, you can always use rsync out-of-band. For example, you can keep a local package dir rsync'd from mirrors.kernel.org or another mirror site that runs rsync and then run setup on the local dir.) Brian
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Brian Dessent wrote: Max Bowsher wrote: Indeed, libcurl was one which came to mind. I've yet to play with it, but one thing I'm a little worried about is it being *too* extensive - I don't want setup.exe to grow very much. I'm certainly going to look into libcurl, but I'm open to other possibilities too. Sure, that's a concern. However current setup.exe is only ~270kb when stripped, and even if that, say, doubled I don't see what the big deal would be. It seems to me that being anorexic about the .exe size at the cost of robust net/url code is a premature optimization. As a test I grabbed the stock 7.12.2 sources and compiled a static libcurl.a with everything but http and ftp support disabled. Before stripping the file is 2.0M, after stripping it's 129k. The corresponding curl.exe is 163k after strip. The only external lib that this .exe depends on is zlib, which is in setup already, no? Thankyou, those numbers make me feel rather more optimistic about using libcurl in setup. Max.
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Christopher Faylor wrote: It would be nice if setup.exe supported the rsync protocol. Lapo and Brian have already covered this, so I'll just summarise: * No library available implementing the rsync protocol (librsync implements a *different* protocol !!!) * Package file scarcely ever change whilst keeping the same name. Also, they are compressed, so a small change is likely to cause the entire contents to change anyway. Therefore, rsync would not help. Would it help if setup.exe was broken down into two different programs? One for accessing the network and another for manipulating packages? Why would it? I think it would be unnecessary complexity. All we need to do is pick an appropriately compact ftp/http lib (and it's beginning to look like libcurl is appropriately compact) and use it. Max.
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Gerrit P. Haase wrote: Is it already compressed (ie. upx executable compressor)? Yes, it is. Works wonders, too - the compression ratio is 36%. Max.
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
John Morrison wrote: If you are worried about the size of setup, would it help to move /cygwin.bat and /cygwin.ico into base-files? I've an update going out for this soon and it would be little/no trouble... (I could even rename them to have a captial 'C' so they don't clash with /cygdrive ;) It would probably only save a few KB. However, I think base-files would be a far more appropriate place for these to live. They are used in setup.exe's shortcut creation, though, so I think we should delay the move until we can transfer the shortcut creation to base-files too. Max.
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Max Bowsher wrote: * Package file scarcely ever change whilst keeping the same name. Also, they are compressed, so a small change is likely to cause the entire contents to change anyway. Therefore, rsync would not help. It could eb somewhat used anyways just doing a ln -s of the old file on the new name, THEN rsyncing it... I shoudl try but in theory it should read it for calculating the diff, then if writes a NEW file with the transfer and delete the old, so it would both have the pro of rsync and the pro of not copy the ACTUAL file (creating all the symlinks for every newer package in setup.ini is not so much work). In fact, I think I will try and do such a bash script to do this, to test the idea. Maybe only for packages ALREADY locally present. Lapo - -- Lapo Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796) -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (Cygwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iEYEARECAAYFAkGV6IkACgkQaJiCLMjyUvu3PgCeI51AcZKvkfBhuRTC7WMv/is1 tcsAoKl4VYyMmPgAjcpJLG3FEx75aW0H =qWrz -END PGP SIGNATURE-
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
On Sat, 2004-11-13 at 11:57 +0100, Lapo Luchini wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Max Bowsher wrote: * Package file scarcely ever change whilst keeping the same name. Also, they are compressed, so a small change is likely to cause the entire contents to change anyway. Therefore, rsync would not help. It could eb somewhat used anyways just doing a ln -s of the old file on the new name, THEN rsyncing it... I shoudl try but in theory it should read it for calculating the diff, then if writes a NEW file with the transfer and delete the old, so it would both have the pro of rsync and the pro of not copy the ACTUAL file (creating all the symlinks for every newer package in setup.ini is not so much work). In fact, I think I will try and do such a bash script to do this, to test the idea. Maybe only for packages ALREADY locally present. Calculating which file to use as a basis for rsync is easy without symlinks: Its just the 'closest to the version being downloaded'. Define closest as you would intuitively: two package files with the same x.y.z components are closer than two with the same x.y and different z. As for using rsync on package files, there is strong evidence from the debian project that it is marginal, unless you have the zlib with the patched hunk-reset which is much more likey to keep consistent data in successive minor releases of a package. For bz2, I don't know for sure, but I suspect a similar issue will apply. Rob signature.asc Description: This is a digitally signed message part
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
On Sat, 2004-11-13 at 12:11 +0100, Lapo Luchini wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Robert Collins wrote: Calculating which file to use as a basis for rsync is easy without symlinks: Its just the 'closest to the version being downloaded'. Define closest as you would intuitively: two package files with the same x.y.z components are closer than two with the same x.y and different z. In my idea symlinks were not used to select the version, just to let rsync think the file is already present locally, and with old file content, so that its content is used for optimization. Thats my point : rsync doesn't require the same file name to use the content for optimisation. You tell rsync what file to use as the basis, and what file to write to. Cheers, Rob signature.asc Description: This is a digitally signed message part
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Robert Collins wrote: Thats my point : rsync doesn't require the same file name to use the content for optimisation. You tell rsync what file to use as the basis, and what file to write to. This, if you rsync one file at a time, but if you create the symlinks in every directory, then you can rsync the whole directory tree with a single command, and is much more efficient. (only one connection for all the files) Anyway... this could lead to a more-optimized script to download most of it, not really to be included in setup so... -- Lapo Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796)
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
On Sun, 2004-11-14 at 03:08 +0100, Lapo Luchini wrote: Robert Collins wrote: Thats my point : rsync doesn't require the same file name to use the content for optimisation. You tell rsync what file to use as the basis, and what file to write to. This, if you rsync one file at a time, but if you create the symlinks in every directory, then you can rsync the whole directory tree with a single command, and is much more efficient. (only one connection for all the files) Huh? Doing what I suggest has nothing to do with the # of connections, the protocol supports it completely. It might, if you are spawning rsync locally each time, with the file list option, you could (easily) add a new file destination list, and that would provide the single connection, custom basis approach. Anyway... this could lead to a more-optimized script to download most of it, not really to be included in setup so... Sure. Rob signature.asc Description: This is a digitally signed message part
Re: setup RFC: Ditch homegrown http/ftp code and use a library?
Max Bowsher wrote: Indeed, libcurl was one which came to mind. I've yet to play with it, but one thing I'm a little worried about is it being *too* extensive - I don't want setup.exe to grow very much. I'm certainly going to look into libcurl, but I'm open to other possibilities too. Sure, that's a concern. However current setup.exe is only ~270kb when stripped, and even if that, say, doubled I don't see what the big deal would be. It seems to me that being anorexic about the .exe size at the cost of robust net/url code is a premature optimization. As a test I grabbed the stock 7.12.2 sources and compiled a static libcurl.a with everything but http and ftp support disabled. Before stripping the file is 2.0M, after stripping it's 129k. The corresponding curl.exe is 163k after strip. The only external lib that this .exe depends on is zlib, which is in setup already, no? Brian
