Re: rsync to copy files?
* Bob Friesenhahn wrote on Thu, Jun 24, 2004 at 02:16:37AM CEST: On Wed, 23 Jun 2004, Warren Young wrote: Bob Friesenhahn wrote: I have learned that using 'rsync' to copy files improves the install time quite dramatically for repeat installs. *snip* Instead of chasing this solution, try patching autoconf to use 'cp -u' instead of 'install'. I was not aware of this cp option. It seems to be limited to GNU cp. On Linux systems, or systems where GNU cp is available, this seems like a reasonable choice. If you change the install semantics, please consider keeping the 'one install line echoed per file' rule. Please also make the new behavior (e.g. not overwriting files with newer timestamp) optional, the default being the old behavior. Otherwise, hard-to-debug failures could occur. Regards, Ralf
Re: rsync to copy files?
On Thu, 24 Jun 2004, Ralf Wildenhues wrote: * Bob Friesenhahn wrote on Thu, Jun 24, 2004 at 02:16:37AM CEST: On Wed, 23 Jun 2004, Warren Young wrote: Bob Friesenhahn wrote: I have learned that using 'rsync' to copy files improves the install time quite dramatically for repeat installs. *snip* Instead of chasing this solution, try patching autoconf to use 'cp -u' instead of 'install'. I was not aware of this cp option. It seems to be limited to GNU cp. On Linux systems, or systems where GNU cp is available, this seems like a reasonable choice. If you change the install semantics, please consider keeping the 'one install line echoed per file' rule. Please also make the new behavior (e.g. not overwriting files with newer timestamp) optional, the default being the old behavior. Otherwise, hard-to-debug failures could occur. an easy way to replace 'install' is to overwrite the INSTALL variable temporarily (at least with gnu make): make INSTALL=myinstall install where myinstall could be a little shell script that accomodates the replacement to 'install', e.g. handles the commandline options in the right way. This way we regularly replace install by ln (hardlink) where possible (same filesystem and partition) saving time and disk space. Volker -- Volker Börchers TECON Systems AG, http://www.tecon.de
Re: rsync to copy files?
Please reply to list only (I have Mail-Followup-To: set), or at least use the same message for both replies, so I don't have to reply twice. Thank you. * Volker Boerchers wrote on Thu, Jun 24, 2004 at 03:57:35PM CEST: On Thu, 24 Jun 2004, Ralf Wildenhues wrote: * Bob Friesenhahn wrote on Thu, Jun 24, 2004 at 02:16:37AM CEST: [using 'cp -u' or similar within Automake's install* rules] If you change the install semantics, please consider keeping the 'one install line echoed per file' rule. Please also make the new behavior (e.g. not overwriting files with newer timestamp) optional, the default being the old behavior. Otherwise, hard-to-debug failures could occur. an easy way to replace 'install' is to overwrite the INSTALL variable temporarily (at least with gnu make): make INSTALL=myinstall install Yes, I know, thank you. I was not referring to a particular way to install a particular software, but rather to the way the Automake install rules should work. Say, I test a package in a second build tree with different compile options, and then decide to re-install the old package from the first build tree, I do not want to end up with a mixed installation even if I forgot to 'make uninstall' in between. I just would not expect 'make install' to behave that way. Does that make the intention of my comment clear? Regards, Ralf
Re: rsync to copy files?
On Thu, 24 Jun 2004, Ralf Wildenhues wrote: an easy way to replace 'install' is to overwrite the INSTALL variable temporarily (at least with gnu make): make INSTALL=myinstall install Yes, I know, thank you. I was not referring to a particular way to install a particular software, but rather to the way the Automake install rules should work. Say, I test a package in a second build tree with different compile options, and then decide to re-install the old package from the first build tree, I do not want to end up with a mixed installation even if I forgot to 'make uninstall' in between. I just would not expect 'make install' to behave that way. Does that make the intention of my comment clear? Yes, very. From comments posted it seems that we can summarize: o Any alternative to install/cp should not be the default, even if the necessary tools are found since performance improvements depend on the relative performance of the filesystems involved (and there may be no improvement at all if the file must actually be copied and both files are on local disk). o The replacement must reliably update the file if it has changed. I believe that rsync looks at the absolute time stamp (with optional fuzz factor) while it seems that GNU 'cp -c' only checks the source file's date to see if it is newer and does not check file size. Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
rsync to copy files?
A package I am working with needs to install many large files. In the course of package development it is necessary to execute 'make install' many times. These large files never change. I have learned that using 'rsync' to copy files improves the install time quite dramatically for repeat installs. Even for files that change sometimes (executables, libraries, formatted documentation), it is quite common for subsequent installs to copy identical files since the files are already built/generated. Perhaps automake might consider checking for 'rsync' and making use of it if it is available (and works)? Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
Re: rsync to copy files?
Bob Friesenhahn wrote: I have learned that using 'rsync' to copy files improves the install time quite dramatically for repeat installs. This should only be true when the transfer channel is much slower than the disks on which the files are stored. rsync must read both the source and destination files, and do calculations on both, before it knows which parts of the files need to be transferred. When the transfer channel is a network, it's likely that the time spent doing this extra disk I/O and calculation is more than offset by the savings in network time. When the source and destination are on the same machine, there should be no such benefit. Instead of chasing this solution, try patching autoconf to use 'cp -u' instead of 'install'.
Re: rsync to copy files?
On Wed, 23 Jun 2004, Warren Young wrote: Bob Friesenhahn wrote: I have learned that using 'rsync' to copy files improves the install time quite dramatically for repeat installs. This should only be true when the transfer channel is much slower than the disks on which the files are stored. rsync must read both the source and destination files, and do calculations on both, before it knows which parts of the files need to be transferred. When the transfer channel is a network, it's likely that the time spent doing this extra disk I/O and calculation is more than offset by the savings in network time. When the source and destination are on the same machine, there should be no such benefit. That is true. However, usually reading is faster than writing, particularly if NFS is involved. Also, rsync can first check both the file's timestamp and size. In my test case, the source files are accessed via NFS (over 100BASE-T), and the 'make install' is executed on the target system. Instead of chasing this solution, try patching autoconf to use 'cp -u' instead of 'install'. I was not aware of this cp option. It seems to be limited to GNU cp. On Linux systems, or systems where GNU cp is available, this seems like a reasonable choice. Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
