Re: proposal to speed rsync with lots of files

2009-03-11 Thread Mag Gam
Using inotify with rsync is a great idea. If one has a job that runs daily to get differences on a very large filesytem with very small files, then can do this (assuming the initial copy is already completed): inotify watch source filesystem (or tree) take down all the notices in a txt file

Re: proposal to speed rsync with lots of files

2009-03-09 Thread Fabian Cenedese
At 07:58 06.03.2009 -0800, Wayne Davison wrote: On Thu, Mar 05, 2009 at 03:27:50PM -0800, Peter Salameh wrote: My proposal is to first send a checksum of the file list for each directory. If is found to be identical to the same checksum on the remote side then the list need not be sent for

Re: proposal to speed rsync with lots of files

2009-03-06 Thread Wayne Davison
On Thu, Mar 05, 2009 at 03:27:50PM -0800, Peter Salameh wrote: My proposal is to first send a checksum of the file list for each directory. If is found to be identical to the same checksum on the remote side then the list need not be sent for that directory! My rZync source does something

proposal to speed rsync with lots of files

2009-03-05 Thread Peter Salameh
Hello, I have followed the discussion of speeding up rsync when there are lots of files, and I have a proposal which I think would greatly speed rsync when doing routine mirroring of large filesystems. One of the speed-limiting issues with rsync is having to send huge file lists when

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Kyle Lanclos
Peter Salameh wrote: One of the speed-limiting issues with rsync is having to send huge file lists when mirroring large file systems, even for incremental updates where only a small part of the file system might have changed. Personally, I find that the sending of the file list, whether

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Peter Salameh
Kyle Lanclos wrote: Peter Salameh wrote: One of the speed-limiting issues with rsync is having to send huge file lists when mirroring large file systems, even for incremental updates where only a small part of the file system might have changed. Personally, I find that the sending of

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Jamie Lokier
Kyle Lanclos wrote: Peter Salameh wrote: One of the speed-limiting issues with rsync is having to send huge file lists when mirroring large file systems, even for incremental updates where only a small part of the file system might have changed. Personally, I find that the sending of

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Jamie Lokier
Peter Salameh wrote: My proposal is to first send a checksum of the file list for each directory. If is found to be identical to the same checksum on the remote side then the list need not be sent for that directory! ... It might even be possible to use the rsync checksum algorithm on the

Re: proposal to speed rsync with lots of files

2009-03-05 Thread lewis butler
On 5-Mar-2009, at 16:27, Peter Salameh wrote: One of the speed-limiting issues with rsync is having to send huge file lists when mirroring large file systems, even for incremental updates where only a small part of the file system might have changed. My proposal is to first send a checksum