Re: inefficient: --checksum calculation shouldn't be done for new files

2011-07-03 Thread Jamie Lokier
Carlos Carvalho wrote: > When --checksum is used they're calculated in both ends to see if the > file should be transfered. This is of course not necessary if the file > doesn't exist in the destination. However, the checksum is still > calculated by the sender, which is often a very large overhead

Re: Rsync backup issues using relative paths and LVM snapshots

2010-07-16 Thread Jamie Lokier
Alex Ferrara wrote: > My problem is that if I mark a directory to have a snapshot created > before rsync and use the -R (relative) option, the directory > structure on the destination system will be the relative path of > where I mounted the snapshot (/mnt/sync-snapshot in my case). If I > don't us

Re: rsync 3.0.7 network errors on MS-Windows

2010-06-02 Thread Jamie Lokier
andrew.mar...@uk.bnpparibas.com wrote: > >I am experiencing intermittent network failures on rsync 3.0.7 built >using cygwin for Windows-XP (SP2). I am using GCC v4.4.2 and the >latext version of cygwin. >The rsync error long indicates things like: >rsync: writefd_unbuffered fa

Re: rsync problem with cygwin 1.7.?

2010-04-21 Thread Jamie Lokier
Wiesner Thomas wrote: > Hello. > My laptop runs XP Home SP3 and my workstation XP Prof. SP3. > > I use Cygwin and rsync to sync my files to a Debian 4.0 server which > runs rsync 2.6.9 in daemon mode. > > It had always worked quite will until I upgraded to Cygwin 1.7.x > (the laptop runs Cygwin

Re: Getting EEXIST out of make_bak_dir()

2010-04-19 Thread Jamie Lokier
Jordan Russell wrote: > (followup to thread from last month) > > On 3/9/2010 10:09 AM, Wayne Davison wrote: > > On Mon, Mar 08, 2010 at 10:02:26AM -0600, Mike Bombich wrote: > >> rsync: make_bak_dir mkdir > >> "/Volumes/Backup/_Archive_2010_March_07_22-27-43/Users/jsmith/Library/Mail/Mailboxes/ >

Re: UDT4 and rsync? support to socket.c?

2010-03-25 Thread Jamie Lokier
Jamie Lokier wrote: > I'll be interested if TCP cannot reach anywhere near the same > performance level, because TCP is hardware assisted if you have > sensible 10G hardware, and I can't imagine UDT4 needing any less RAM > on an unreliable network. Having just read the pr

Re: UDT4 and rsync? support to socket.c?

2010-03-25 Thread Jamie Lokier
Jan Wagner wrote: > Jamie Lokier kirjoitti: > >Jan Wagner wrote: > > > >>Hi, has anyone of the devels considered adding UDT4 fast reliable udp > >>transport to socket.c, as a user-selectable alternative to using default > >>slow TCP? > >> >

Re: UDT4 and rsync? support to socket.c?

2010-03-23 Thread Jamie Lokier
Jan Wagner wrote: > Hi, has anyone of the devels considered adding UDT4 fast reliable udp > transport to socket.c, as a user-selectable alternative to using default > slow TCP? > > It could give a 4 to 10-fold throughput improvement to rsync speed over > wide area networks. If you're seeing th

Re: O_DIRECT, avoiding system cache?

2010-03-14 Thread Jamie Lokier
Tomasz Chmielewski wrote: > Is it possible (planned?) to make rsync avoid going through system cache > and use direct IO? > > Right now, if you decide to backup your desktop system (but it's not > only about desktop systems; rather more about one-time-only data > transfers) to external disk, yo

Re: rsync daemon performance

2010-03-09 Thread Jamie Lokier
Paul Slootman wrote: > On Sat 06 Mar 2010, Wayne Davison wrote: > > On Thu, Feb 25, 2010 at 5:20 PM, Tom Dickson wrote: > > > > > Is it possible to get a "pool" of waiting daemons, similar to how apache > > > runs? > > > > No, there is no support for that in rsync at the moment. I don't think i

Re: cygwin + rsync issue under Windows 7 x64

2010-03-07 Thread Jamie Lokier
Wayne Davison wrote: >I'd imagine that both ssh and rsync start using a lot of CPU because >the socketpair must be indicating that it is ready for a write (or >read) but the actual write() (or read()) fails to return any bytes (as >long as errno is something like EAGAIN, EINTR, or E

Re: Latency and Rsync Transfers

2010-02-01 Thread Jamie Lokier
Neal B wrote: >Thanks for your reply. I have been experimenting with the buffer >settings and when specifying it actually causes the transfers to go >slower. >I am running an rsync server using xinet.d and an rsync client. I >have tried specifying the sockopts on just the clie

Re: rsync/cwgwin hangs during transfers

2010-01-19 Thread Jamie Lokier
Steven Hartland wrote: > We've never been able to use rsync on cygwin reliably. We tried loads of > things and whenever its been brought up on the cygwin list, the answer has > always been that its down to issues with pipes implementation, which was a > short coming of the underlining OS so nothing

Re: How do I make rsync ignore unreadable files (damaged sectors)?

2009-12-30 Thread Jamie Lokier
Matthias Schniedermeyer wrote: > On 22.12.2009 19:24, Stefan Nowak wrote: > >>> On 22.12.2009 16:39, Stefan Nowak wrote: > >>> > >>> The only low-budget test ideas I have: > >>> > >>> The CD scratching a la Tomas Gustavsson seems the only easily > >>> achievable > >>> solution. But then it is not

Re: Will rsync support btrfs file system?

2009-12-06 Thread Jamie Lokier
Benjamin R. Haskell wrote: > find /home/ -xdev -type d | sudo inotifywait --fromfile - -m | perl -lnwe > 'BEGIN{$|=1;} print unless $h{$_}; $h{$_}++' The biggest problem in my experience is it can take 5 minutes of thrashing to set up the inotifies on a large /home directory, from cold cache and

Re: DO NOT REPLY [Bug 6881] New: --bwlimit option uses KiB/s, but is documented as (what amounts to) kB/s

2009-11-09 Thread Jamie Lokier
samba-b...@samba.org wrote: > Given that this is a network transfer rate, it'd be more proper (and > consistent with other applications) to change the function to work > in SI kilobytes per second (i.e. use 1000 instead of 1024), but > that's backwards-incompatible. If you'd like to go this route,

Re: Nice little performance improvement

2009-10-17 Thread Jamie Lokier
Mike Connell wrote: > > Hi, > > >Interesting. If you're not using incremental recursion (the default in > >rsync >= 3.0.0), I can see that the "du" would help by forcing the > >destination I/O to overlap the file-list building in time. But with > >incremental recursion, the "du" shouldn't be ne

Re: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)

2009-08-26 Thread Jamie Lokier
JW wrote: > On Monday 13 July 2009 14:18:38 Ryan Malayter wrote: > > It would be a big boost for large files if rsync "remembered" the > > hashes on each end, so it didn't have to re-read the files on every > > run if the files were unchanged. This is a feature that rsync's > > developers have reje

Re: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)

2009-07-15 Thread Jamie Lokier
Ryan Malayter wrote: > So, when transferring a large file, it goes something like this from > the sender's perspective: > 1) sending file list > 2) receving file list > 3) file is different! Recevier, please give me some hashes > 4) got hashes > 5) begin transfer, calculating my hashes and co

Re: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)

2009-07-13 Thread Jamie Lokier
Ryan Malayter wrote: > Your log file indicates that rsync is indeed working as designed > finding lots of data matches: > >Literal data: 123736377 bytes >Matched data: 17889663500 bytes > > This means that rsync only had to transfer 118 MB instead of 16+ GB. > It does this by trading CPU

Re: Possibility to porting Rsync to NT and other platforms

2009-06-11 Thread Jamie Lokier
John E. Malmberg wrote: > Hasanat Kazmi wrote: > >Hello, > >I am looking into possibilities of porting RSync for windows. Does anybody > >have an idea that which libraries and dependencies RSync uses which can not > >be compiled on windows (so thats why we use cygwin) > > I have not looked at this

Re: Does rsync detect file corruption?

2009-05-24 Thread Jamie Lokier
Daniel Carrera wrote: > Jamie Lokier wrote: > >Daniel Carrera wrote: > >>But there is no way to distinguish between file corruption and a > >>legitimate change. All you can do is keep old backups for a few days or > >>weeks and hope that you detect t

Re: Does rsync detect file corruption?

2009-05-24 Thread Jamie Lokier
Daniel Carrera wrote: > But there is no way to distinguish between file corruption and a > legitimate change. All you can do is keep old backups for a few days or > weeks and hope that you detect the file corruption before the backup > rotation deletes all the good copies. I'm under the impress

Re: Rolling algorithm vs CRC

2009-05-22 Thread Jamie Lokier
Hasanat Kazmi wrote: > Hello, > I have previously mailed on list that I am trying to port rsync to NT. I was > wondering that whether CRC can be used to find check sums rather then rolling > algorithm. I havnt found any document on web comparing rolling algorithm with > CRC. It will only find non-

Re: Rolling algorithm vs CRC

2009-05-22 Thread Jamie Lokier
Ryan Malayter wrote: > On Fri, May 22, 2009 at 9:02 AM, Hasanat Kazmi wrote: > > Hello, > > I have previously mailed on list that I am trying to port rsync to NT. I was > > wondering that whether CRC can be used to find check sums rather then > > rolling algorithm. I havnt found any document on we

Re: Is it possible to make rsync VMware split .vmdk's aware?

2009-03-26 Thread Jamie Lokier
Bas Bahlmann || Steady IT Systeembeheer wrote: > Hi Matt, > > I am working a couple of days now with monolithic files and I keep > hitting io time outs on large vmdk's while the line (IPSec tunnel) stays > online. I don't know if it's relevant, but rsync can be extremely slow at transmitting larg

Re: Weird rsync or shell problem?

2009-03-22 Thread Jamie Lokier
Wayne Davison wrote: > On Sat, Mar 21, 2009 at 07:23:58PM +0100, axel.we...@cbc.de wrote: > > SCHNITT/Render Files/#342#200#242Vorlage Bleskin_990101_NB_n-tv/Vorlage > > Blesk-FIN-000b > > The character sequence 0343 0200 0242 is E2 80 A2 in hex. Seems to be a > common sequence for something

Re: dynamic bandwidth allocation

2009-03-14 Thread Jamie Lokier
Felipe Alvarez wrote: > Hi list > > I was wondering if this is (or could be) possible with rsync. I was > wondering if there was a way to change the bandwidth limit > (--bwlimit=KBPS) dynamically while rsync is running? Could this be > possible maybe with signals like USR1 add 5KBPS and USR2 subtr

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Jamie Lokier
Peter Salameh wrote: > My > proposal is to first send a checksum of the file list for each > directory. If is found to be identical to the same checksum on the > remote side then the list need not be sent for that directory! ... > It might even be possible to use the rsync checksum algorithm on

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Jamie Lokier
Kyle Lanclos wrote: > Peter Salameh wrote: > > One of the speed-limiting issues with rsync is having to send huge file > > lists when mirroring large file systems, even for incremental updates > > where only a small part of the file system might have changed. > > Personally, I find that the send

Re: Renamed files and directories

2009-03-02 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: > >Right, but it has to be done in a separate pass if you're to compare > >all files with each other, not just one destination file. And you > >need all the RAM, too. It's like the worst case of "rsync -H". > > > What I tried to point out is that when the DB is u

Re: rsycing very small files

2009-02-28 Thread Jamie Lokier
lewis butler wrote: > On 27-Feb-2009, at 21:16, Daniel.Li wrote: > > On Fri, 2009-02-27 at 08:20 -0500, Mag Gam wrote: > >>it works. But takes hours to do it. Was wondering if there was a > >>faster way > > > >How much speed do u get to backup these files? Average? > > I would thing that rsync 3

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
Jamie Lokier wrote: > David Howe wrote: > > Jamie Lokier wrote: > > I am less worried about individual file renames and/or "missing" the > > opportunity to diff a large file that has been both moved and updated, > > than having to resync multiple gigs of stu

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
David Howe wrote: > Jamie Lokier wrote: > I am less worried about individual file renames and/or "missing" the > opportunity to diff a large file that has been both moved and updated, > than having to resync multiple gigs of stuff over a slow link, because > some user

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: > >But you need to verify and update the DB contents - which requires > >stat on all the files mentioned in the DB. In other words you might > >have to scan everything :-) > > > This already takes place while Rsync does its job, so it has not to be > done separat

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: > The highest speed and efficiency is to only observe time and size as > then just a stat-call is needed. But in more complex situations you > have to take also the checksum, inode-number, etc into account. In > previous posts there were many ideas to cope with thi

Re: Renamed files and directories

2009-02-25 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: > Hmmm, right, IF and only IF you notice the rename at the source on > time, you can do so at destination. But in practise, I see its > getting more and more impossible to keep up with the growing number > of hosts. Just keeping a DB with characteristics like check

Re: Renamed files and directories

2009-02-25 Thread Jamie Lokier
David Howe wrote: > N.J. van der Horn (Nico) wrote: > > What is the current status of both rename-patches ? > > Are there alternative measures ? > > > > Frequently users reorganise directories and files. > > Recently a directory of 40GB was renamed... > > It took 3 weeks to re-copy all over an ADS

Re: [patch] Replace illegal characters in filenames for FAT (switch)

2009-01-28 Thread Jamie Lokier
Matt McCutchen wrote: > > I'm not sure that's a grand idea really, considering the --tr option is > > obviously far more flexible, and users might not agree what it should be > > an alias for (less arguing when it's all hardcoded :)). > > I was suggesting that you define the alias for your own mac

Re: Windows 2003 Cygwin Netapp remote filesystem

2008-12-26 Thread Jamie Lokier
Michael Chletsos wrote: > Ok so I have figured out the problem with my rsync daemon is the fact > that rsync interprets // as / and therefore is not seeing this as a > unc path, but rather a absolute path. > > and /cygdrive/h does not work because it is not setup outside of the > cygwin environmen

Re: UDP

2008-12-08 Thread Jamie Lokier
Charles Darwin wrote: > My question is does rsync use UDP? If not by defaut, then how do I > enable it? Can I compile rsync with UDP as default protocol? rsync uses a byte stream over TCP, SSH, or any other application implementing a byte stream that you want, using the "-e" option. UDP is not

Re: rsync / checksum small block / xfer small block

2008-12-04 Thread Jamie Lokier
alexus wrote: > okay, so you saying if i have large db, and i made a change rsync will > not re-transfer the whole file, it will just transfer small portion of > that file? am I correct? does it say something like that in > documentation anywhere? In the very first paragraph of the manual:

Re: rsync / checksum small block / xfer small block

2008-12-03 Thread Jamie Lokier
alexus wrote: > not quite what i need > > let's take another example > > let's say i have a mysql db, and only few rows gets changed on daily > basis, yet that data file itself is huge, so rsync checks for checksum > sees that it's different and xfer the whole file, i use remote site, > so xfer t

Re: Rsync of LVM Snapshots copies whole file

2008-08-21 Thread Jamie Lokier
Paul Slootman wrote: > If you don't mind that the destination copy is invalid for some time > (e.g. if it's just used for backup), _and_ you know that data won't be > moved, only updated at random places, you might try --inplace. That way > the existing copy is updated, instead of copying the data

Re: Restoring from backup, preserving uids

2008-08-11 Thread Jamie Lokier
Matt McCutchen wrote: > > Another problem would likely be with bash and sh, since sh is often a > > hard link to bash (at least on OS X it is). Having sh be a different > > version is probably not a Good Thing. > > On my brother's OS X system, sh and bash are different binaries. They're diff

Re: Major rsync issue - overwriting files!!!

2008-03-28 Thread Jamie Lokier
Nathan Griffiths wrote: > The thing is, the same script (with different source/destination > variables) runs FLAWLESSLY on another file server! I'm thinking the spaces in some of your paths are quite significant. The shell script does not quote spaces properly in $LINK_DEST, when that's saved in

Re: Bad habits

2008-03-20 Thread Jamie Lokier
Mike Frysinger wrote: > > Hence it doesn't make much sense to cc directly; respond to the mailing > > list and the response should reach the persons concerned. > > which gets back to the point of expecting people to be fully aware of the > mailing list policy. many contributors are subscribed to

Re: Could you make the 2.6.9 source available via http again?

2008-03-05 Thread Jamie Lokier
Tom May wrote: > I noticed you've made rsync 3.0.0 available at > http://www.samba.org/ftp/rsync/rsync-3.0.0.tar.gz, which is great! > > However, until it's upgraded (if ever), the buildroot system I'm > using to build embedded linux tries to download rsync 2.6.9 from > http://www.samba.org/ftp/rs

Re: --o-direct option

2008-02-27 Thread Jamie Lokier
Jerome Haltom wrote: > The problem is that during the rsync process the user's machine is > barely usable. The reason is because rsync reads these 2GB files... many > GBs of them. This causes the user's machine to repeatidly trash the page > cache. This really is Linux's fault. It should realize th

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: > The patch truncates the file with ftruncate if a transfer fails in > receiver.c. This should avoid the problem you mention. I was thinking of a user-abort (Control-C) or crash, but this is good. > Even if this didn't > occur, the file would exist on the FS with the predefined

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: > > Was that simply due to writing too-small block to NTFS? In other > > words, would increasing the size of write() calls have fixed it > > instead, without leaving allocated but unused disk space in the case > > of a user-abort with --partial, --partial-dir or --inplace? > > It

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: > > Though, did I get the right impression that NTFS generates lots of > > extents for small writes even when nothing else is running? > > The fragmentation on NTFS was a problem even when nothing else was running > on the server. The preallocation patch made all the difference o

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: > > Any idea why Glibc's posix_fallocate makes any difference? > > > > Doesn't it simply write a lot of zeros? In that case, why doesn't > > rsync writing lots of data sequentially result in the same number of > > extents? > > The destination server had a lot of other processes r

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: > Destination file on XFS > - ftruncate, 59GB file, Execution time 52776 secs, 1235 extents > - posix_fallocate, 59GB file, Execution time 53919 secs, 11 extents Any idea why Glibc's posix_fallocate makes any difference? Doesn't it simply write a lot of zeros? In th

Re: [PATCH] Prefer fallocate if available, and use posix_fallocate only on Cygwin.

2008-02-25 Thread Jamie Lokier
Wayne Davison wrote: > On Sun, Feb 24, 2008 at 04:29:17PM +0000, Jamie Lokier wrote: > > When compiled on a Linux system which doesn't have SYS_fallocate > > (because it doesn't have a very recent kernel), but does have > > posix_fallocate (because Glibc has ha

Re: [PATCH] Prefer fallocate if available, and use posix_fallocate only on Cygwin.

2008-02-24 Thread Jamie Lokier
Wayne Davison wrote: > On Sat, Feb 23, 2008 at 10:49:25PM -0500, Matt McCutchen wrote: > > This change on top of the current preallocate branch implements the > > behavior I described of using fallocate if available (Linux syscall that > > uses filesystem-level preallocation support) or otherwise >

Re: preallocate CPU usage - pre4

2008-02-23 Thread Jamie Lokier
Rob Bosch wrote: > I took a stab at modifying the preallocate.diff patch file replacing it with > ftruncate (attached). Do you think the file looks OK for Linux (obviously > cygwin should use posix_fallocate)? I replaced posix_fallocate with > ftruncate and also removed the check for HAVE_POSIX_F

Re: Is there a way to force rsync to be monothreaded (ie to don't fork)?

2008-02-21 Thread Jamie Lokier
Matt McCutchen wrote: > There are two forks on a local run: the sender forks off the generator, > and the generator forks off the receiver. You can eliminate the first > fork by accessing either the source or the destination via ssh to > localhost (or, equivalently, "support/lsh" in the rsync sour

Re: Is there a way to force rsync to be monothreaded (ie to don't fork)?

2008-02-21 Thread Jamie Lokier
Vitorio wrote: > Hello people, > the questio is all in the subject: Is there a way to force rsync to > be monothreaded (ie to don't fork)? > The reason for this is that the Carbon API isn't fork-safe and > fonction calls I do to the pretiger resource fork randomly don't work > when rsync fork

Re: preallocate CPU usage - pre4

2007-11-18 Thread Jamie Lokier
Matt McCutchen wrote: > I notice that the Linux kernel 2.6.23 has gained a system call > "fallocate" that preallocates at the filesystem level like Cygwin's > implementation of posix_fallocate; thus, preallocation may become (at > least slightly) helpful on Linux. Unfortunately, neither ext2 nor >

Re: rsyncing files that might change

2007-11-01 Thread Jamie Lokier
Franc Carter wrote: > >Unfortunately, yes. > Shouldn't that be caught by the fact that the source file has a new > (or at least different) time stamp now? > >Sorry, I should have given a clearer example. >All in one second >1. a process modifies the file and h

Re: micro rsync

2007-10-30 Thread Jamie Lokier
Paul Slootman wrote: > > Just wondering if anybody has thought about this. I would like to > > attempt to setup a router with dd-wrt and a NAS device as a home > > backup system without a computer. The processor is 200+MHz and I > > can have a maximum of 6MB on the device and a mounted samba > >

Re: can rsync scan files only with mtime since T?

2007-08-24 Thread Jamie Lokier
Matt McCutchen wrote: > > Is this doable with current rsync? > > No. A request for enhancement has been entered for a --newer option > that would do this: https://bugzilla.samba.org/show_bug.cgi?id=2423 . > At present, I can think of two things you might try: > > 1. Use `find' to list the files

Re: rsync replacement

2007-07-18 Thread Jamie Lokier
Jason Haar wrote: > Jamie Lokier wrote: > > Check out the "TCP: advanced congestion control" option in a 2.6 Linux > > kernel, and there is plenty of research on the topic. See SCTP and > > DSCP (among others) for the more transaction oriented side. > > >

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Matt McCutchen wrote: > Thus, syncdat gets #2 and #3 but (it seems) not #1. Rsync running on > a TCP-over-MTP tunnel would get #1 and #2 but not #3. To get all > three benefits, we would need to make a program that has both delta > transmission like rsync and a parallelized protocol like syncdat

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Mike Jackson wrote: >We are not claiming superiority, just that we provide performance >gains over TCP when going over wan or congested networks. In-fact, we >have a ftp server set up in Singapore if you would like to compare our >technology to your ftp solution. you can fin

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Andreas Kotes wrote: > seems like they've implemented something similiar TCP on top of UDP > which does a seriously better job (the information they provide points > in that direction). Shame they don't give it to the public for free, > like they got TCP, UDP, IP, DNS, SMTP, HTTP, ... ... ... > >

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Matt McCutchen wrote: > > Does anyone have any experience with 'syncdat' from Data > >Expedition? How does it compare to rsync? > > I looked at the syncdat feature list ( > http://www.dataexpedition.com/syncdat/features.html ). Aside from the > claim of much better performance, syncdat appears

Re: DO NOT REPLY [Bug 4768] problem sync big filesystem over slow connection

2007-07-13 Thread Jamie Lokier
[EMAIL PROTECTED] wrote: > (In reply to comment #4) > > i can't find the option --no-tweak-hlinked in rsync. > > That's because there is no such option in rsync. It's a proposed patch. It's > also not needed if you use --ignore-existing, as I suggested. > > I'm not planning to add the option yo

Re: checksum-xattr.diff [CVS update: rsync/patches]

2007-06-30 Thread Jamie Lokier
Matt McCutchen wrote: > Second, it is impossible to make xattr-based checksum caching > foolproof against same-second modification. Suppose a file is written > during second 5 and then rsync caches its checksum during second 8; > now the file has mtime 5 and ctime 8. Sometime later, rsync notices

Re: rsync mechanics question

2007-05-09 Thread Jamie Lokier
Tom Riley wrote: > However, the curiosity comes in with my source data taking up 86gigs of > data on a 100g partition, and as the copy progresses the destination > drive is reporting 240 gigs of usage. > > So as far as I can tell, rsync is working and the data integrity seems > good, it's simply t

Re: rsync & SSL 'for real'

2007-04-21 Thread Jamie Lokier
Andreas Kotes wrote: > > >> There is no license issue. > > There would be a serious licence issue the other way round, but BSD is a > tad more permissive than the GPL is, so - no problem there BUT: there is > an advertisement clause, so rsync would need to display certain messages > when compiled

Re: rsyncing many files and hard links: optimisation suggestions?

2006-09-29 Thread Jamie Lokier
Judith Retief wrote: > If the problem is the actual disk access, then I can't think of anything to > do. If it is the sorting, then cutting down the batch sizes should help, at > the expense of having copies of some files rather than hard links. You can tell whether it's the disk accesses or the s

Re: Rsyncing a very large directory tree (over 50,000 files)

2006-06-14 Thread Jamie Lokier
Matt McCutchen wrote: > Yes, rsync will send the complete file list each time it runs. It seems > odd to me that building the file list would take 15 minutes; when I back > up the system partition of my computer (300,000 files) rsync takes > perhaps 5 minutes to build the file list. That surely d

Re: Usage of a telnet client instead of ssh or rsh

2006-05-15 Thread Jamie Lokier
Mike Daws wrote: > > ssh isn't always an option. E.g. to reach HP's testdrive machines, > > telnet is the only available option. > > > > I've done rsync over telnet, in binary mode and with the terminal set > > to raw, using Perl and the Perl Net::Telnet module, and it mostly > > worked but there

Re: Usage of a telnet client instead of ssh or rsh

2006-05-13 Thread Jamie Lokier
Jan-Benedict Glaw wrote: > On Sat, 2006-05-13 20:27:03 +0200, Paul Slootman <[EMAIL PROTECTED]> wrote: > > On Fri 12 May 2006, Matt McCutchen wrote: > > > Wayne beat me to it. But I was going to say, you might be able to write > > > a wrapper script that sends the rsync command and arguments down

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
Shachar Shemesh wrote: > >While you're there, one little trick I've found that speeds up > >scanning large directory hierarchies is to stat() or open() entries in > >inode-number order. For some filesystems it makes no difference, but > >for others it reduces the average disk seek time as on many

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
Wayne Davison wrote: > On Mon, Mar 06, 2006 at 07:18:45PM +0200, Shachar Shemesh wrote: > > In fact, I know of at least one place where they don't use rsync because > > they don't have enough RAM+SWAP to hold the list of files in memory. > > > > As far as future directions for rsync, I think this

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
Shachar Shemesh wrote: > >Hmm. My home directory, on my laptop (a mere 60GB disk), does contain > >millions of files, and it takes about 20 minutes to build the list on > >a good day. 100Mbps network, but it's I/O bound not network bound. > > > >It looks a lot like the number of files is more sig

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
jp wrote: > 100gb of 4-40MB files sounds like my home PC full of digital photos I've > taken. It backs up to a linux PC right beside it with rsync. I don't > really call it that big a project for rsync. Big things for rsync are > millions of files. At 100mbps, it takes a few seconds to build the

Re: chmod/chown on receiver

2006-02-15 Thread Jamie Lokier
Wayne Davison wrote: > On Tue, Feb 14, 2006 at 10:17:51AM +0100, Blickwinkel wrote: > > Thanks, I was trying your hint with the su command, but somehow > > "--server" seems to get passed to su and fails: > > That is a GNU thing with them reordering options unless POSIXLY_CORRECT > is set to "1" i

Re: information on identifying hard links to a file

2006-02-09 Thread Jamie Lokier
Wayne Davison wrote: > On Thu, Feb 09, 2006 at 03:04:17PM +0100, Paul Slootman wrote: > > compare inode and device number. When those are the same, the two files > > must be hardlinked. > > Also, rsync only considers files that have a link count larger than 1 > (see stat()'s st_nlink) since this a

Re: Rsync 2.6.7pre1 is now available

2006-02-09 Thread Jamie Lokier
Wayne Davison wrote: > > - (below) in order to have the rules that are read-in from the file > > + (below) in order to have the rules that are read in from the file > > I consider the original a good use of hyphenation to help distinguish > the phrase "are read-in from a file" (using the past-

Re: Reading for area during rsync

2006-01-31 Thread Jamie Lokier
Matt McCutchen wrote: > > But this won't work if the change that occurred on the sending side > > after the transfer started happens within the same second, and the > > mtimes have only one second resolution, will it? > > > > That's quite likely, if the file is reasonably small and the first > > r

Re: Reading for area during rsync

2006-01-30 Thread Jamie Lokier
Wayne Davison wrote: > > There is an area I would like to rsync with remote site. Is there a > > problem reading/writing to that area during the time rsync is in > > progress? > > Rsync should handle changes fairly well, but it is not perfect. If the > currently-active file is changed while it is

Re: Any change of rsync using threads instead of fork?

2005-12-19 Thread Jamie Lokier
Wayne Davison wrote: > On Sat, Dec 17, 2005 at 11:22:58PM +0000, Jamie Lokier wrote: > > rsync: mkstemp "/mnt/storage/bin/" failed: Success (0) > > That makes me wonder if the thread handling is not properly giving > each thread its own errno. I agree, that seems li

Re: Any change of rsync using threads instead of fork?

2005-12-17 Thread Jamie Lokier
So far I am not having luck with the threads version: rsync: mkstemp "/mnt/storage/bin/" failed: Success (0) ./rsync: io.c: 334: push_redo_num: Assertion `am_receiver()' failed. is typical. Or SIGSEGV. There is something very fishy going on, and I suspect it isn't the rsync code, but someth

Re: Any change of rsync using threads instead of fork?

2005-12-09 Thread Jamie Lokier
Wayne Davison wrote: > On Thu, Nov 24, 2005 at 03:22:23AM +0000, Jamie Lokier wrote: > > Is there any likelihood of changing the rsync code to use threads > > instead of processes? > > I just thought about this a bit more, and it didn't seem as large a task > as I ha

Re: Any change of rsync using threads instead of fork?

2005-12-09 Thread Jamie Lokier
Nelson H. F. Beebe wrote: > List traffic today asks about changing rsync to use lightweight > threads instead of heavyweight fork. > > Before rushing into building a threads version of rsync, please READ > this recent article You didn't post a link directly to the article, just to the gateway pag

Any change of rsync using threads instead of fork?

2005-11-23 Thread Jamie Lokier
On a typical embedded Linux device, with no MMU, there is no fork() or it returns ENOSYS. The nearest replacements are vfork() (which is only useful before exec*()), or to create threads with pthread_create(). rsync would be a very useful program on such devices, and I was a bit disappointed to b