Any change of rsync using threads instead of fork?

2005-11-23 Thread Jamie Lokier
On a typical embedded Linux device, with no MMU, there is no fork() or it returns ENOSYS. The nearest replacements are vfork() (which is only useful before exec*()), or to create threads with pthread_create(). rsync would be a very useful program on such devices, and I was a bit disappointed to

Re: Any change of rsync using threads instead of fork?

2005-12-09 Thread Jamie Lokier
Nelson H. F. Beebe wrote: List traffic today asks about changing rsync to use lightweight threads instead of heavyweight fork. Before rushing into building a threads version of rsync, please READ this recent article You didn't post a link directly to the article, just to the gateway page.

Re: Any change of rsync using threads instead of fork?

2005-12-09 Thread Jamie Lokier
Wayne Davison wrote: On Thu, Nov 24, 2005 at 03:22:23AM +, Jamie Lokier wrote: Is there any likelihood of changing the rsync code to use threads instead of processes? I just thought about this a bit more, and it didn't seem as large a task as I had originally assumed it would be. I

Re: Any change of rsync using threads instead of fork?

2005-12-17 Thread Jamie Lokier
So far I am not having luck with the threads version: rsync: mkstemp /mnt/storage/bin/ failed: Success (0) ./rsync: io.c: 334: push_redo_num: Assertion `am_receiver()' failed. is typical. Or SIGSEGV. There is something very fishy going on, and I suspect it isn't the rsync code, but

Re: Any change of rsync using threads instead of fork?

2005-12-19 Thread Jamie Lokier
Wayne Davison wrote: On Sat, Dec 17, 2005 at 11:22:58PM +, Jamie Lokier wrote: rsync: mkstemp /mnt/storage/bin/ failed: Success (0) That makes me wonder if the thread handling is not properly giving each thread its own errno. I agree, that seems likely. Unfortunately I don't have

Re: Reading for area during rsync

2006-01-30 Thread Jamie Lokier
Wayne Davison wrote: There is an area I would like to rsync with remote site. Is there a problem reading/writing to that area during the time rsync is in progress? Rsync should handle changes fairly well, but it is not perfect. If the currently-active file is changed while it is being

Re: Reading for area during rsync

2006-01-31 Thread Jamie Lokier
Matt McCutchen wrote: But this won't work if the change that occurred on the sending side after the transfer started happens within the same second, and the mtimes have only one second resolution, will it? That's quite likely, if the file is reasonably small and the first rsync

Re: Rsync 2.6.7pre1 is now available

2006-02-09 Thread Jamie Lokier
Wayne Davison wrote: - (below) in order to have the rules that are read-in from the file + (below) in order to have the rules that are read in from the file I consider the original a good use of hyphenation to help distinguish the phrase are read-in from a file (using the past-tense of

Re: information on identifying hard links to a file

2006-02-09 Thread Jamie Lokier
Wayne Davison wrote: On Thu, Feb 09, 2006 at 03:04:17PM +0100, Paul Slootman wrote: compare inode and device number. When those are the same, the two files must be hardlinked. Also, rsync only considers files that have a link count larger than 1 (see stat()'s st_nlink) since this allows

Re: chmod/chown on receiver

2006-02-15 Thread Jamie Lokier
Wayne Davison wrote: On Tue, Feb 14, 2006 at 10:17:51AM +0100, Blickwinkel wrote: Thanks, I was trying your hint with the su command, but somehow --server seems to get passed to su and fails: That is a GNU thing with them reordering options unless POSIXLY_CORRECT is set to 1 in the

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
jp wrote: 100gb of 4-40MB files sounds like my home PC full of digital photos I've taken. It backs up to a linux PC right beside it with rsync. I don't really call it that big a project for rsync. Big things for rsync are millions of files. At 100mbps, it takes a few seconds to build the

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
Shachar Shemesh wrote: Hmm. My home directory, on my laptop (a mere 60GB disk), does contain millions of files, and it takes about 20 minutes to build the list on a good day. 100Mbps network, but it's I/O bound not network bound. It looks a lot like the number of files is more significant

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
Wayne Davison wrote: On Mon, Mar 06, 2006 at 07:18:45PM +0200, Shachar Shemesh wrote: In fact, I know of at least one place where they don't use rsync because they don't have enough RAM+SWAP to hold the list of files in memory. As far as future directions for rsync, I think this is the

Re: Question about rsync and BIG mirror

2006-03-06 Thread Jamie Lokier
Shachar Shemesh wrote: While you're there, one little trick I've found that speeds up scanning large directory hierarchies is to stat() or open() entries in inode-number order. For some filesystems it makes no difference, but for others it reduces the average disk seek time as on many common

Re: Usage of a telnet client instead of ssh or rsh

2006-05-13 Thread Jamie Lokier
Jan-Benedict Glaw wrote: On Sat, 2006-05-13 20:27:03 +0200, Paul Slootman [EMAIL PROTECTED] wrote: On Fri 12 May 2006, Matt McCutchen wrote: Wayne beat me to it. But I was going to say, you might be able to write a wrapper script that sends the rsync command and arguments down the

Re: Usage of a telnet client instead of ssh or rsh

2006-05-15 Thread Jamie Lokier
Mike Daws wrote: ssh isn't always an option. E.g. to reach HP's testdrive machines, telnet is the only available option. I've done rsync over telnet, in binary mode and with the terminal set to raw, using Perl and the Perl Net::Telnet module, and it mostly worked but there were

Re: Rsyncing a very large directory tree (over 50,000 files)

2006-06-14 Thread Jamie Lokier
Matt McCutchen wrote: Yes, rsync will send the complete file list each time it runs. It seems odd to me that building the file list would take 15 minutes; when I back up the system partition of my computer (300,000 files) rsync takes perhaps 5 minutes to build the file list. That surely

Re: rsyncing many files and hard links: optimisation suggestions?

2006-09-29 Thread Jamie Lokier
Judith Retief wrote: If the problem is the actual disk access, then I can't think of anything to do. If it is the sorting, then cutting down the batch sizes should help, at the expense of having copies of some files rather than hard links. You can tell whether it's the disk accesses or the

Re: rsync SSL 'for real'

2007-04-21 Thread Jamie Lokier
Andreas Kotes wrote: There is no license issue. There would be a serious licence issue the other way round, but BSD is a tad more permissive than the GPL is, so - no problem there BUT: there is an advertisement clause, so rsync would need to display certain messages when compiled with

Re: rsync mechanics question

2007-05-09 Thread Jamie Lokier
Tom Riley wrote: However, the curiosity comes in with my source data taking up 86gigs of data on a 100g partition, and as the copy progresses the destination drive is reporting 240 gigs of usage. So as far as I can tell, rsync is working and the data integrity seems good, it's simply taking

Re: checksum-xattr.diff [CVS update: rsync/patches]

2007-06-30 Thread Jamie Lokier
Matt McCutchen wrote: Second, it is impossible to make xattr-based checksum caching foolproof against same-second modification. Suppose a file is written during second 5 and then rsync caches its checksum during second 8; now the file has mtime 5 and ctime 8. Sometime later, rsync notices

Re: DO NOT REPLY [Bug 4768] problem sync big filesystem over slow connection

2007-07-13 Thread Jamie Lokier
[EMAIL PROTECTED] wrote: (In reply to comment #4) i can't find the option --no-tweak-hlinked in rsync. That's because there is no such option in rsync. It's a proposed patch. It's also not needed if you use --ignore-existing, as I suggested. I'm not planning to add the option you

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Andreas Kotes wrote: seems like they've implemented something similiar TCP on top of UDP which does a seriously better job (the information they provide points in that direction). Shame they don't give it to the public for free, like they got TCP, UDP, IP, DNS, SMTP, HTTP, ... ... ...

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Mike Jackson wrote: We are not claiming superiority, just that we provide performance gains over TCP when going over wan or congested networks. In-fact, we have a ftp server set up in Singapore if you would like to compare our technology to your ftp solution. you can find

Re: rsync replacement

2007-07-17 Thread Jamie Lokier
Matt McCutchen wrote: Thus, syncdat gets #2 and #3 but (it seems) not #1. Rsync running on a TCP-over-MTP tunnel would get #1 and #2 but not #3. To get all three benefits, we would need to make a program that has both delta transmission like rsync and a parallelized protocol like syncdat and

Re: rsync replacement

2007-07-18 Thread Jamie Lokier
Jason Haar wrote: Jamie Lokier wrote: Check out the TCP: advanced congestion control option in a 2.6 Linux kernel, and there is plenty of research on the topic. See SCTP and DSCP (among others) for the more transaction oriented side. Hi there Jamie Like yourself, our WAN (VPN over

Re: can rsync scan files only with mtime since T?

2007-08-24 Thread Jamie Lokier
Matt McCutchen wrote: Is this doable with current rsync? No. A request for enhancement has been entered for a --newer option that would do this: https://bugzilla.samba.org/show_bug.cgi?id=2423 . At present, I can think of two things you might try: 1. Use `find' to list the files that

Re: micro rsync

2007-10-30 Thread Jamie Lokier
Paul Slootman wrote: Just wondering if anybody has thought about this. I would like to attempt to setup a router with dd-wrt and a NAS device as a home backup system without a computer. The processor is 200+MHz and I can have a maximum of 6MB on the device and a mounted samba storage

Re: rsyncing files that might change

2007-11-01 Thread Jamie Lokier
Franc Carter wrote: Unfortunately, yes. Shouldn't that be caught by the fact that the source file has a new (or at least different) time stamp now? Sorry, I should have given a clearer example. All in one second 1. a process modifies the file and hence

Re: preallocate CPU usage - pre4

2007-11-18 Thread Jamie Lokier
Matt McCutchen wrote: I notice that the Linux kernel 2.6.23 has gained a system call fallocate that preallocates at the filesystem level like Cygwin's implementation of posix_fallocate; thus, preallocation may become (at least slightly) helpful on Linux. Unfortunately, neither ext2 nor

Re: Is there a way to force rsync to be monothreaded (ie to don't fork)?

2008-02-21 Thread Jamie Lokier
Vitorio wrote: Hello people, the questio is all in the subject: Is there a way to force rsync to be monothreaded (ie to don't fork)? The reason for this is that the Carbon API isn't fork-safe and fonction calls I do to the pretiger resource fork randomly don't work when rsync forks,

Re: Is there a way to force rsync to be monothreaded (ie to don't fork)?

2008-02-21 Thread Jamie Lokier
Matt McCutchen wrote: There are two forks on a local run: the sender forks off the generator, and the generator forks off the receiver. You can eliminate the first fork by accessing either the source or the destination via ssh to localhost (or, equivalently, support/lsh in the rsync source

Re: preallocate CPU usage - pre4

2008-02-23 Thread Jamie Lokier
Rob Bosch wrote: I took a stab at modifying the preallocate.diff patch file replacing it with ftruncate (attached). Do you think the file looks OK for Linux (obviously cygwin should use posix_fallocate)? I replaced posix_fallocate with ftruncate and also removed the check for

Re: [PATCH] Prefer fallocate if available, and use posix_fallocate only on Cygwin.

2008-02-24 Thread Jamie Lokier
Wayne Davison wrote: On Sat, Feb 23, 2008 at 10:49:25PM -0500, Matt McCutchen wrote: This change on top of the current preallocate branch implements the behavior I described of using fallocate if available (Linux syscall that uses filesystem-level preallocation support) or otherwise

Re: [PATCH] Prefer fallocate if available, and use posix_fallocate only on Cygwin.

2008-02-25 Thread Jamie Lokier
Wayne Davison wrote: On Sun, Feb 24, 2008 at 04:29:17PM +, Jamie Lokier wrote: When compiled on a Linux system which doesn't have SYS_fallocate (because it doesn't have a very recent kernel), but does have posix_fallocate (because Glibc has had it for a while), I think that patch

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: Destination file on XFS - ftruncate, 59GB file, Execution time 52776 secs, 1235 extents - posix_fallocate, 59GB file, Execution time 53919 secs, 11 extents Any idea why Glibc's posix_fallocate makes any difference? Doesn't it simply write a lot of zeros? In that

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: Any idea why Glibc's posix_fallocate makes any difference? Doesn't it simply write a lot of zeros? In that case, why doesn't rsync writing lots of data sequentially result in the same number of extents? The destination server had a lot of other processes running at

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: Though, did I get the right impression that NTFS generates lots of extents for small writes even when nothing else is running? The fragmentation on NTFS was a problem even when nothing else was running on the server. The preallocation patch made all the difference on NTFS

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: Was that simply due to writing too-small block to NTFS? In other words, would increasing the size of write() calls have fixed it instead, without leaving allocated but unused disk space in the case of a user-abort with --partial, --partial-dir or --inplace? It could

Re: Fragmentation on XFS

2008-02-25 Thread Jamie Lokier
Rob Bosch wrote: The patch truncates the file with ftruncate if a transfer fails in receiver.c. This should avoid the problem you mention. I was thinking of a user-abort (Control-C) or crash, but this is good. Even if this didn't occur, the file would exist on the FS with the predefined

Re: --o-direct option

2008-02-27 Thread Jamie Lokier
Jerome Haltom wrote: The problem is that during the rsync process the user's machine is barely usable. The reason is because rsync reads these 2GB files... many GBs of them. This causes the user's machine to repeatidly trash the page cache. This really is Linux's fault. It should realize the

Re: Could you make the 2.6.9 source available via http again?

2008-03-05 Thread Jamie Lokier
Tom May wrote: I noticed you've made rsync 3.0.0 available at http://www.samba.org/ftp/rsync/rsync-3.0.0.tar.gz, which is great! However, until it's upgraded (if ever), the buildroot system I'm using to build embedded linux tries to download rsync 2.6.9 from

Re: Bad habits

2008-03-20 Thread Jamie Lokier
Mike Frysinger wrote: Hence it doesn't make much sense to cc directly; respond to the mailing list and the response should reach the persons concerned. which gets back to the point of expecting people to be fully aware of the mailing list policy. many contributors are subscribed to many

Re: Major rsync issue - overwriting files!!!

2008-03-28 Thread Jamie Lokier
Nathan Griffiths wrote: The thing is, the same script (with different source/destination variables) runs FLAWLESSLY on another file server! I'm thinking the spaces in some of your paths are quite significant. The shell script does not quote spaces properly in $LINK_DEST, when that's saved in

Re: Rsync of LVM Snapshots copies whole file

2008-08-21 Thread Jamie Lokier
Paul Slootman wrote: If you don't mind that the destination copy is invalid for some time (e.g. if it's just used for backup), _and_ you know that data won't be moved, only updated at random places, you might try --inplace. That way the existing copy is updated, instead of copying the data to

Re: rsync / checksum small block / xfer small block

2008-12-03 Thread Jamie Lokier
alexus wrote: not quite what i need let's take another example let's say i have a mysql db, and only few rows gets changed on daily basis, yet that data file itself is huge, so rsync checks for checksum sees that it's different and xfer the whole file, i use remote site, so xfer takes

Re: rsync / checksum small block / xfer small block

2008-12-04 Thread Jamie Lokier
alexus wrote: okay, so you saying if i have large db, and i made a change rsync will not re-transfer the whole file, it will just transfer small portion of that file? am I correct? does it say something like that in documentation anywhere? In the very first paragraph of the manual: It

Re: UDP

2008-12-08 Thread Jamie Lokier
Charles Darwin wrote: My question is does rsync use UDP? If not by defaut, then how do I enable it? Can I compile rsync with UDP as default protocol? rsync uses a byte stream over TCP, SSH, or any other application implementing a byte stream that you want, using the -e option. UDP is not a

Re: Windows 2003 Cygwin Netapp remote filesystem

2008-12-26 Thread Jamie Lokier
Michael Chletsos wrote: Ok so I have figured out the problem with my rsync daemon is the fact that rsync interprets // as / and therefore is not seeing this as a unc path, but rather a absolute path. and /cygdrive/h does not work because it is not setup outside of the cygwin environment,

Re: Renamed files and directories

2009-02-25 Thread Jamie Lokier
David Howe wrote: N.J. van der Horn (Nico) wrote: What is the current status of both rename-patches ? Are there alternative measures ? Frequently users reorganise directories and files. Recently a directory of 40GB was renamed... It took 3 weeks to re-copy all over an ADSL-link.

Re: Renamed files and directories

2009-02-25 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: Hmmm, right, IF and only IF you notice the rename at the source on time, you can do so at destination. But in practise, I see its getting more and more impossible to keep up with the growing number of hosts. Just keeping a DB with characteristics like checksum

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: The highest speed and efficiency is to only observe time and size as then just a stat-call is needed. But in more complex situations you have to take also the checksum, inode-number, etc into account. In previous posts there were many ideas to cope with this.

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: But you need to verify and update the DB contents - which requires stat on all the files mentioned in the DB. In other words you might have to scan everything :-) This already takes place while Rsync does its job, so it has not to be done separately.

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
David Howe wrote: Jamie Lokier wrote: I am less worried about individual file renames and/or missing the opportunity to diff a large file that has been both moved and updated, than having to resync multiple gigs of stuff over a slow link, because some user renamed a directory. An approximate

Re: Renamed files and directories

2009-02-26 Thread Jamie Lokier
Jamie Lokier wrote: David Howe wrote: Jamie Lokier wrote: I am less worried about individual file renames and/or missing the opportunity to diff a large file that has been both moved and updated, than having to resync multiple gigs of stuff over a slow link, because some user renamed

Re: rsycing very small files

2009-02-28 Thread Jamie Lokier
lewis butler wrote: On 27-Feb-2009, at 21:16, Daniel.Li wrote: On Fri, 2009-02-27 at 08:20 -0500, Mag Gam wrote: it works. But takes hours to do it. Was wondering if there was a faster way How much speed do u get to backup these files? Average? I would thing that rsync 3.x would make

Re: Renamed files and directories

2009-03-02 Thread Jamie Lokier
N.J. van der Horn (Nico) wrote: Right, but it has to be done in a separate pass if you're to compare all files with each other, not just one destination file. And you need all the RAM, too. It's like the worst case of rsync -H. What I tried to point out is that when the DB is updated

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Jamie Lokier
Kyle Lanclos wrote: Peter Salameh wrote: One of the speed-limiting issues with rsync is having to send huge file lists when mirroring large file systems, even for incremental updates where only a small part of the file system might have changed. Personally, I find that the sending of

Re: proposal to speed rsync with lots of files

2009-03-05 Thread Jamie Lokier
Peter Salameh wrote: My proposal is to first send a checksum of the file list for each directory. If is found to be identical to the same checksum on the remote side then the list need not be sent for that directory! ... It might even be possible to use the rsync checksum algorithm on the

Re: dynamic bandwidth allocation

2009-03-14 Thread Jamie Lokier
Felipe Alvarez wrote: Hi list I was wondering if this is (or could be) possible with rsync. I was wondering if there was a way to change the bandwidth limit (--bwlimit=KBPS) dynamically while rsync is running? Could this be possible maybe with signals like USR1 add 5KBPS and USR2 subtracts

Re: Weird rsync or shell problem?

2009-03-22 Thread Jamie Lokier
Wayne Davison wrote: On Sat, Mar 21, 2009 at 07:23:58PM +0100, axel.we...@cbc.de wrote: SCHNITT/Render Files/#342#200#242Vorlage Bleskin_990101_NB_n-tv/Vorlage Blesk-FIN-000b The character sequence 0343 0200 0242 is E2 80 A2 in hex. Seems to be a common sequence for something (since

Re: Is it possible to make rsync VMware split .vmdk's aware?

2009-03-26 Thread Jamie Lokier
Bas Bahlmann || Steady IT Systeembeheer wrote: Hi Matt, I am working a couple of days now with monolithic files and I keep hitting io time outs on large vmdk's while the line (IPSec tunnel) stays online. I don't know if it's relevant, but rsync can be extremely slow at transmitting large

Re: Rolling algorithm vs CRC

2009-05-22 Thread Jamie Lokier
Ryan Malayter wrote: On Fri, May 22, 2009 at 9:02 AM, Hasanat Kazmi hasanatka...@gmail.com wrote: Hello, I have previously mailed on list that I am trying to port rsync to NT. I was wondering that whether CRC can be used to find check sums rather then rolling algorithm. I havnt found any

Re: Rolling algorithm vs CRC

2009-05-22 Thread Jamie Lokier
Hasanat Kazmi wrote: Hello, I have previously mailed on list that I am trying to port rsync to NT. I was wondering that whether CRC can be used to find check sums rather then rolling algorithm. I havnt found any document on web comparing rolling algorithm with CRC. It will only find

Re: Does rsync detect file corruption?

2009-05-24 Thread Jamie Lokier
Daniel Carrera wrote: But there is no way to distinguish between file corruption and a legitimate change. All you can do is keep old backups for a few days or weeks and hope that you detect the file corruption before the backup rotation deletes all the good copies. I'm under the impression

Re: Does rsync detect file corruption?

2009-05-24 Thread Jamie Lokier
Daniel Carrera wrote: Jamie Lokier wrote: Daniel Carrera wrote: But there is no way to distinguish between file corruption and a legitimate change. All you can do is keep old backups for a few days or weeks and hope that you detect the file corruption before the backup rotation deletes

Re: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)

2009-07-13 Thread Jamie Lokier
Ryan Malayter wrote: Your log file indicates that rsync is indeed working as designed finding lots of data matches: Literal data: 123736377 bytes Matched data: 17889663500 bytes This means that rsync only had to transfer 118 MB instead of 16+ GB. It does this by trading CPU and

Re: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)

2009-07-15 Thread Jamie Lokier
Ryan Malayter wrote: So, when transferring a large file, it goes something like this from the sender's perspective: 1) sending file list 2) receving file list 3) file is different! Recevier, please give me some hashes 4) wait 20+ minutes for receiver to compute hashes got hashes 5)

Re: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)

2009-08-26 Thread Jamie Lokier
JW wrote: On Monday 13 July 2009 14:18:38 Ryan Malayter wrote: It would be a big boost for large files if rsync remembered the hashes on each end, so it didn't have to re-read the files on every run if the files were unchanged. This is a feature that rsync's developers have rejected,

Re: Nice little performance improvement

2009-10-17 Thread Jamie Lokier
Mike Connell wrote: Hi, Interesting. If you're not using incremental recursion (the default in rsync = 3.0.0), I can see that the du would help by forcing the destination I/O to overlap the file-list building in time. But with incremental recursion, the du shouldn't be necessary because

Re: DO NOT REPLY [Bug 6881] New: --bwlimit option uses KiB/s, but is documented as (what amounts to) kB/s

2009-11-09 Thread Jamie Lokier
samba-b...@samba.org wrote: Given that this is a network transfer rate, it'd be more proper (and consistent with other applications) to change the function to work in SI kilobytes per second (i.e. use 1000 instead of 1024), but that's backwards-incompatible. If you'd like to go this route, I

Re: Will rsync support btrfs file system?

2009-12-06 Thread Jamie Lokier
Benjamin R. Haskell wrote: find /home/ -xdev -type d | sudo inotifywait --fromfile - -m | perl -lnwe 'BEGIN{$|=1;} print unless $h{$_}; $h{$_}++' The biggest problem in my experience is it can take 5 minutes of thrashing to set up the inotifies on a large /home directory, from cold cache and

Re: How do I make rsync ignore unreadable files (damaged sectors)?

2009-12-30 Thread Jamie Lokier
Matthias Schniedermeyer wrote: On 22.12.2009 19:24, Stefan Nowak wrote: On 22.12.2009 16:39, Stefan Nowak wrote: The only low-budget test ideas I have: The CD scratching a la Tomas Gustavsson seems the only easily achievable solution. But then it is not sure whether the OS does

Re: rsync/cwgwin hangs during transfers

2010-01-19 Thread Jamie Lokier
Steven Hartland wrote: We've never been able to use rsync on cygwin reliably. We tried loads of things and whenever its been brought up on the cygwin list, the answer has always been that its down to issues with pipes implementation, which was a short coming of the underlining OS so nothing

Re: Latency and Rsync Transfers

2010-02-01 Thread Jamie Lokier
Neal B wrote: Thanks for your reply. I have been experimenting with the buffer settings and when specifying it actually causes the transfers to go slower. I am running an rsync server using xinet.d and an rsync client. I have tried specifying the sockopts on just the client,

Re: cygwin + rsync issue under Windows 7 x64

2010-03-07 Thread Jamie Lokier
Wayne Davison wrote: I'd imagine that both ssh and rsync start using a lot of CPU because the socketpair must be indicating that it is ready for a write (or read) but the actual write() (or read()) fails to return any bytes (as long as errno is something like EAGAIN, EINTR, or

Re: rsync daemon performance

2010-03-09 Thread Jamie Lokier
Paul Slootman wrote: On Sat 06 Mar 2010, Wayne Davison wrote: On Thu, Feb 25, 2010 at 5:20 PM, Tom Dickson tdick...@finarta.net wrote: Is it possible to get a pool of waiting daemons, similar to how apache runs? No, there is no support for that in rsync at the moment. I don't

Re: O_DIRECT, avoiding system cache?

2010-03-14 Thread Jamie Lokier
Tomasz Chmielewski wrote: Is it possible (planned?) to make rsync avoid going through system cache and use direct IO? Right now, if you decide to backup your desktop system (but it's not only about desktop systems; rather more about one-time-only data transfers) to external disk, you

Re: UDT4 and rsync? support to socket.c?

2010-03-25 Thread Jamie Lokier
Jan Wagner wrote: Jamie Lokier kirjoitti: Jan Wagner wrote: Hi, has anyone of the devels considered adding UDT4 fast reliable udp transport to socket.c, as a user-selectable alternative to using default slow TCP? It could give a 4 to 10-fold throughput improvement to rsync speed over

Re: UDT4 and rsync? support to socket.c?

2010-03-25 Thread Jamie Lokier
Jamie Lokier wrote: I'll be interested if TCP cannot reach anywhere near the same performance level, because TCP is hardware assisted if you have sensible 10G hardware, and I can't imagine UDT4 needing any less RAM on an unreliable network. Having just read the presentation poster, UDT does

Re: Getting EEXIST out of make_bak_dir()

2010-04-19 Thread Jamie Lokier
Jordan Russell wrote: (followup to thread from last month) On 3/9/2010 10:09 AM, Wayne Davison wrote: On Mon, Mar 08, 2010 at 10:02:26AM -0600, Mike Bombich wrote: rsync: make_bak_dir mkdir /Volumes/Backup/_Archive_2010_March_07_22-27-43/Users/jsmith/Library/Mail/Mailboxes/

Re: rsync problem with cygwin 1.7.?

2010-04-21 Thread Jamie Lokier
Wiesner Thomas wrote: Hello. My laptop runs XP Home SP3 and my workstation XP Prof. SP3. I use Cygwin and rsync to sync my files to a Debian 4.0 server which runs rsync 2.6.9 in daemon mode. It had always worked quite will until I upgraded to Cygwin 1.7.x (the laptop runs Cygwin 1.7.4

Re: rsync 3.0.7 network errors on MS-Windows

2010-06-02 Thread Jamie Lokier
andrew.mar...@uk.bnpparibas.com wrote: I am experiencing intermittent network failures on rsync 3.0.7 built using cygwin for Windows-XP (SP2). I am using GCC v4.4.2 and the latext version of cygwin. The rsync error long indicates things like: rsync: writefd_unbuffered failed

Re: Rsync backup issues using relative paths and LVM snapshots

2010-07-16 Thread Jamie Lokier
Alex Ferrara wrote: My problem is that if I mark a directory to have a snapshot created before rsync and use the -R (relative) option, the directory structure on the destination system will be the relative path of where I mounted the snapshot (/mnt/sync-snapshot in my case). If I don't use

Re: inefficient: --checksum calculation shouldn't be done for new files

2011-07-03 Thread Jamie Lokier
Carlos Carvalho wrote: When --checksum is used they're calculated in both ends to see if the file should be transfered. This is of course not necessary if the file doesn't exist in the destination. However, the checksum is still calculated by the sender, which is often a very large overhead.