Re: Transfering large files (was: Unison hangs on copy)
On Tue, Jul 26, 2011 at 03:24:11PM +0200, Claudius Hubig wrote:
Victor Munoz vmu...@macul.ciencias.uchile.cl wrote:
Both replicas are very large, and the few listed changes showed no
large files, or so I thought. But it turns out I was wrong, and a 20M
file was involved. I deleted the cache file, and reconstructed the
mirror, and I finally discovered that unison hangs when this
particular large files is involved. There are other large (and larger,
4 times larger, for instance) files in the replicas, but they are
identical, and unison doesn't complain. Only with this file, which
differs.
So I tried rsyncinc another file, 23 M in size (with --progress
option), and surprise, it stops when 11% transfer is reached. Tried
with scp, and same magic number: 11% and it stops.
What happens if you try to copy this file locally or on some other
device or force reading it with, for example, dd (dd
if=/path/to/yourfile of=/dev/null). Maybe the storage device on which
it is located has some bad blocks. You could also try to run scp
within strace to see where or what hangs.
I tried various combinations of this. I tried several different files,
all above 1M, none was fully transfered by rsync or scp :-( Looks like
an issue with keeping the connection alive. Using dd doesn't give
problems. scp within strace shows, at the end, messages like this:
write(6, )\377K\373)\377X\373#\377\373
\377n\373$\377\206\373\\377\231\373\36\377\257\373\\377\323\373...,
16384) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=6, events=POLLOUT}], 1, -1) = ? ERESTART_RESTARTBLOCK (To
be restarted)
--- SIGALRM (Alarm clock) @ 0 (0) ---
By this time the copy is stalled.
Victor
--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/20110730230655.gb18...@llacolen.ciencias.uchile.cl
Re: Transfering large files (was: Unison hangs on copy)
On Tuesday 26 July 2011 09:28:49 am Victor Munoz wrote: Fine, thanks for the speculation and the interest. You're welcome, and good luck! Randy Kramer -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107271853.41032.rhkra...@gmail.com
Re: Transfering large files (was: Unison hangs on copy)
I've reordered some of your comments for my convenience in responding. On Tuesday 26 July 2011 12:32:18 am Victor Munoz wrote: Thought you guys who helped me with my original question would like to know how this evolved. Thank you! So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. I'm somewhat surprised, because when I was thinking of large files, I was thinking of CD image size files. (I used to transfer these over a 33 kbps modem, requiring about 65 hours per image, spread out over (typically) 5 nights. Oh, the good old days ;-) I don't really have any good thoughts to offer. I'd be looking to make sure that no temporary storage areas have been filled (presumably /tmp on either machine), in fact, I'd probably do a df for all partitions on both machines and make sure all of them have plenty of space (a nice big multiple of 23 MB--I don't really know how much storage rsync, or scp need, but I'm just grasping at straws looking for possibilities. While rsync may do some thinking (i.e., pause) during a 23 MB file transfer, I don't think scp would, but there still might be something that is timing out. All from me is just random speculation, however. Good luck--I hope someone else will have some better suggestions for you. Randy Kramer So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. I'm also losing my connection to the remote machine as I type this mail, so I have some kind of issue with the connection, and I really hope it is the the reason, and not the size of the files itself. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107260725.41630.rhkra...@gmail.com
Re: Unison hangs on copy
On Tue, Jul 26, 2011 at 12:16:49AM +0300, Johannes Fichtinger wrote: Hi On Saturday 23 Jul 2011 06:07:58 Victor Munoz wrote: Today, I lost the ability to synchronize between two machines, one had I just wonder, are there any changes on this? If not, should we submit a bug report in Debian? Hi. I posted an update on this today. Now I don't think it's a problem with unison, as I detected one particular file which was giving trouble, but it seems I actually have a problem with large files (20M is enough to hang it). Which does not make me happy anyway. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726131932.ga11...@llacolen.ciencias.uchile.cl
Re: Fwd: Unison hangs on copy
On Mon, Jul 25, 2011 at 09:10:35PM -0400, Brad Alexander wrote: I think that I read somewhere that Unison is barely being developed any more (I'll look for a citation), basically only major bug/security fixes, but no new features, as I recall. I have been looking for a replacement for it. Thus far, I found persy (http://persy.digitalkultur.net/), but have not had a chance to put it through its paces. I found it in an article on Debian Administration, which also lists a couple of other alternatives: http://www.debian-administration.org/article/667/Directory/File_synchronization_across_systems Mm, it's a pity, because unison is such a nice program. But if alternatives are better, so be it. Anyway, I'll stay with unison for a while, thanks for the tip. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726132110.gb11...@llacolen.ciencias.uchile.cl
Re: Transfering large files (was: Unison hangs on copy)
Victor Munoz vmu...@macul.ciencias.uchile.cl wrote: Both replicas are very large, and the few listed changes showed no large files, or so I thought. But it turns out I was wrong, and a 20M file was involved. I deleted the cache file, and reconstructed the mirror, and I finally discovered that unison hangs when this particular large files is involved. There are other large (and larger, 4 times larger, for instance) files in the replicas, but they are identical, and unison doesn't complain. Only with this file, which differs. So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. What happens if you try to copy this file locally or on some other device or force reading it with, for example, dd (dd if=/path/to/yourfile of=/dev/null). Maybe the storage device on which it is located has some bad blocks. You could also try to run scp within strace to see where or what hangs. So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. Is only this one file the problem or maybe other files as well? What happens if you copy (not move!) that file to another location on your drive and then try to scp it over? Is the content of the file thus that you could send it via plain netcat? If so, did you try something along the lines of: destination# nc -l yourport | md5sum source# dd if=/path/to/mysterious/file | nc otherhost.example.org yourport Please note that I have not tested the commands given here, so you might want to have a look at the manpages to make sure you got them correct. Best regards, Claudius -- If only you had a personality instead of an attitude. Please use GPG: ECB0C2C7 4A4C4046 446ADF86 C08112E5 D72CDBA4 http://chubig.net/ http://nightfall.org -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726152411.23a3d...@ares.home.chubig.net
Re: Transfering large files (was: Unison hangs on copy)
On Tue, Jul 26, 2011 at 07:25:41AM -0400, Randy Kramer wrote: So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. I'm somewhat surprised, because when I was thinking of large files, I was thinking of CD image size files. (I used to transfer these over a 33 kbps modem, requiring about 65 hours per image, spread out over (typically) 5 nights. Oh, the good old days ;-) Indeed, this is very frustrating, 20M is not a huge file, but it's the size which is giving me trouble (ok, it could be less, haven't tried, but I don't want to be too obsessive :-) I don't really have any good thoughts to offer. I'd be looking to make sure that no temporary storage areas have been filled (presumably /tmp on either machine), in fact, I'd probably do a df for all partitions on both machines and make sure all of them have plenty of space (a nice big multiple of 23 MB--I don't really know how much storage rsync, or scp need, but I'm just grasping at straws looking for possibilities. While rsync may do some thinking (i.e., pause) during a 23 MB file transfer, I don't think scp would, but there still might be something that is timing out. I thought exactly the same thing, that's why I tried scp after rsync. One of the machines reports this: df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 178G 146G 24G 87% / tmpfs1007M 0 1007M 0% /lib/init/rw udev 1002M 172K 1002M 1% /dev tmpfs1007M 0 1007M 0% /dev/shm This is the remote machine I'm trying to synchronize with every night. I will check at home the other one when I can. All from me is just random speculation, however. Good luck--I hope someone else will have some better suggestions for you. Fine, thanks for the speculation and the interest. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726132849.gc11...@llacolen.ciencias.uchile.cl
Re: Transfering large files (was: Unison hangs on copy)
On Tue, Jul 26, 2011 at 03:24:11PM +0200, Claudius Hubig wrote: Victor Munoz vmu...@macul.ciencias.uchile.cl wrote: So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. What happens if you try to copy this file locally or on some other device or force reading it with, for example, dd (dd if=/path/to/yourfile of=/dev/null). Maybe the storage device on which it is located has some bad blocks. You could also try to run scp within strace to see where or what hangs. Ok, I will try this and your other suggestions when I'm home later in the day, since I don't have access to that machine now. So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. Is only this one file the problem or maybe other files as well? What happens if you copy (not move!) that file to another location on your drive and then try to scp it over? After I noticed unison had problems with this file, I took another file, about the same size, in other directory, and did the rsync and scp experiments with those files. So I don't think it is a problem with that particular file or disk sector. But I will check your other suggestions as soon as I can. Thanks, Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726133425.gd11...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
Hi On Saturday 23 Jul 2011 06:07:58 Victor Munoz wrote: Today, I lost the ability to synchronize between two machines, one had I just wonder, are there any changes on this? If not, should we submit a bug report in Debian? Johannes -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107260016.49383.newslet...@fichtinger.org
Fwd: Unison hangs on copy
Sorry, didn't reply to all... --b -- Forwarded message -- From: Brad Alexander stor...@gmail.com Date: Mon, Jul 25, 2011 at 9:10 PM Subject: Re: Unison hangs on copy To: Johannes Fichtinger newslet...@fichtinger.org I think that I read somewhere that Unison is barely being developed any more (I'll look for a citation), basically only major bug/security fixes, but no new features, as I recall. I have been looking for a replacement for it. Thus far, I found persy (http://persy.digitalkultur.net/), but have not had a chance to put it through its paces. I found it in an article on Debian Administration, which also lists a couple of other alternatives: http://www.debian-administration.org/article/667/Directory/File_synchronization_across_systems --b On Mon, Jul 25, 2011 at 5:16 PM, Johannes Fichtinger newslet...@fichtinger.org wrote: Hi On Saturday 23 Jul 2011 06:07:58 Victor Munoz wrote: Today, I lost the ability to synchronize between two machines, one had I just wonder, are there any changes on this? If not, should we submit a bug report in Debian? Johannes -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107260016.49383.newslet...@fichtinger.org
Transfering large files (was: Unison hangs on copy)
On Fri, Jul 22, 2011 at 11:07:58PM -0400, Victor Munoz wrote: Today, I lost the ability to synchronize between two machines, one had lenny until today morning, and the other was sid. I was using apt-pinning to keep both unison versions compatible. I few days ago I posted a problem with unison, which appeared after a particular update I did. Someone suggested it could be a problem with large files, and I said no, no large files involved. Ok, I should I was wrong. Both replicas are very large, and the few listed changes showed no large files, or so I thought. But it turns out I was wrong, and a 20M file was involved. I deleted the cache file, and reconstructed the mirror, and I finally discovered that unison hangs when this particular large files is involved. There are other large (and larger, 4 times larger, for instance) files in the replicas, but they are identical, and unison doesn't complain. Only with this file, which differs. So I tried rsyncinc another file, 23 M in size (with --progress option), and surprise, it stops when 11% transfer is reached. Tried with scp, and same magic number: 11% and it stops. So now I think I have a problem with transfering large files. Somehow the upgrade caused this, because almost all previous transfers involved that 20M file, but now I am unable to copy this and other large files between both machines. I'm also losing my connection to the remote machine as I type this mail, so I have some kind of issue with the connection, and I really hope it is the the reason, and not the size of the files itself. Thought you guys who helped me with my original question would like to know how this evolved. Regards, Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110726043218.ga7...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
On Fri, Jul 22, 2011 at 11:11:40PM -0600, Bob Proulx wrote: This makes me think that there is a NAT router or other device that is sometimes losing the virtual circuit between the machines. Starting a new connection works but the old connection is sometimes idled out? Usually people run into that problem with idle connections and need to set up a diddle to keep the connection alive. But you are using it so it doesn't quite match but I am mentioning it anyway. You might check your router hardware. You might run a ping in another window and seeing if you are seeing any packet loss. If you found a high packet loss between the systems then that would point to the hardware between them. Mm... not obviouss this is the case. I have a ping to the router simultaneously with the unison attempt, and while unison is waiting to copy files, ping keeps responding, no packet loss. It might be something with the router. Now, I guess ping and unison use different ports. Would it make sense? (I undertand unison uses rsync, or a version of it, to synchronize files, and rsync works, so I'd think that copying files with unison should work too, if it were a port problem...) Victor Bob -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723061145.gb7...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
Victor Munoz wrote: Mm... not obviouss this is the case. I have a ping to the router simultaneously with the unison attempt, and while unison is waiting to copy files, ping keeps responding, no packet loss. That seems good anyway. It might be something with the router. Now, I guess ping and unison use different ports. Ping is an ICMP (internet control message protocol) and rsync and unison both use TCP. So not different ports but completely different protocols. Both are part of IP (Internet Protocol). Would it make sense? (I undertand unison uses rsync, or a version of it, to synchronize files, and rsync works, so I'd think that copying files with unison should work too, if it were a port problem...) I agree that if ssh and rsync is working then unison should work too. And so with this I am exhausted of ideas for you. Bob signature.asc Description: Digital signature
Re: Unison hangs on copy
On Sat, Jul 23, 2011 at 12:21:11AM -0600, Bob Proulx wrote: Ping is an ICMP (internet control message protocol) and rsync and unison both use TCP. So not different ports but completely different protocols. Both are part of IP (Internet Protocol). Got it, not thinking clearly :-) I was actually thinking in something like an iptables rule blocking selectively either of them, and mixed that with ports. But of course, no rules currently active :-( Would it make sense? (I undertand unison uses rsync, or a version of it, to synchronize files, and rsync works, so I'd think that copying files with unison should work too, if it were a port problem...) I agree that if ssh and rsync is working then unison should work too. And so with this I am exhausted of ideas for you. Fine, thanks for the attempt. Weird problem, and it's been a long day. I'll have to make a few more systematic and controlled experiments tomorrow. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723063815.gc7...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
Hi Victor Am Samstag, 23. Juli 2011 schrieb Victor Munoz: Today, I lost the ability to synchronize between two machines, one had lenny until today morning, and the other was sid. I was using apt-pinning to keep both unison versions compatible. Yesterday night I was able to sync both machines for the last time. Then, today morning, I was not. It connects to the server, it recognizes the changes, and it offers me all the changes for approval. Then, when it has to start copying, it just waits for ever, and no progress is done. From yesterday morning on I see exactly the same problem on my machine. I am running latest SID here. On another machine with SID updated the last time one week ago running unison 2.32.52-3+b1 unison works perfectly. Johannes -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107231336.33384.newslet...@fichtinger.org
Re: Unison hangs on copy
On Sat, Jul 23, 2011 at 01:36:33PM +0200, Johannes Fichtinger wrote: Am Samstag, 23. Juli 2011 schrieb Victor Munoz: Today, I lost the ability to synchronize between two machines, one had lenny until today morning, and the other was sid. I was using apt-pinning to keep both unison versions compatible. Yesterday night I was able to sync both machines for the last time. Then, today morning, I was not. It connects to the server, it recognizes the changes, and it offers me all the changes for approval. Then, when it has to start copying, it just waits for ever, and no progress is done. From yesterday morning on I see exactly the same problem on my machine. I am running latest SID here. On another machine with SID updated the last time one week ago running unison 2.32.52-3+b1 unison works perfectly. I use unison several times a day to sync over 100GiB of data. I see it stall quite frequently. I think it's a bug in unison, possibly timing related. I see it only when syncing over the internet on a DSL connection, not over a LAN. Note that sometimes it /does/ come back--you just have to wait, possibly several tens of minutes. This can happen if you have changed a big (multi-GiB in size) file, and it wants to reconcile the changes. But it can appear to be stuck. I have on occasion had to resort to deleting the cache/index files in ~/.unison/(arb|frp)* and getting it to rebuild them from scratch on both the client and server. Note that version 2.40.61 (not in Debian) fixes quite a few of these problems, but is still not perfect. But if you have the time to build it, it's a big improvement. The build scripts are slightly buggy though--it took me a while to work out and fix! Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linux http://people.debian.org/~rleigh/ `. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `-GPG Public Key: 0x25BFB848 Please GPG sign your mail. signature.asc Description: Digital signature
Re: Unison hangs on copy
On 07/23/2011 07:36 AM, Johannes Fichtinger wrote: Hi Victor Am Samstag, 23. Juli 2011 schrieb Victor Munoz: Today, I lost the ability to synchronize between two machines, one had lenny until today morning, and the other was sid. I was using apt-pinning to keep both unison versions compatible. Yesterday night I was able to sync both machines for the last time. Then, today morning, I was not. It connects to the server, it recognizes the changes, and it offers me all the changes for approval. Then, when it has to start copying, it just waits for ever, and no progress is done. From yesterday morning on I see exactly the same problem on my machine. I am running latest SID here. On another machine with SID updated the last time one week ago running unison 2.32.52-3+b1 unison works perfectly. Johannes Hmmm. I'm going to add what will probably just be noise, but thought my situation just might be worth mentioning. I run Debian testing on four workstations. I use aptitude to upgrade all of them every day around 11:00-12:00 UTC. I also use unison-gtk to synchronize data among them. Yesterday following the updates I temporarily lost ability to synchronize two of these systems using unison-gtk 2.32.52-3+. Here's where the water gets much muddier. I rebooted the two troublesome systems. That was no help at all. In fact, I lost my ability to connect to the network with both of those systems. The tray icon (I use wicd.) and ifconfig indicated that I had a connected state with the correct IPs, but I couldn't hit my gateway with either of these two systems. I had the two of these systems which travel amongst various networks set to use ICMP filtering (including ping and pong). Guess which ones were having trouble. I changed the ICMP filtering to allow ping and pong, and everything started working -- the network connection and unison-gtk. In reviewing /var/log/aptitude I notice that the only dependency of unison-gtk that was upgraded in the past couple of days was libc6 (2.13-7 - 2.13-10), and that was during yesterday's upgrade session -- just before the trouble started. I have not bothered to re-instate ICMP filtering to its previous state of refusing ping and pong because -- a) I'm lazy, and b) its paranoid and silly and probably not good network etiquette for a network I've been given permission to visit. (I'm a consultant there.) What I'm reporting here is obviously not an exact match to the situations you (Victor and Johannes) are reporting. Victor specifically said he's not using any iptables rules, and I'm assuming Johannes would have mentioned any significant variation from that. Victor saw some more general network misbehavior, but Johannes didn't report any such issue. Of course, we're all using different versions of Debian. I'm only sending this because I found the confluence of similar behaviors on three (four?) different versions of Debian at exactly the same time period to be interesting. I thought there might be a peripheral -- if not a central -- connection here. Best regards, Gilbert -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4e2ac834.6090...@comcast.net
Re: Unison hangs on copy
On Saturday 23 July 2011 08:25:10 am Roger Leigh wrote: I use unison several times a day to sync over 100GiB of data. I see it stall quite frequently. I think it's a bug in unison, possibly timing related. I see it only when syncing over the internet on a DSL connection, not over a LAN. Don't know if this helps, but if unison uses rsync (which, iirc, someone mentioned it does), you should be aware that rsync has quite a bit of overhead (in some sense) to determine exactly which pieces of a file are different and then transmits only those. So, again, iirc, it does calculations both at the rsync server and the rsync client to get checksums (not quite the word I'm looking for, but close) of pieces of the file on both ends, then exchanges those checksums to see where differences exist, then exchanges the necessary pieces of the file, and finally reassembles the file. With very large files, it is not surprising to see the link quiet for periods of time while calculations are being done at each end. (I once read the academic thesis that the author of rsync wrote about the rsync algorithm--I don't accurately remember the details, but what I wrote above gives the general idea.) Randy Kramer -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107230921.22704.rhkra...@gmail.com
Re: Unison hangs on copy
On Sat, Jul 23, 2011 at 01:36:33PM +0200, Johannes Fichtinger wrote: Hi Victor Am Samstag, 23. Juli 2011 schrieb Victor Munoz: Today, I lost the ability to synchronize between two machines, one had lenny until today morning, and the other was sid. I was using apt-pinning to keep both unison versions compatible. Yesterday night I was able to sync both machines for the last time. Then, today morning, I was not. It connects to the server, it recognizes the changes, and it offers me all the changes for approval. Then, when it has to start copying, it just waits for ever, and no progress is done. From yesterday morning on I see exactly the same problem on my machine. I am running latest SID here. On another machine with SID updated the last time one week ago running unison 2.32.52-3+b1 unison works perfectly. At least I'm not the only one... Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723153813.ga9...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
On Sat, Jul 23, 2011 at 01:25:10PM +0100, Roger Leigh wrote: I use unison several times a day to sync over 100GiB of data. I see it stall quite frequently. I think it's a bug in unison, possibly timing related. I see it only when syncing over the internet on a DSL connection, not over a LAN. I have a lot of files being mirrored, but in this particular case there is no big file involved, and it's only a few files, about 20. Note that sometimes it /does/ come back--you just have to wait, possibly several tens of minutes. This can happen if you have changed a big (multi-GiB in size) file, and it wants to reconcile the changes. But it can appear to be stuck. I thought so, but at least one time I tried letting it go, and it finally exited with a 'lost connection to server' message. I have on occasion had to resort to deleting the cache/index files in ~/.unison/(arb|frp)* and getting it to rebuild them from scratch on both the client and server. Note that version 2.40.61 (not in Debian) fixes quite a few of these problems, but is still not perfect. But if you have the time to build it, it's a big improvement. The build scripts are slightly buggy though--it took me a while to work out and fix! Mm, I guess I will have to consider this. Deleting the cache would take a while given the number of files involved by now, but anyway, maybe I should try. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723154542.gb9...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
On Sat, Jul 23, 2011 at 09:21:22AM -0400, Randy Kramer wrote: On Saturday 23 July 2011 08:25:10 am Roger Leigh wrote: I use unison several times a day to sync over 100GiB of data. I see it stall quite frequently. I think it's a bug in unison, possibly timing related. I see it only when syncing over the internet on a DSL connection, not over a LAN. Don't know if this helps, but if unison uses rsync (which, iirc, someone mentioned it does), you should be aware that rsync has quite a bit of overhead (in some sense) to determine exactly which pieces of a file are different and then transmits only those. So, again, iirc, it does calculations both at the rsync server and the rsync client to get checksums (not quite the word I'm looking for, but close) of pieces of the file on both ends, then exchanges those checksums to see where differences exist, then exchanges the necessary pieces of the file, and finally reassembles the file. With very large files, it is not surprising to see the link quiet for periods of time while calculations are being done at each end. (I once read the academic thesis that the author of rsync wrote about the rsync algorithm--I don't accurately remember the details, but what I wrote above gives the general idea.) In this case, there are no big files or many files involved, so it doesn't seem to be the problem. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723155126.gc9...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
On Sat, Jul 23, 2011 at 09:10:12AM -0400, Gilbert Sullivan wrote: In reviewing /var/log/aptitude I notice that the only dependency of unison-gtk that was upgraded in the past couple of days was libc6 (2.13-7 - 2.13-10), and that was during yesterday's upgrade session -- just before the trouble started. I have not bothered to re-instate ICMP filtering to its previous state of refusing ping and pong because -- a) I'm lazy, and b) its paranoid and silly and probably not good network etiquette for a network I've been given permission to visit. (I'm a consultant there.) What I'm reporting here is obviously not an exact match to the situations you (Victor and Johannes) are reporting. Victor specifically said he's not using any iptables rules, and I'm assuming Johannes would have mentioned any significant variation from that. Victor saw some more general network misbehavior, but Johannes didn't report any such issue. Of course, we're all using different versions of Debian. I'm only sending this because I found the confluence of similar behaviors on three (four?) different versions of Debian at exactly the same time period to be interesting. I thought there might be a peripheral -- if not a central -- connection here. Yes, it's interesting what you mention. I've also thought that this is not specifically related to unison, so I'm trying to update the sid machine, in case magic happens and some relevant package is fixed. Unfortunately, I hadn't updated in a while, and solving a few dependencies is braking other packages, so I have to do that with care. In fact, last weekend update was complex, and it was difficult for me to find a clue by checking the logs. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723160001.gd9...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
Just noticed that my replies today to this thread went with copy to the individual posters and the list... Sorry for the duplicate emails. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723160227.ge9...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
On Saturday 23 July 2011 11:51:26 am Victor Munoz wrote: On Sat, Jul 23, 2011 at 09:21:22AM -0400, Randy Kramer wrote: With very large files, it is not surprising to see the link quiet for periods of time while calculations are being done at each end. In this case, there are no big files or many files involved, so it doesn't seem to be the problem. Sorry it didn't help. ;-) Randy Kramer -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201107231215.48670.rhkra...@gmail.com
Unison hangs on copy
Today, I lost the ability to synchronize between two machines, one had lenny until today morning, and the other was sid. I was using apt-pinning to keep both unison versions compatible. Yesterday night I was able to sync both machines for the last time. Then, today morning, I was not. It connects to the server, it recognizes the changes, and it offers me all the changes for approval. Then, when it has to start copying, it just waits for ever, and no progress is done. Last weekend I upgraded the sid machine, so I thought that could be the problem. I decided to upgrade the lenny machine to squeeze, just in case. Nothing. I have unison version 2.32.52 in both. So, basically, I don't know what to do know. I haven't found ideas in google or Debian Bugs, only problems like hanging when contacting server and such, but not after going through all the stages before actually copying. Any ideas? Thanks in advance, Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723030758.ga6...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
Victor Munoz wrote: I have unison version 2.32.52 in both. So, basically, I don't know what to do know. ... Any ideas? I am using 2.27.57 and have not seen the problem. Both Sid and Squeeze have unison 2.27.57 available. You might try downgrading both to 'unison2.27.57' and that seems like it should work just fine, since it has been working. If it does then you can even double blind the test by upgrading again to 2.32.52 and if the problem is recreated. If so then file a bug report and return to unison2.27.57 on both machines so that you can keep running. Bob signature.asc Description: Digital signature
Re: Unison hangs on copy
On Fri, Jul 22, 2011 at 09:51:07PM -0600, Bob Proulx wrote: I am using 2.27.57 and have not seen the problem. Both Sid and Squeeze have unison 2.27.57 available. You might try downgrading both to 'unison2.27.57' and that seems like it should work just fine, since it has been working. If it does then you can even double blind the test by upgrading again to 2.32.52 and if the problem is recreated. If so then file a bug report and return to unison2.27.57 on both machines so that you can keep running. Tried, but didn't work. I think it could be something not directly related to unison, but wouldn't know what or where to look for. I have a netbook with sid, and I could synchronize with the squeeze machine, when both were with 2.32.52. Also, I have noticed today that I suddenly lose the connection between both machines. (In fact, I'm writing this at home, where the sid machine is, and connecting via ssh to the squeeze machine, and I've had to restart this email twice because the terminal hangs.) So maybe it's something to do with ssh, but then again, wouldn't know what, as I can ssh, rsync, scp between both and everthing looks fine. And unison itself is fine, until it is time to copy files. Victor -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110723044643.ga7...@llacolen.ciencias.uchile.cl
Re: Unison hangs on copy
Victor Munoz wrote: ... I have noticed today that I suddenly lose the connection between both machines. (In fact, I'm writing this at home, where the sid machine is, and connecting via ssh to the squeeze machine, and I've had to restart this email twice because the terminal hangs.) So maybe it's something to do with ssh, but then again, wouldn't know what, as I can ssh, rsync, scp between both and everthing looks fine. And unison itself is fine, until it is time to copy files. This makes me think that there is a NAT router or other device that is sometimes losing the virtual circuit between the machines. Starting a new connection works but the old connection is sometimes idled out? Usually people run into that problem with idle connections and need to set up a diddle to keep the connection alive. But you are using it so it doesn't quite match but I am mentioning it anyway. You might check your router hardware. You might run a ping in another window and seeing if you are seeing any packet loss. If you found a high packet loss between the systems then that would point to the hardware between them. Bob signature.asc Description: Digital signature
