Re: [gentoo-user] sync'ing two computers (not related to emerge)
Issue is resolved (solution below) An unstable link is probably triggering your issue. If the link dies and comes back, unison (at least version 2.27.57, what I have here) gets confused and just sits there. What's interesting is that if unison is killed on the initiating machine, the remote process stops as well, which leads me to believe this is a bug in unison; they can talk, but they don't do anything. I've glanced at the code before despite my lack of familiarity with OCaml, as my wireless connection will often die, causing much grief, but I've never found a solution. The problem was actually related to ssh... setting those in sshd_config fixed it: ClientAliveInterval 15 TCPKeepAlive no --- The TCPKeepAlive option set to yes doesnt seem sufficient, it was suggested by someone to use ClientAliveInterval instead. (or on the client end i think you could set ServerAliveInterval) Thanks all! Simon
Re: [gentoo-user] sync'ing two computers (not related to emerge)
How long is unison taking to check for changes? I can usually reconcile changes in my home directory (approximately 45G, 125,000 files) in less than 10-15 seconds between my slow laptop drive and a remote machine. However, if you are syncing with a Windows machine, expect the sync to be much slower. Fast checking is not safe on Windows, as unison can miss changes, so it scans every file every time unless you tell it otherwise. Ok i had to try a few times and now with tcpkeepalive being turned on it doesnt disconnect. But when the scan is done, it 'reconciles' the changes (so it has received the changes from the remote end) and Begins the transfer of a dozen files, but not a bit is transfered. The sshd and unison process on the remote host are waiting/sleeping or in coma, but nothing happens. I just tried again and let it sit on the transfer to see if the processes would wake up, but it read from remote host that connection timed out. I'm not sure but this may be due to the fact i'm on an unstable link... I'll try once more with a ssh session on the side to see if that session will get disconnected or not... It would be so nice if unison could 'remember' the changes on my HD against its local archive... anyway... Thanks, Simon
Re: [gentoo-user] sync'ing two computers (not related to emerge)
http://www.linuxjournal.com/article/5976 http://kitenet.net/~joey/svnhome/ http://www.google.com/search?q=homedir+version+control Hey thanks Stroller, I'd be interested in these solutions, but i dont have enough space to copy everything in double (keep a backup). So what i do, and this is why i need unison, is that my older computer is the backup of my newer computer. Only a few files here and there are not copied over (those that are computer specific). But your idea was very good and I'll keep note of it... I'm thinking on buying a hard drive as soon as it becomes possible (unfortunately in my situation, this may not happen until a few more months) and when i get that, I'll set it up as a central repository (if i havent' found any better solution until then). Thanks!
Re: [gentoo-user] sync'ing two computers (not related to emerge)
On Sun, 2009-05-03 at 10:32 -0400, Simon wrote: How long is unison taking to check for changes? I can usually reconcile changes in my home directory (approximately 45G, 125,000 files) in less than 10-15 seconds between my slow laptop drive and a remote machine. However, if you are syncing with a Windows machine, expect the sync to be much slower. Fast checking is not safe on Windows, as unison can miss changes, so it scans every file every time unless you tell it otherwise. Ok i had to try a few times and now with tcpkeepalive being turned on it doesnt disconnect. But when the scan is done, it 'reconciles' the changes (so it has received the changes from the remote end) and Begins the transfer of a dozen files, but not a bit is transfered. The sshd and unison process on the remote host are waiting/sleeping or in coma, but nothing happens. I just tried again and let it sit on the transfer to see if the processes would wake up, but it read from remote host that connection timed out. I'm not sure but this may be due to the fact i'm on an unstable link... I'll try once more with a ssh session on the side to see if that session will get disconnected or not... It would be so nice if unison could 'remember' the changes on my HD against its local archive... anyway... Thanks, Simon An unstable link is probably triggering your issue. If the link dies and comes back, unison (at least version 2.27.57, what I have here) gets confused and just sits there. What's interesting is that if unison is killed on the initiating machine, the remote process stops as well, which leads me to believe this is a bug in unison; they can talk, but they don't do anything. I've glanced at the code before despite my lack of familiarity with OCaml, as my wireless connection will often die, causing much grief, but I've never found a solution. You could try syncing subdirectories iteratively within what you want to sync and hope that your connection does not die during the smaller sync, but that really isn't a good solution. I personally use this method to speed up synchronization if, for example, I only need my latest CS project synchronized but not any new music I may have. Unison is smart enough to use the same hash database if you use the same root directories and use the Path directive in your configuration to limit what is synchronized, so it should not re-hash your files on a later sync. As for other solutions, if you cannot get unison to work with your connection, I would suggest using a DVCS. I tried synchronizing my home directory with git before finding unison. The DVCS approach is better than SVN or CVS because it takes up less space and does not require a server. The disadvantages are the increased space used to store the backend files and the need to commit all of your files before pushing/pulling to/from the other machine (though I guess that could be an advantage, depending on your needs). It isn't optimal, but it can work, despite the git manual specifically mentioning that git was not made to do such a thing. Mercurial might work better, but I've never tried to use it. You might hit the same issues if the connection dies; I don't know. If you do find something that works similar to unison, but is more robust, I'd love to hear about it. I've tired a number of things, but I keep coming back to unison, despite its issues. Regards, Brandon Vargo
Re: [gentoo-user] sync'ing two computers (not related to emerge)
On Saturday 02 May 2009 23:33:38 Simon wrote: hi there! I've been using unison to synchronise and backup my computers. unison is based on rsync IIRC but with the difference that it 'remembers' the state of the folder that was synchronised. This way, if I delete a file on A, when sync'ing it will be deleted on B. While rsync would leave the file on B... I assume you are syncing from A to B? In that case you should look at the various --delete* options to rsync I see rsync as a very good 'cp' program, but I need something that can deal with conflicts (file changed on both hosts/directories) and can propagate changes like deletes properly. Is there a way to get this behavior with rsync (maybe i missed something)? This cannot be done with rsync. It's very much an efficient scp replacement that works with deltas. And there is a definite source and destination, what you are describing is like how source repos work The reason i'm asking is that unison has been having problems recently in sync'ing two pc... On my laptop, when i start the sync, it first checks it for changes against the 'unison snapshot'. Changes done since last sync are those that will be propagated. But it takes so long to check for changes (disk input rate is too slow i think) that once it finished, when it's time to send the changes to the remote server, the remote end has already disconnected! Surely unison has a --timeout or similar option? rsync has it, and it's such an obviously needed config option -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] sync'ing two computers (not related to emerge)
On Sat, 2009-05-02 at 17:33 -0400, Simon wrote: hi there! I've been using unison to synchronise and backup my computers. unison is based on rsync IIRC but with the difference that it 'remembers' the state of the folder that was synchronised. This way, if I delete a file on A, when sync'ing it will be deleted on B. While rsync would leave the file on B... I see rsync as a very good 'cp' program, but I need something that can deal with conflicts (file changed on both hosts/directories) and can propagate changes like deletes properly. Is there a way to get this behavior with rsync (maybe i missed something)? No. Unison implements the rsync protocol for transferring files, but it does not actually use rsync itself. You can make rsync transfer files that are not in either the destination or a reference location with the --compare-dest option, for example if doing an incremental backup, but this will not propagate the deletion of a file or directory. The delete file will come back on the next synchronization. The reason i'm asking is that unison has been having problems recently in sync'ing two pc... On my laptop, when i start the sync, it first checks it for changes against the 'unison snapshot'. Changes done since last sync are those that will be propagated. But it takes so long to check for changes (disk input rate is too slow i think) that once it finished, when it's time to send the changes to the remote server, the remote end has already disconnected! Thanks for the help/suggestions! If you are using ssh with unison, are you using the KeepAlive yes option in your ssh configuration? If not, add it, and your connection should not close from inactivity. If you are using direct sockets, unison will use a keepalive so it can timeout if the communication link is broken. How long is unison taking to check for changes? I can usually reconcile changes in my home directory (approximately 45G, 125,000 files) in less than 10-15 seconds between my slow laptop drive and a remote machine. However, if you are syncing with a Windows machine, expect the sync to be much slower. Fast checking is not safe on Windows, as unison can miss changes, so it scans every file every time unless you tell it otherwise. Regards, Brandon Vargo
Re: [gentoo-user] sync'ing two computers (not related to emerge)
If you are using ssh with unison, are you using the KeepAlive yes option in your ssh configuration? If not, add it, and your connection should not close from inactivity. If you are using direct sockets, unison will use a keepalive so it can timeout if the communication link is broken. Well i was using it, but i found out the option was commented out on my host's config... i set it to yes and trying again now... will have results in a few mins How long is unison taking to check for changes? I can usually reconcile changes in my home directory (approximately 45G, 125,000 files) in less than 10-15 seconds between my slow laptop drive and a remote machine. However, if you are syncing with a Windows machine, expect the sync to be much slower. Fast checking is not safe on Windows, as unison can miss changes, so it scans every file every time unless you tell it otherwise. Well, the problem seems to be with my /usr/portage that had not been sync'ed for a long time (i guess it has to make thorough difference checking?). So there is lots of files to check. It really takes minutes to scan for changes, i'm timing this run, if it fails i'll post how long it takes to do the whole scan until it starts transfering and finds out it got timed out. Sync'ing is done from linux to linux. The problem is really hardware related; i'm thinking on getting a new portable HD, but since i'm about to loose my job,i might want to wait! ;) Thanks!
Re: [gentoo-user] sync'ing two computers (not related to emerge)
On 2 May 2009, at 22:33, Simon wrote: ... I've been using unison to synchronise and backup my computers. unison is based on rsync IIRC but with the difference that it 'remembers' the state of the folder that was synchronised. This way, if I delete a file on A, when sync'ing it will be deleted on B. While rsync would leave the file on B... I see rsync as a very good 'cp' program, but I need something that can deal with conflicts (file changed on both hosts/directories) and can propagate changes like deletes properly. ... http://www.linuxjournal.com/article/5976 http://kitenet.net/~joey/svnhome/ http://www.google.com/search?q=homedir+version+control Stroller.
