Re: rsync or other tool
with total size of 4MB. My problem is that even with this short file list and such small sized files, system leads to OUT OF MEMORY error within 2 hours killing my application and many other applications! Is there any solution? Should I look for some other tool which is equivalent for rsync but can be used for such small optimized transfers? Perhaps a little routine at the start to ensure that the previous RSync had finished before the next one has started? I assume this is the reason for the memory errors? -- Stuart Halliday This email is the property of ECS Technology Ltd. This company is registered in Scotland with company number 212513. VAT registered GB 761 7456 12 http://www.ecs-tech.com/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync: mkstemp /20070129_1012/.status.csv.IWS933 (in test) failed: No such file or directory (2)
Hi, --- + rsync -av --timeout 120 /home/20070129_1012/status.csv --port 5873 fileserver::test/20070129_1012/status.csv building file list ... done status.csv rsync: mkstemp /20070129_1012/.status.csv.IWS933 (in test) failed: No such file or directory (2) sent 347 bytes received 38 bytes 770.00 bytes/sec total size is 228 speedup is 0.59 rsync error: some files could not be transferred (code 23) at main.c(789) --- I keep getting these problems on and off (no every time!) when I run my rsync command. I'm using rsync 2.6.4 and also tried to compile 2.6.9 on both ends but that failed to resolve my problem. Any ideas? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Question on --backup --backup-dir Switches For Incremental Backs
I current do some rsync backups with a command like so every day rsync -az -e ssh --stats --delete --exclude stuff / [EMAIL PROTECTED]:/home/user/ What I want to do is have some incremental backups in there in subdirectories. So, for example, something like this on the remote server /home/user/something.tuesday /home/user/something.friday I thought the --backup --backup-dir Switches were used to store just the files that had changed in seperate directories, am I wrong on that? I've read everything I could find, including the big rsnapshot scripts, but I'm not able to do what I want, it seems so simple but something's not right, am I wrong that subdirs should have just files that are new or have changed. I tried various things like this, but had no luck rsync -az -e ssh --stats --delete --backup --backup-dir=/home.Thursday --exclude stuff / [EMAIL PROTECTED]:/home/user/ Thanks! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync 3.0.0 CVS version
Sweet! I'll test it soon! On 1/27/07, Wayne Davison [EMAIL PROTECTED] wrote: On Sun, Jan 21, 2007 at 08:43:56AM +0200, Shai wrote: When I start the rsync, either with the rsync protocol or rsh, i found that it'll start doing the rsync and just halt after a few hundred MBs or even up to a couple GBs. I had not encountered this hang until today. The backtrace implicated a problem in the wait_for_receiver() routine, and I figured out that every now and then the io_flush() call could end up reading the last available message from the receiver, giving the read_msg_fd() call nothing to read (causing a deadlock). I checked in a fix for this and released a new nightly tar file. Thanks for your help in testing rsync! ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on --backup --backup-dir Switches For Incremental Backs
On Mon 29 Jan 2007, Blake Carver wrote: I current do some rsync backups with a command like so every day rsync -az -e ssh --stats --delete --exclude stuff / [EMAIL PROTECTED]:/home/user/ What I want to do is have some incremental backups in there in subdirectories. So, for example, something like this on the remote server /home/user/something.tuesday /home/user/something.friday I thought the --backup --backup-dir Switches were used to store just the files that had changed in seperate directories, am I wrong on that? You're right on that account. Why do you want incremental backups? Things like dirvish or rsnapshot (I'm not so familiar with rsnapshot) will create complete backups that don't take that more space than incrementals (the main difference is the space needed for the directory structure, common files are hard-linked between versions). If I then want to restore from a specific day, I don't have to gather all the incremental versions one after the other, and that's not taking into account files that have been deleted between days... I'd recommend using the --link-dest option. Start with one day, e.g. Monday, and then use that as a basis for Tuesday: rsynz -az --stats --delete --exclude stuff --link-dest=../something.monday / [EMAIL PROTECTED]:/home/user/something.tuesday/ You may want to also use -H in addition to -a to preserve hardlinks from the source. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync 3.0.0 CVS version
On Sat 27 Jan 2007, Wayne Davison wrote:
I had not encountered this hang until today. The backtrace implicated
a problem in the wait_for_receiver() routine, and I figured out that
every now and then the io_flush() call could end up reading the last
available message from the receiver, giving the read_msg_fd() call
nothing to read (causing a deadlock).
I checked in a fix for this and released a new nightly tar file.
Unfortunately the current CVS version (updated a couple of hours ago) still
hangs :(
The directory structure I reproduce this with is an image storage for an
classifieds site (comparable to Ebay, but no auction stuff).
The images are stored according to the ad number, e.g. 10473158 is
stored as .../images/10/1047/10473158_[1-9]_{thumb,detail,list,etc}.jpg
Currently in the images/10/1047/ directory there are more than 2
files. Using the old rsync, it can take up to 12 hours to traverse one
of the top 90 directories. It's an ideal candidate for the new
incremental list thing...
Just now, with various straces running, the sending process got to:
lstat(1037/1037_1.jpg, {st_mode=S_IFREG|0664, st_size=10187, ...}) = 0
lstat(1037/1037_1_detail.jpg, {st_mode=S_IFREG|0664, st_size=4082, ...})
= 0
lstat(1037/1037_1_list.jpg, {st_mode=S_IFREG|0664, st_size=919, ...}) = 0
lstat(1037/1037_1_thumb.jpg, {st_mode=S_IFREG|0664, st_size=1269, ...}) = 0
select(5, [4], [], NULL, {60, 0}) = 1 (in [4], left {60, 0})
read(4, \4\0\0kW\206\r\0\4\0\0kX\206\r\0\4\0\0kY\206\r\0\4\0\0..., 8184) = 96
select(5, [4], [], NULL, {60, 0}) = 0 (Timeout)
select(5, [4], [], NULL, {60, 0}) = 0 (Timeout)
...
and then remained waiting in select().
Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync 3.0.0 CVS version
On Mon, Jan 29, 2007 at 06:33:19PM +0100, Paul Slootman wrote: Just now, with various straces running, the sending process got to: The important process in the generator, since it controls all the work. (It is the first process on the receiving side, and forks the receiver). Attaching to the generator and getting a backtrace and the output of the *first_flist plus ...-next structs should help narrow down what is happening. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on --backup --backup-dir Switches For Incremental Backs
On Mon, Jan 29, 2007 at 10:34:39AM -0500, Blake Carver wrote: I thought the --backup --backup-dir Switches were used to store just the files that had changed in seperate directories, am I wrong on that? It stores the old files that are being updated or deleted, moving (or copying) them before they are changed in the current transfer. If you want to store the new, changed files, use one or more --compare-dest options (one pointing at an old full backup, and an extra option for any intervening incrementals). However, I think that --link-dest is a better way to go for any OS that supports the hard-linking of files. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: problem with the delete option
Hi Wayne, any idea what more i could do to know what is going on? Thanx in advance! - Original Message - From: Manuel Kissoyan [EMAIL PROTECTED] To: Wayne Davison [EMAIL PROTECTED] Cc: rsync list rsync@lists.samba.org Sent: Saturday, January 27, 2007 2:32 PM Subject: Re: problem with the delete option ok, this going more complicated In the sending side do not exists a directory called piusfont, in the receving side yes... there are no entries with this directory name, no delete_in_dir(DIRECTORY) or make_file(), what next? btw, dunno if this have something to do but im running the rsync through the backup server so the backup server is the receiving side and the other server is the sending side, so basically the backup server is running rsync to pull the data. I also did find something with another user account that it is in the backup server but not anymore in the other server: delete_in_dir(home/iccdom/public_html/administrator) This was in the logs in example but rsync havent deleted it from the backup server. Thank you! - Original Message - From: Wayne Davison [EMAIL PROTECTED] To: Manuel Kissoyan [EMAIL PROTECTED] Cc: rsync list rsync@lists.samba.org Sent: Friday, January 26, 2007 12:36 PM Subject: Re: problem with the delete option On Mon, Jan 22, 2007 at 08:18:52PM -0300, Manuel Kissoyan wrote: ok, i did add the -vv and it simply not running any delete command, the only thing close to that is lines like this: You need a third -v to see mentions of where rsync is deleting. It will say delete_in_dir(DIRECTORY) for every directory whose contents is being transferred. You should also look for the make_file() calls from the generator, which is the directory reading on the sending side that is used for deletions. If you see a message about a file being protected, that indicates that your exclude options have ensured that that particular file cannot be deleted. so basicaly rsync is not deleting the files that are not there in the sending side. If you weren't using --files-from, I would have said that you were being bitten by this bug: - Fixed a recent bug where --delete was not working when transferring from the root (/) of the filesystem with --relative enabled. With the --files-from option, I can't duplicate your problem, so it is probably a case of you expecting rsync to delete in a directory that it is not sending or your not expecting an exclude to protect a file from deletion. If not, please let us know what you discover. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Timeout, server not responding
On 1/29/07, Alexandros Papadopoulos [EMAIL PROTECTED] wrote: Wait a minute, can one use ssh at the transport and still connect to an rsync daemon module? Yes indeed! Rsync logs into the remote machine and executes a remote rsync process as in plain rsync over SSH, except it tells the remote process to run as a single-use daemon instead a server, using an rsyncd.conf on the remote machine. The daemon handles the client's request using the daemon protocol over the SSH connection and exits when the session is finished. See USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION in the man page. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cannot preserve uids and gids with AIX 5.3 w/rsync 2.6.2
On Mon, 2007-01-29 at 17:13 -0800, Wayne Davison wrote: On Mon, Jan 29, 2007 at 05:49:23PM -0600, Jeff Mundine wrote: Trying to get an AIX (5.3) to run rsync 2.6.2 as a daemon For any process to be able to affect a file's ownership, it must be running as root. The default daemon config runs each transfer process as user nobody. To change this, specify uid/gid in your rsyncd.conf: uid = 0 gid = 0 Yep, that was the one. I had suspected something like that was the case, but am a little dumb when it comes to some of the permissions and system usage. However, if I ever attempt to add a flag to preserve permissions, like: rsync -a test_file test_box::test_module these are the errors: That's very strange (and, as an aside, the test_module doesn't match your cited config file, which had a single test module). You can at least rule out that this was already fixed by trying 2.6.9 and seeing if the errors persist. yeah, type-o, replaced alot of the info to mask server names and he such.. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync 3.0.0 CVS version
On Mon, Jan 29, 2007 at 06:33:19PM +0100, Paul Slootman wrote: Unfortunately the current CVS version (updated a couple of hours ago) still hangs :( I found another potential hang scenario that could happen if the generator was having to wait for a new file list to arrive, but failed to tell the sender that the oldest file-list had been completed. The code that I just checked into CVS handles this all in a much better fashion, so I'm hoping that it should make it through your big transfer without hanging now. (The nightly tar file was updated as well.) ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Tue Jan 30 03:37:47 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv4869 Modified Files: OLDNEWS Log Message: Improved a sentence. Revisions: OLDNEWS 1.38 = 1.39 http://www.samba.org/cgi-bin/cvsweb/rsync/OLDNEWS?r1=1.38r2=1.39 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Jan 30 03:57:42 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv10872 Modified Files: proto.h Log Message: Regenerated. Revisions: proto.h 1.328 = 1.329 http://www.samba.org/cgi-bin/cvsweb/rsync/proto.h?r1=1.328r2=1.329 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Jan 30 03:57:45 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv10892 Modified Files: flist.c Log Message: Changed how send_extra_file_list() checks if we have enough future work. Revisions: flist.c 1.389 = 1.390 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c?r1=1.389r2=1.390 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Jan 30 03:57:48 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv10911 Modified Files: generator.c Log Message: Moved the code that checks for finished file-lists, for finished redo sends, and finished hard-links into check_for_finished_files(), a new function that takes the place of check_for_finished_hlinks(). This lets us signal the sender more rapidly when a file-list is done (which ensures that it sends us more work to do in a timely manner), and also avoids two potential deadlocks in the old code. Revisions: generator.c 1.325 = 1.326 http://www.samba.org/cgi-bin/cvsweb/rsync/generator.c?r1=1.325r2=1.326 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Jan 30 03:57:52 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv10929 Modified Files: io.c Log Message: - Renamed done_cnt - msgdone_cnt. - Changed increment_active_files() to call check_for_finished_files() and to avoid calling both io_flush() and read_msg_fd() in the same iteration of a loop (since the former may have called the latter). Revisions: io.c1.219 = 1.220 http://www.samba.org/cgi-bin/cvsweb/rsync/io.c?r1=1.219r2=1.220 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Jan 30 03:57:55 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv10945 Modified Files: rsync.c Log Message: The code can now set cur_flist to NULL sooner than it used to, so the flist_for_ndx() routine needs to work when cur_flist is NULL. Revisions: rsync.c 1.207 = 1.208 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.c?r1=1.207r2=1.208 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/patches
Date: Tue Jan 30 04:07:41 2007 Author: wayned Update of /data/cvs/rsync/patches In directory dp.samba.org:/tmp/cvs-serv16332 Modified Files: detect-renamed.diff Log Message: Fixed failing hunks. Revisions: detect-renamed.diff 1.20 = 1.21 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/detect-renamed.diff?r1=1.20r2=1.21 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
