Static server side listing
I've looking for a solution for this and no amount of googling has come up with anything. Is it possible to provide a static listing on a server, say every 24 hours, that a standard end-user rsync can pull and use? I have a lot of files to provide and the idea of every request dynamically providing a file list in real time is killing my server and is simply not needed. I am quite prepared to swap in (atomically) an alternate file tree every 24 hours as long as I can also provide a static file list. I know the files will not change for 24 hours and could easily handle a 10Kb to 100Kb static list being downloaded, plus the actual delta downloads, but adding 100/sec listings of 20,000 files is a killer. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Static server side listing
On 22/12/11 14:44, Kevin Korb wrote: Is it possible to provide a static listing on a server, say every 24 hours, that a standard end-user rsync can pull and use? Sounds like a job for a snapshot. If you are on Linux that would be an lvm2 snapshot. Other operating systems with basic volume management usually have an equivalent. Thanks. That sounds like a way to manage the archived files but I don't understand how that would eliminate the dynamic generation of listings that rsyncd provides to clients? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Static server side listing
On 22/12/11 14:59, Kevin Korb wrote: Is it possible to provide a static listing on a server, say every 24 hours, that a standard end-user rsync can pull and use? Sounds like a job for a snapshot. If you are on Linux that would be an lvm2 snapshot. Other operating systems with basic volume management usually have an equivalent. Thanks. That sounds like a way to manage the archived files but I don't understand how that would eliminate the dynamic generation of listings that rsyncd provides to clients? It would provide a static copy. Sure, of the archived files themselves. An alternate file tree would do the same thing and give me more flexibility to prepare the alt tree that is about to be swapped in at the next 24 hour swap over point. The rest would be up to the OS to cache in RAM. My 20k files situation could grow to 100k files so something has to index them all dynamically 100/sec. Even if the directory indexes are cached in ram that is still a lot of load on the cpus just to do something that could be done once with the results provided as a simple single static file, if it were possible. I guess my question is now, what would it take to redirect what rsyncd would normally send back to a client, as a listing, to a local server file and then tell rsyncd to use that single pre prepared file for future listing requests from clients? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: keep 2 dirs in sync
On 2011-07-01 09:40 PM, Chris Dennis wrote: I have two hosts (my portable and my desktop). I work on both hosts at different times and so I keep a few dirs sync'd between the two. I have a docs dir where I may be modifying files, adding files, renaming files and deleting files on *either* host. You may want to investigate unison, which is designed for such usage. It's written in OCaml. Or syncany -- www.syncany.org Sigh, excellent concept but it's mostly in Java which is unacceptable for me. Would anyone know of any project written in C/C++ based directly on rsync, and maybe Git? If not then would anyone be interested in starting such a project with similar goals to Syncany but only for *nix? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync sleep
On Mon, 7 Jun 2010 4:07:22 am Mag Gam wrote: Is it possible to sleep 1 second after each file is rsynced? Interesting idea but not that I know of. Ofcourse, I can put this in a for loop and do a sleep after each file is done, I was wondering if there was anything native in rsync for this type of operation. If you are concerned about giving the rest of the system some time to breathe then just nice the rsync process. nice -n 19 rsync ... etc --markc -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync sleep
On 2010-06-07, Eberhard Moenkeberg wrote: Is it possible to sleep 1 second after each file is rsynced? If you are concerned about giving the rest of the system some time to breathe then just nice the rsync process. nice -n 19 rsync ... etc This would not help, regarding i/o. Care to elaborate? It wasn't obvious whether Mag Gam was concerned about bandwidth usage or process usage and seeing that Benjamin already provided the --bwlimit hint I thought I would add a process friendly clue as well. --markc -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync sleep
On 2010-06-07, Mag Gam wrote: I am more concerned with write penalty. We use netApps, and if there is a huge write (10gb file) i would like to give the filer to recover before I can start syncing more data. Perhaps as Eberhard suggested, ionice might be useful and is part of the util-linux-ng package... -c The scheduling class. 1 for real time, 2 for best-effort, 3 for idle. Idle. A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument. ionice -c3 rsync ... etc --markc -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync via restricted ssh command=
I've read a few tutorials about how to use rsync via ssh using the command= functionality to restrict where the user can sync to. I've got this on the on the destination side in it's ~/.ssh/authorized_keys... command=rsync --server -vvnlogDtpre.iLsf --timeout=999 . bkp,no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa AAetc I've tried variations of the above but I basically get this error... % rsync -avv --delay-updates --delete-after /path/to/files/ remote:bkp opening connection using: ssh remote rsync --server -vvlogDtpre.iLsf --delete-after --delay-updates . bkp building file list ... 3 files to consider delta-transmission enabled ./ ^CKilled by signal 2. rsync error: unexplained error (code 255) at rsync.c(543) [sender=3.0.7] after waiting for a few minutes at the ./ for 2 simple 20 bytes test files to transfer I finally hit ctrl-c. On the destination side there are two rsync processes before the ctrl-c... remote% px rsync admin25537 0.0 0.0 11096 940 ?Ss 17:49 0:00 rsync --server -vvnlogDtpre.iLsf --delete-after --delay-updates . bkp admin25538 0.0 0.0 11096 380 ?S17:49 0:00 rsync --server -vvnlogDtpre.iLsf --delete-after --delay-updates . bkp Is it obvious that I am doing something wrong, or how can I determine what the unexplained error (code 255) might be, or does anyone know of a reliable tutorial outlining how to do this please? If I remove the command=rsync part from the remote ends authorized_keys then I can successfully transfer files using exactly the same source rsync command, so it's not wrong perms or keys etc, for example... % rsync -avv --delay-updates --delete-after --progress --stats /path/to/files/ remote:bkp opening connection using: ssh remote rsync --server -vvlogDtpre.iLsf --delete-after --delay-updates . bkp building file list ... 3 files to consider delta-transmission enabled ./ testfile 12 100%0.00kB/s0:00:00 (xfer#1, to-check=1/3) testfile2 13 100% 12.70kB/s0:00:00 (xfer#2, to-check=0/3) total: matches=0 hash_hits=0 false_alarms=0 data=25 rsync[26335] (sender) heap statistics: arena: 270336 (bytes from sbrk) ordblks:1 (chunks not in use) smblks: 4 hblks: 2 (chunks from mmap) hblkhd:532480 (bytes from mmap) allmem:802816 (bytes from sbrk + mmap) usmblks:0 fsmblks: 208 uordblks: 211632 (bytes used) fordblks: 58704 (bytes free) keepcost: 58496 (bytes in releasable chunk) deleting in . Number of files: 3 Number of files transferred: 2 Total file size: 25 bytes Total transferred file size: 25 bytes Literal data: 25 bytes Matched data: 0 bytes File list size: 67 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 206 Total bytes received: 53 sent 206 bytes received 53 bytes 172.67 bytes/sec total size is 25 speedup is 0.10 --markc -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync via restricted ssh command=
On 2010-06-04, Matt McCutchen wrote: I've read a few tutorials about how to use rsync via ssh using the command= functionality to restrict where the user can sync to. ... or (my preference) use a single-use rsync daemon. See: https://bugzilla.samba.org/show_bug.cgi?id=4163 I didn't know it was possible to use both --server and --daemon at the same time and now I have something that works. Thanks Matt. --markc -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
