sync files with different attributes
I am in a situation where my destination has a different owner, file creation time and permissions. But the file content is the exactly the same. I am using --ignore-times, and -size-only. It works. However, is it possible to get rsync to change the ownership and time of the file and even owner (I am doing this as root). -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: selectively copying files
bump
On Tue, Jan 11, 2011 at 8:52 PM, Mag Gam magaw...@gmail.com wrote:
Hello All,
I am trying to sync 2 directories.
src/year/month/day/fileA.csv
src/year/month/day/fileB.csv
src/year/month/day/fileC.csv
..
src/year/month/day/fileZ.csv
I would to sync only file{B,D,T}.csv to my target directory so it
would look like this.
tgt/year/month/day/fileB.csv
tgt/year/month/day/fileD.csv
tgt/year/month/day/fileT.csv
what is the best way to do this?
TIA
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
selectively copying files
Hello All,
I am trying to sync 2 directories.
src/year/month/day/fileA.csv
src/year/month/day/fileB.csv
src/year/month/day/fileC.csv
..
src/year/month/day/fileZ.csv
I would to sync only file{B,D,T}.csv to my target directory so it
would look like this.
tgt/year/month/day/fileB.csv
tgt/year/month/day/fileD.csv
tgt/year/month/day/fileT.csv
what is the best way to do this?
TIA
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
hdfs and rsync
Currently, I sync our Unix filesystem with hdfs with provided hdfs tools. I was wondering if anyone used rsync to accomplish this. TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync sleep
Is it possible to sleep 1 second after each file is rsynced? Ofcourse, I can put this in a for loop and do a sleep after each file is done, I was wondering if there was anything native in rsync for this type of operation. TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync sleep
I am more concerned with write penalty. We use netApps, and if there is a huge write (10gb file) i would like to give the filer to recover before I can start syncing more data. bandwidth isn't the issue. On Sun, Jun 6, 2010 at 6:18 PM, Eberhard Moenkeberg emoe...@gwdg.de wrote: Hi, On Mon, 7 Jun 2010, Mark Constable wrote: On 2010-06-07, Eberhard Moenkeberg wrote: Is it possible to sleep 1 second after each file is rsynced? If you are concerned about giving the rest of the system some time to breathe then just nice the rsync process. nice -n 19 rsync ... etc This would not help, regarding i/o. Care to elaborate? It wasn't obvious whether Mag Gam was concerned about bandwidth usage or process usage and seeing that Benjamin already provided the --bwlimit hint I thought I would add a process friendly clue as well. The real I/O gets done outside the rsync process, by the kernel. SUSE had a tool named ionice earlier (probably it is gone meanwhile) which really could delay I/O at the kernel phase. Viele Gruesse Eberhard Moenkeberg (emoe...@gwdg.de, e...@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoe...@gwdg.de Tel.: +49 (0)551 201-1551 - Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: g...@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Bernhard Neumair Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 - -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
data deduplication
I know rsync can do many things but I was wondering if anyone is using it for data deduplication on a large filesystem. I have a filesystem which is about 2TB and I want to make sure I don't have the same data in a different place of a filesystem. Is there an algorithm for that? TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: data deduplication
Thanks On Tue, May 25, 2010 at 1:26 PM, Benjamin Watkins ben-l...@constant-technologies.com wrote: On 5/25/2010 6:41 AM, Mag Gam wrote: I know rsync can do many things but I was wondering if anyone is using it for data deduplication on a large filesystem. I have a filesystem which is about 2TB and I want to make sure I don't have the same data in a different place of a filesystem. Is there an algorithm for that? While rsync is not an appropriate tool for this, I have successfully used dupseek in the past. http://freshmeat.net/projects/dupseek/ It is a perl script, so I expect you should be able to use it on any platform you need. It show support for POSIX/Linux, but I expect it can run under Windows as well if you are comfortable with Cygwin. I'm sure there are many more tools like this. I used this one because it was optimized for large files. -Ben -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
syncing ext3 to fat32
I am trying to rsync some files from ext3 to fat32 pen drive. What is the correct way to do this? I am currently using, --progress -av --no-o --no-g --exclude '*iso' /ext3 /fat32 Are there any other options I should consider? TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: content of file
Basically, I am trying to read a file on the network without taking up all of my bandwidth. Typically, I rsync the file very slowly with bwlimits and do a tail -f | grep foo Instead of copying the whole file, I rather use some sort of pipe. thats all On Fri, Aug 28, 2009 at 2:23 PM, Matt McCutchenm...@mattmccutchen.net wrote: On Thu, 2009-08-27 at 22:57 -0400, Mag Gam wrote: Is it possible to stream the content of a file using rsync to stdout instead of placing it into a file? No. Consider rdiff, which lets you call each of the three steps of the delta-transfer algorithm from a script. Or if you explain your use case further, I might have more ideas. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: content of file
ofcourse, but I really don't want to copy the file to destination. I would like to direct it to a buffer or a pipe. Is that possible? On Fri, Aug 28, 2009 at 2:49 AM, Simon Powellsi...@tranmeremail.org.uk wrote: You could just cat it? On 28 Aug 2009, at 03:57, Mag Gam wrote: Is it possible to stream the content of a file using rsync to stdout instead of placing it into a file? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- This email was Anti Virus checked by Astaro Security Gateway. http://www.astaro.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
content of file
Is it possible to stream the content of a file using rsync to stdout instead of placing it into a file? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync compiling question
Hello all, Is it possible to compile rsync to be self contained. Meaning, I want to have 1 binary which will have all its libraries compiled into it and ofcourse rsync in it. I want to have a self contained version of rsync. TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: proposal to speed rsync with lots of files
Using inotify with rsync is a great idea. If one has a job that runs daily to get differences on a very large filesytem with very small files, then can do this (assuming the initial copy is already completed): inotify watch source filesystem (or tree) take down all the notices in a txt file (absolute path) use rsync with the results from the txt file and place them in the destination repository re-resync again to be 100% sure. I like this idea. On Fri, Mar 6, 2009 at 11:58 AM, Wayne Davison way...@samba.org wrote: On Thu, Mar 05, 2009 at 03:27:50PM -0800, Peter Salameh wrote: My proposal is to first send a checksum of the file list for each directory. If is found to be identical to the same checksum on the remote side then the list need not be sent for that directory! My rZync source does something like that for directories: it treats a directory-list transfer like a file transfer. That means that the receiving side sends a set of checksums to the sending side telling it what it's version of the directory looks like, and then the sender sends a normal set of delta data that lets the receiver reconstruct the sender's version of the directory (which it compares to its own). One potential drawback is having to deal with false checksum-matches (which should be rare, but would require the dir data to be resent) I hadn't optimized it for block size or (possibly) data order to make it more efficient, but it is an interesting idea for speeding up a slow connection. I'm not sure if it would really help out that much for a more modern, faster connection, because rsync sends the file-list data at the same time as it is being scanned, and sometimes the scan is the bottle-neck. The best way to optimize sending of really large numbers of files that are mostly the same is to start to leverage a file-change notification system, such as inotify. Using that, it is possible to distill a list of what files/directories need to be copied, and to just copy what is needed. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsycing very small files
it works. But takes hours to do it. Was wondering if there was a faster way On Fri, Feb 27, 2009 at 8:04 AM, Paul Slootman paul+rs...@wurtel.net wrote: On Fri 27 Feb 2009, Mag Gam wrote: I have to rsync 200k files which range in size from 5kb to 800kb. Is there an optimal way to do this using rsync? or shall I use tar for this? Just do it, there's no reason why it shouldn't work. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Include/Exclude problems
hi, I am trying to rsync a very large filesystem which is about 3TB, but naturally I want to exclude a lot of things. However, I am really struggling with excluding directories. SRC=/dasd/december/2008 #Notice there is no trailing slash TARGET=/backup/december/2008 #Notice there is no trailing slash I want to exclude /dasd/december/2008/Data /dasd/december/2008/Logs /dasd/december/2008/Catalogs #NOTICE: I added a trailing slash for $SRC but nothing for $TARGET rsync --exclude Data/** --exclude Catalogs/** --exclude Logs/** --numeric-ids --archive --delete --delete-excluded --human-readable --stats --verbose --progress $SRC/ $TARGET.0 For some reason Data Logs and Catalogs are still being synced. BTW, I am using the hardlink method to keep snapshots. So before the rsync I am cp -al from 0 to 1 and removing the oldest Any ideas? TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Include/Exclude problems
Thanks Matt for the response.
The *** was the magic bullet and it works. Thanks.
I was curious about the --link-dest argument. Will this speed up
anything? I noticed it has the same default behavior of cp -al
If its faster or more efficient I would be more than happy to implement this.
On Sun, Jan 25, 2009 at 11:03 AM, Matt McCutchen m...@mattmccutchen.net wrote:
On Sun, 2009-01-25 at 10:29 -0500, Mag Gam wrote:
I am trying to rsync a very large filesystem which is about 3TB, but
naturally I want to exclude a lot of things. However, I am really
struggling with excluding directories.
SRC=/dasd/december/2008 #Notice there is no trailing slash
TARGET=/backup/december/2008 #Notice there is no trailing slash
I want to exclude
/dasd/december/2008/Data
/dasd/december/2008/Logs
/dasd/december/2008/Catalogs
#NOTICE: I added a trailing slash for $SRC but nothing for $TARGET
rsync --exclude Data/** --exclude Catalogs/** --exclude Logs/**
--numeric-ids --archive --delete --delete-excluded --human-readable
--stats --verbose --progress $SRC/ $TARGET.0
For some reason Data Logs and Catalogs are still being synced.
Your rules should match all the contents of the Data, Logs, and Catalogs
dirs, so the only thing I can guess is that the shell is expanding the
whildcards. To avoid that, quote the rules:
--exclude 'Data/**' --exclude 'Catalogs/**' --exclude 'Logs/**'
Still, those rules may not be quite what you want. The
directories /dasd/december/2008/{Data,Logs,Catalogs} themselves are not
excluded; to exclude them, you can change the trailing ** on each rule
to *** or just remove it and rely on exclusion of the directory
short-circuiting the traversal of its contents (see the man page).
Also, the rules will match directories named Data, Logs, or Catalogs
anywhere under /dasd/december/2008 . You probably want to anchor them
with a leading slash to match only the specific directories you listed.
With both of these changes, the rules would be:
--exclude /Data/ --exclude /Catalogs/ --exclude /Logs/
BTW, I am using the hardlink method to keep snapshots. So before the
rsync I am cp -al from 0 to 1 and removing the oldest
You might consider --link-dest=$TARGET.1 instead; it will save you a
step and avoid corrupting the attributes of old backups.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
file compression on target side
Hello All, I have been using rsync to backup several filesystems by using Mike Rubel's hard link method (http://www.mikerubel.org/computers/rsync_snapshots/). The problem is, I am backing up a lot of ASCII .log, csv, and .txt files. These files are large and can range anywhere from 1GB to 30GB. I was wondering if on the target side (the backup side), if I can use some sort of compression. I am using ext3 filesystem. Any ideas? TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: file compression on target side
Thanks all. I figured this was the only solution available. Too bad I am using Linux and don't think my RAID controller is supported under Solaris. On Mon, Jan 19, 2009 at 10:41 AM, Kyle Lanclos lanc...@ucolick.org wrote: You wrote: The problem is, I am backing up a lot of ASCII .log, csv, and .txt files. These files are large and can range anywhere from 1GB to 30GB. I was wondering if on the target side (the backup side), if I can use some sort of compression. I am using ext3 filesystem. One could always switch to the ZFS filesystem; compression is but one of many good reasons to do so. I'm not sure what an equivalent Linux-based solution would be. --Kyle -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: file compression on target side
yep. ZFS on fuse is just too slow. I suppose I will wait for ZFS on Linux (pipe dream) or try to switch to Solaris 10 on x86 On Mon, Jan 19, 2009 at 1:34 PM, Ryan Malayter malay...@gmail.com wrote: On Mon, Jan 19, 2009 at 12:33 PM, Ryan Malayter malay...@gmail.com wrote: You can switch to a filesystem that supports transparent encrytpion (Reiser, ZFS, NTFS, others depending on your OS). Rsync would be completely unaware of any file-system level compression in that case. Oops. I meant transparent compression, not transparent encryption. -- RPM -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: file compression on target side
Using Redhat 4.5; I have been researching this for weeks and all signs and wisemen (such as yourself) point to the Holy Grail -- ZFS! On a side node, brtfs nor ext4 won't help us too much. Strange that ZFS is being ported to FreeBSD but a license dispute between GPL and CDDL? I guess GPL isn't all its cracked out to be... (no flame intended). Eitherway, thanks for everyone's time and replies. TIA On Mon, Jan 19, 2009 at 4:14 PM, Ryan Malayter malay...@gmail.com wrote: On Mon, Jan 19, 2009 at 2:34 PM, Mag Gam magaw...@gmail.com wrote: ZFS on fuse is just too slow. I suppose I will wait for ZFS on Linux (pipe dream) or try to switch to Solaris 10 on x86 There will never be ZFS in the Linux kernel because of license incompatibilites. The linux answer to ZFS is btrfs, which is still in development, and not much of an answer in my opinion ;-). Also, there does not appear to be any stock linux kernel filesystem that supports transparent compression read/write. SquashFS is read-only. What Linux distribution are you using? It might bundle a patch or other filesystems. I would suggest trying gzip --rsyncable. Compress the files with gzip --rsyncable at the source, and rsync should be able to find significant matches (especially for updates of log files). -- RPM -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: snapshots without NFS
Thanks for the fast response Vitorio. Do you happen to have a simple example? I been trying to look thru google but unsuccessful. On Wed, Nov 26, 2008 at 8:11 AM, Mac User FR [EMAIL PROTECTED] wrote: Mag Gam a écrit : Is it possible to implement snapshots schema without NFS or filesystem mount? I would like to implement it with ssh only. Has anyone done this before? tia Yes, it's possible using pre-xfer and post-xfer commands on the server side. Read man rsyncd.conf for more details on how to do it. Best Regards, Vitorio -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync to get filestats
Thanks Matt. I suppose I could use rsync to know how big a directory is then...right? rsync --progress -avzL -n /source /foo That should give me the total number of bytes to transfer On Wed, Oct 15, 2008 at 3:03 PM, Matt McCutchen [EMAIL PROTECTED] wrote: On Tue, 2008-10-14 at 07:31 -0400, Mag Gam wrote: Does rsync use stat()? does find use stat() when running with printf? I think the stat() is the most expensive part. Rsync and find -printf both use stat(2). Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync to get filestats
Thanks. Does rsync use stat()? does find use stat() when running with printf? I think the stat() is the most expensive part. On Tue, Oct 14, 2008 at 12:30 AM, Matt McCutchen [EMAIL PROTECTED] wrote: On Tue, 2008-10-14 at 00:28 -0400, Mag Gam wrote: Great. Thanks matt. I was using the find method, but I want to find a better way to get my directory sizes. I suppose a find -printf may work. find -printf will give you the sizes of the directories (as lists of entries) themselves. If you want subtree sizes, use du. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
using rsync to get filestats
Would it be more efficient to use rsync to get filestats instead of using the 'find' command? I would like to know how big a directory is on a filesystem, but this directory has millions of small files. I was wondering if rsync would be more efficient than find when using -n options. TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync to get filestats
Great. Thanks matt. I was using the find method, but I want to find a better way to get my directory sizes. I suppose a find -printf may work. If anyone else has suggestions please let me know. TIA On Mon, Oct 13, 2008 at 11:57 PM, Matt McCutchen [EMAIL PROTECTED] wrote: On Mon, 2008-10-13 at 22:50 -0400, Mag Gam wrote: Would it be more efficient to use rsync to get filestats instead of using the 'find' command? I would like to know how big a directory is on a filesystem, but this directory has millions of small files. I was wondering if rsync would be more efficient than find when using -n options. I assume that by filestats you mean stat(2) information for each file, not aggregate statistics. My guess is that find would be faster than rsync -n because it doesn't send the stat information to other processes like rsync does. In addition, find -printf has a configurable output format, while the rsync --list-only output format is fixed. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync many small files
At our lab we have storage with many small files. For example a directory can contain over 15,000 files and each file averages about 75k. I would like to sync this to another filesystem on a different server but I am not sure if there is a rsync tuning flag I can use for such a intensive job. I am using rsync 3.0.3. Also, I would like to use little memory as possible. Any thoughts? TIA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
