Re: rsync as a de-duplication-only tool, using --link-dest

a hard link to file2 # rm "$file1" # ln "$file2" "$file1" echo "Hard linked: $file2 to $File1" # else # echo "Different: $file1" fi fi done Note that I only tested it a little bit which is wh

rsync as a de-duplication-only tool, using --link-dest

Recently I was thinking about --link-dest= and if it was possible to use rsync to de-duplicate two nearly-identical directory structures. Normally I would use a tool like hardlink, jdupes, or rdfind, but in this case the files are huge and numerous, so hashing them would take forever. I did a

Is this the best way to combine --times and --link-dest along with a de-duplicator?

I'm copying files using --link-dest to avoid duplication. I'm also using a de-duplicator (rmlint) to further reduce duplication. For files that are duplicates, I've rmlint set to use the timestamp of the oldest file. This ends up with starting conditions where the source of a

rsync -H with --link-dest

Hello, man rsync for -H includes: "If you specify a --link-dest directory that contains hard links, the linking of the destination files against the --link-dest files can cause some paths in the destination to become linked together due to the --link-dest associations." Can anyone e

Destination file lost with --link-dest and --backup-dir options

Hello, I am facing an issue when using --link-dest and --backup-dir together. When a source file differs from the file in the destination directory and the --backup and --backup-dir options are provided then a copy of the destination file is made in --backup-dir. However, if additionally

[Bug 12199] multiple link-dest dirs not working

https://bugzilla.samba.org/show_bug.cgi?id=12199 Wayne Davison changed: What|Removed |Added Status|NEW |RESOLVED Resolution|---

[Bug 8594] Allow --link-dest argument to be a file

|RESOLVED --- Comment #1 from Wayne Davison --- If you have just one file you can run the ln yourself. Rsync needs the file in a dir using the same name for the link-dest option to work. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #9 from Ben RUBSON --- Good news, thank you very much Wayne ! Glad to see you back :) -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 Wayne Davison changed: What|Removed |Added Resolution|--- |FIXED Status|NEW

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #7 from Ben RUBSON --- Wayne, let's merge this ? Many thanks ! -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe o

Re: --link-dest. Time to 'building file list' incrementing

Don't know how I missed this when I was trying to figure out the cheapest solution. I don't need the complexity but it seems to be worth investigating. ThanksOn 8 Jan 2019 16:15, Andrew McGlashan via rsync wrote: > > On 8/1/19 8:56 pm, John Simpson via rsync wrote: > > Any ideas anyone? > > How

Re: --link-dest. Time to 'building file list' incrementing

On 8/1/19 8:56 pm, John Simpson via rsync wrote: > Any ideas anyone? How about using snapshots and doing the rsync off those? https://www.thewindowsclub.com/vss-volume-shadow-copy-service https://blogs.technet.microsoft.com/josebda/2007/10/10/the-basics-of-the-volume-shadow-copy-service-vss/

Re: --link-dest. Time to 'building file list' incrementing

ia rsync <rsync@lists.samba.org> wrote:KevinThe link-dest parameter is a single directory (the previous day's directory), the destination is today's directory.I haven't tried deleting a backup,  there's no particular need in space terms,  at the current rate there's enough

Re: --link-dest. Time to 'building file list' incrementing

gt;> On 4 Jan 2019 09:53, John Simpson via rsync wrote: >> >> Kevin >> >> The link-dest parameter is a single directory (the previous day's >> directory), the destination is today's directory. >> >> I haven't tried deleting a backup,

Re: --link-dest. Time to 'building file list' incrementing

my experience that takes a lot > > longer than running one so if you need 15 hours to run a backup I would > > expect deleting one to take about a week. > > > > On 1/3/19 4:23 AM, John Simpson via rsync wrote: > > > > > > > > > I've been

Re: --link-dest. Time to 'building file list' incrementing

Kevin The link-dest parameter is a single directory (the previous day's directory), the destination is today's directory. I haven't tried deleting a backup, there's no particular need in space terms, at the current rate there's enough space for several years of dail

Re: --link-dest. Time to 'building file list' incrementing

It does normally take some time to analyze large trees of files. It has to call stat() on each file to get the size and timestamp. However, 15 hours seems a bit excessive even though I have never tried to do this on Windows or a NAS system. Just to be clear, is your --link-dest parameter a

--link-dest. Time to 'building file list' incrementing

I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things sta

[Bug 13656] New: --link-dest target with symbolic links from different user produces unnecessary error

https://bugzilla.samba.org/show_bug.cgi?id=13656 Bug ID: 13656 Summary: --link-dest target with symbolic links from different user produces unnecessary error Product: rsync Version: 3.1.3 Hardware: All OS

[Bug 13569] New: --link-dest may follow symlinks and failure to hard link a non-regular file is fatal

https://bugzilla.samba.org/show_bug.cgi?id=13569 Bug ID: 13569 Summary: --link-dest may follow symlinks and failure to hard link a non-regular file is fatal Product: rsync Version: 3.1.3 Hardware: All OS

BUG: --link-dest may follow symlinks and failure to hard link a non-regular file is fatal

Hi, following the instructions on https://bugzilla.samba.org/createaccount-save.html, I've applied for a bugzilla account at bugzilla-maintena...@samba.org but didn't receive a reply, so I report through this list. With --link-dest the search for a candidate to link from, follow

Re: Aw: Re: link-dest and batch-file

Von: "Kevin Korb via rsync" >> An: rsync@lists.samba.org >> Betreff: Re: link-dest and batch-file >> >> If you are using ZFS then forget --link-dest. Just rsync to the same >> zfs mount every time and do a zfs snapshot after the rsync finishes. >> Then de

Aw: Re: link-dest and batch-file

But don‘t forget —inplace, otherwise snapshots would not be efficient > Gesendet: Mittwoch, 18. Juli 2018 um 21:53 Uhr > Von: "Kevin Korb via rsync" > An: rsync@lists.samba.org > Betreff: Re: link-dest and batch-file > > If you are using ZFS then forget --link-d

Re: link-dest and batch-file

If you are using ZFS then forget --link-dest. Just rsync to the same zfs mount every time and do a zfs snapshot after the rsync finishes. Then delete old backups with a zfs destroy. On 07/18/2018 03:42 PM, Дугин Сергей via rsync wrote: > Hello. > > I need that during today'

Re: link-dest and batch-file

Hello. I need that during today's backup, the metadata about the files is saved in a file, so that tomorrow when creating a backup with the option "link-dest" instead of this option I would specify a file with metadata, then rsync will not scan the folder spec

Re: link-dest and batch-file

You have to have a script that places a "successful" file in the root of the completed rsync... And use that to figure out what to do for link-dest at the top of the script... I use something more like daily.0-daily.7 and monthly.0-monthly.3 for the folders and rotate them dail

Re: link-dest and batch-file

I don't know how the OP manages their backups. I write out a backupname.current symlink pointing to the new backup once it is completed. That is what I use as the --link-dest parameter and what I would restore from. If a backup is aborted in the middle, doesn't happen at all, or

Re: link-dest and batch-file

On Tue, Jun 26, 2018 at 12:02 PM, Дугин Сергей via rsync < rsync@lists.samba.org> wrote: > I am launching a cron bash script that does the following: > > Day 1 > /usr/bin/rsync -aH --link-dest /home/backuper/.BACKUP/009/2018-06-25 > root@192.168.1.103:/home/ /home/bac

Re: link-dest and batch-file

experience with many millions of files vs rsync --link-dest is that running the backup isn't a problem. The problem came when it was time to delete the oldest backup. An rm -rf took a lot longer than an rsync. If you haven't gotten there yet maybe you should try one and see if it is going

link-dest and batch-file

Hello. I am launching a cron bash script that does the following: Day 1 /usr/bin/rsync -aH --link-dest /home/backuper/.BACKUP/009/2018-06-25 root@192.168.1.103:/home/ /home/backuper/.BACKUP/009/2018-06-26 Day 2 /usr/bin/rsync -aH --link-dest /home/backuper/.BACKUP/009/2018-06-26

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #6 from Ben RUBSON --- Created attachment 14231 --> https://bugzilla.samba.org/attachment.cgi?id=14231&action=edit Patch using FLAG_PERHAPS_DIR Here is a working patch using the method detailed in comment #2. -- You are receiving t

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #5 from Ben RUBSON --- I reproduced the issue the same way, I meant just creating a directory in my backed-up tree with the name of a just-deleted file, this file remaining in the link-dest folder. I'm not sure the opposit

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #4 from Einhard Leichtfuß --- Did I understand correctly that you were able to reproduce this in a notably different way? I had not sufficiently examined the code to see that in all the other cases the existance of a directory is made

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #2 from Ben RUBSON --- Nice catch, I was able to easily reproduce this issue just creating a directory with the name of a just-deleted file. The path you mention Einhard seems to be the only one where no check is done to be sure a dire

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #3 from Ben RUBSON --- We could also stat() fnamecmpbuf in recv_generator(), but I think it's rather interesting to save such calls. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all

[Bug 13445] Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 --- Comment #1 from Einhard Leichtfuß --- Maybe more or less related: Bug 11866 [0], Bug 12489 [1] [0] https://bugzilla.samba.org/show_bug.cgi?id=11866 [1] https://bugzilla.samba.org/show_bug.cgi?id=12489 -- You are receiving this mail because:

[Bug 13445] New: Fuzzy searching in link-dest tries to open regular file as directory

https://bugzilla.samba.org/show_bug.cgi?id=13445 Bug ID: 13445 Summary: Fuzzy searching in link-dest tries to open regular file as directory Product: rsync Version: 3.1.3 Hardware: All OS: All

[Bug 13139] New: Formatted output turned off when --link-dest is used

https://bugzilla.samba.org/show_bug.cgi?id=13139 Bug ID: 13139 Summary: Formatted output turned off when --link-dest is used Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity

[Bug 11571] rsync-3.1.1 --link-dest arbitrary limit

https://bugzilla.samba.org/show_bug.cgi?id=11571 Wayne Davison changed: What|Removed |Added Resolution|--- |FIXED Status|NEW

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

https://bugzilla.samba.org/show_bug.cgi?id=11866 --- Comment #5 from Ben RUBSON --- Thank you very much for the merge Wayne ! -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

https://bugzilla.samba.org/show_bug.cgi?id=11866 Wayne Davison changed: What|Removed |Added Resolution|--- |FIXED Status|NEW

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

https://bugzilla.samba.org/show_bug.cgi?id=11866 --- Comment #3 from Ben RUBSON --- Hi, Could it be possible to merge this please ? It's really tiny (one character) and easily understandable :) And it avoids silent data loss ! Thank you very much ! Ben -- You are receiving this mail because

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

https://bugzilla.samba.org/show_bug.cgi?id=11866 --- Comment #2 from Ben RUBSON --- Created attachment 13342 --> https://bugzilla.samba.org/attachment.cgi?id=13342&action=edit rsync_double_fuzzy_11866 Bug found, patch attached. Wayne could you please review and commit please ? Thank you very m

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

q 0 9` do echo content_$i > $i done # rsync -a . $usr@$srv:/tmp/dst1/ # rsync -a --link-dest=../dst1 . $usr@$srv:/tmp/dst2/ # ssh $usr@$srv "ls -lin /tmp/dst1 /tmp/dst2" /tmp/dst1: total 45 660319 -rw-r--r-- 2 501 0 10 4 Jul 12:55 0 660320 -rw-r--r-- 2 501 0 10 4 Jul 12

[Bug 12835] New: Allow --link-dest to link to an optionally unexisting directory

https://bugzilla.samba.org/show_bug.cgi?id=12835 Bug ID: 12835 Summary: Allow --link-dest to link to an optionally unexisting directory Product: rsync Version: 3.1.3 Hardware: All OS: All Status

Re: link-dest + fuzzy twice = vanished

I just submitted a bug report : https://bugzilla.samba.org/show_bug.cgi?id=12489 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-

link-dest + fuzzy twice = vanished

Hello, I use --link-dest which works perfectly : rsync -a -R --link-dest="../2016-12-28/" --link-dest="../2016-12-27/" /my/backup/folder ::daemon/mycomputer/2016-12-29/ Now I would like to use --fuzzy --fuzzy, so that rsync algorithm can also work with similarly named files

Re: Use of multiple --link-dest options

On 20/09/16 12:21, John Lane wrote: > I can use --link-dest multiple times for backups so that files affected > by a backup-delete-backup-replace-backup scenario get linked. It works well. > > However, consider this scenario: backup-modify-backup-undo-backup. We have > > * back

Use of multiple --link-dest options

I can use --link-dest multiple times for backups so that files affected by a backup-delete-backup-replace-backup scenario get linked. It works well. However, consider this scenario: backup-modify-backup-undo-backup. We have * backup 1 contains file 'a' * backup 2 contains file

[Bug 12199] multiple link-dest dirs not working

https://bugzilla.samba.org/show_bug.cgi?id=12199 --- Comment #2 from Brian J. Murrell --- No triage on this at least? -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or chan

[Bug 12199] multiple link-dest dirs not working

https://bugzilla.samba.org/show_bug.cgi?id=12199 --- Comment #1 from Kevin Korb --- I have been discussing this in IRC for more than an hour now. The OP's complaint is not that multiple --link-dest paths aren't searched. The OP's complaint is that rsync finds an apparent ma

[Bug 12199] New: multiple link-dest dirs not working

https://bugzilla.samba.org/show_bug.cgi?id=12199 Bug ID: 12199 Summary: multiple link-dest dirs not working Product: rsync Version: 3.0.6 Hardware: All OS: All Status: NEW Severity: major

[Bug 12036] New: Multiple --link-dest, --copy-dest, or --compare-dest flags produce incorrect behavior

https://bugzilla.samba.org/show_bug.cgi?id=12036 Bug ID: 12036 Summary: Multiple --link-dest, --copy-dest, or --compare-dest flags produce incorrect behavior Product: rsync Version: 3.1.2 Hardware: All OS

[Bug 12036] Multiple --link-dest, --copy-dest, or --compare-dest flags produce incorrect behavior

https://bugzilla.samba.org/show_bug.cgi?id=12036 --- Comment #2 from Chris Kuehl --- Created attachment 12289 --> https://bugzilla.samba.org/attachment.cgi?id=12289&action=edit proposed patch -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all

[Bug 12036] Multiple --link-dest, --copy-dest, or --compare-dest flags produce incorrect behavior

https://bugzilla.samba.org/show_bug.cgi?id=12036 --- Comment #1 from Chris Kuehl --- Looking through the code, this sticks out to me: static int try_dests_reg(struct file_struct *file, char *fname, int ndx, char *cmpbuf, stat_x *sxp, int find_exact_for_existing,

[Bug 11866] New: rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

https://bugzilla.samba.org/show_bug.cgi?id=11866 Bug ID: 11866 Summary: rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files Product: rsync Version: 3.1.1 Hardware: All OS

Re: --link-dest not working on remote server (running daemon)

If you specify your target as offsite/backup I think you should specify your link-dest as --link-dest=offsite/backup/backup-2016-02-01-0100 ...perhaps... On 02/08/2016 11:10 PM, Sam Holton wrote: > With the following server config: > > log file = /var/log/rsyncd.log > pid >

Re: --link-dest not working on remote server (running daemon)

/rsyncd.scrt uid = 0 gid = 0 I tried the following for --link-dest and they all tried to transfer all files --link-dest=../backup-2016-02-01-0100 --link-dest=backup-2016-02-01-0100 --link-dest=/backup-2016-02-01-0100 --link-dest=/backup/backup-2016-02-01-0100 --link-dest=./backup/backup-2016-02-01

Re: --link-dest not working on remote server (running daemon)

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 try: --link-dest=../backup-2016-02-01-0100 On 02/08/2016 04:51 PM, Sam Holton wrote: > Thanks for the reply. The link-dest is different. It is Feb 1 while > the source is Feb 2. > > I tried setting path = /media/external/ for the daem

Re: --link-dest not working on remote server (running daemon)

Thanks for the reply. The link-dest is different. It is Feb 1 while the source is Feb 2. I tried setting path = /media/external/ for the daemon and using rsync -a -v -i --delete --link-dest=backup-2016-02-01-0100 --password-file=/media/external/scripts/offsite_rsync.pass /media/external/backup

Re: --link-dest not working on remote server (running daemon)

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 by misaligned I meant that your source is a directory, your link-dest is a directory (with the same name even), and your target is the root of the share. All 3 params should be directories and the link-dest param should be a different date than the

Re: --link-dest not working on remote server (running daemon)

t sure what you mean by "link-dest and your target parameters misaligned". Both servers have the same directory structure so it may be a bit confusing. I'm trying to link to the previous day's backup on the remote server. I forgot to mention that both servers are running 3.0.9

Re: --link-dest not working on remote server (running daemon)

some simple typo. I have > two servers in my setup: > > *Server 1* Doing rsync with --link-dest daily and working as > expected. I'm getting the hard links in the new daily directories. > > > *Server 2* Running rsync daemon mode with following config > > [offsi

--link-dest not working on remote server (running daemon)

Hello, I have read through the list of previous issues regarding this issue but haven't been able to resolve mine. I apologize in advance for the long text and am probably doing some simple typo. I have two servers in my setup: *Server 1* Doing rsync with --link-dest daily and worki

[Bug 11545] -A (preserve ACLs) with --link-dest=DIR fails when DIR has a directory with the same file's name

https://bugzilla.samba.org/show_bug.cgi?id=11545 --- Comment #1 from Iavor Stoev --- Hello, Is there any development on that bug? I've encountered the same issue. Regards -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies

[Bug 11571] New: rsync-3.1.1 --link-dest arbitrary limit

https://bugzilla.samba.org/show_bug.cgi?id=11571 Bug ID: 11571 Summary: rsync-3.1.1 --link-dest arbitrary limit Product: rsync Version: 3.1.1 Hardware: x86 OS: Linux Status: NEW Severity: critical

[Bug 11545] New: -A (preserve ACLs) with --link-dest=DIR fails when DIR has a directory with the same file's name

https://bugzilla.samba.org/show_bug.cgi?id=11545 Bug ID: 11545 Summary: -A (preserve ACLs) with --link-dest=DIR fails when DIR has a directory with the same file's name Product: rsync Version: 3.1.0 Hardware

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

Andrew Gideon wrote: >> btrfs has support for this: you make a backup, then create a btrfs >> snapshot of the filesystem (or directory), then the next time you make a >> new backup with rsync, use --inplace so that just changed parts of the >> file are written to the same blocks and btrfs will ta

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

On Tue, 14 Jul 2015 08:59:25 +0200, Paul Slootman wrote: > btrfs has support for this: you make a backup, then create a btrfs > snapshot of the filesystem (or directory), then the next time you make a > new backup with rsync, use --inplace so that just changed parts of the > file are written to th

Re: Fwd: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

yeah, i read somewhere that zfs DOES have separate tuning for metadata and data cache, but i need to read up on that more. as for heavy block duplication: daily backups of the whole system = alot of dupe. /kc On Thu, Jul 16, 2015 at 05:42:32PM +, Andrew Gideon said: >On Mon, 13 Jul 201

Re: Fwd: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

On Mon, 13 Jul 2015 17:38:35 -0400, Selva Nair wrote: > As with any dedup solution, performance does take a hit and its often > not worth it unless you have a lot of duplication in the data. This is so only in some volumes in our case, but it appears that zfs permits this to be enabled/disabled

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

Ken Chase wrote: > And what's performance like? I've heard lots of COW systems performance > drops through the floor when there's many snapshots. For BTRFS I'd suspect the performance penalty to be fairly small. Snapshots can be done in different ways, and the way BTRFS and (I think) ZFS do it

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

And what's performance like? I've heard lots of COW systems performance drops through the floor when there's many snapshots. /kc On Tue, Jul 14, 2015 at 08:59:25AM +0200, Paul Slootman said: >On Mon 13 Jul 2015, Andrew Gideon wrote: >> >> On the other hand, I do confess that I am sometime

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

On Mon 13 Jul 2015, Andrew Gideon wrote: > > On the other hand, I do confess that I am sometimes miffed at the waste > involved in a small change to a very large file. Rsync is smart about > moving minimal data, but it still stores an entire new copy of the file. > > What's needed is a file sy

Fwd: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

On Mon, Jul 13, 2015 at 5:19 PM, Simon Hobson wrote: > > What's needed is a file system that can do what hard links do, but at the > > file page level. I imagine that this would work using the same Copy On > > Write logic used in managing memory pages after a fork(). > > Well some (all ?) enterp

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

Andrew Gideon wrote: > However, you've made be a little > apprehensive about storebackup. I like the lack of a need for a "restore > tool". This permits all the standard UNIX tools to be applied to > whatever I might want to do over the backup, which is often *very* > convenient. Well if y

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

On Mon, 13 Jul 2015 15:40:51 +0100, Simon Hobson wrote: > The think here is that you are into "backup" tools rather than the > general purpose tool that rsync is intended to be. Yes, that is true. Rsync serves so well as a core component to backup, I can be blind about "something other than rsy

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

idea of using some file system auditing >mechanism to drive - perhaps with an --include-from or --files-from - >what rsync moves. > >Where I get stuck is that I cannot envision how I can provide rsync with >a limited list of files to move that doesn't deny the ben

Re: rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

t of files to move that doesn't deny the benefit of --link- > dest: a complete snapshot of the old file system via [hard] links into a > prior snapshot for those files that are unchanged. The think here is that you are into "backup" tools rather than the general pu

rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

kup of large file systems. These both bring me to the idea of using some file system auditing mechanism to drive - perhaps with an --include-from or --files-from - what rsync moves. Where I get stuck is that I cannot envision how I can provide rsync with a limited list of files to move that doesn

Re: Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)

> How do you handle snapshotting? or do you leave that to the block/fs > virtualization layer? I am guessing this question is directed at me. Firstly, when I used the word snapshot, I was referring to the snapshot in the LBackup context. It is outlined at the following page

Re: Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)

ful snapshot. > >At present, when the next snap shot is started, the previous incomplete snapshot(s) are not used as a link-dest source. As mentioned, this is something I have been looking at for a while. However, there are some edge cases which need to be handled carefully if you use incom

Re: Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)

has a number of checks in place to detect failures during a backup. If this happens, then that backup is not labeled as a successful snapshot. At present, when the next snap shot is started, the previous incomplete snapshot(s) are not used as a link-dest source. As mentioned, this is something

Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)

rsync folks, Henri Shustak wrote: LBackup always starts a new backup snapshot with an empty directory. I have been looking at extending --link-dest options to scan beyond just the previous successful backup to (failed backups / older backups). However, there are all kinds of edge cases which

Re: rsync --link-dest won't link even if existing file is out of date

's fixed now in 3.1 apparently (hasnt made it into debian stable yet, among other distros...). Im just pax -rwl'ing my old backups manually til they're all using the same inodes, then I can continue with mv $olddate $otday; rsync --link-dest=$yest host:/dir $today and things wil

Re: rsync --link-dest won't link even if existing file is out of date

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/14/2015 11:35 PM, Henri Shustak wrote: >> Ill take a look but I imagine I cant backup the 80 Million files >> I need to in under the 5 hours i have for nightly >> maintenance/backups. Currently it's possible by recycling >> directories... I woul

Re: rsync --link-dest won't link even if existing file is out of date

> Ill take a look but I imagine I cant backup the 80 Million files I need > to in under the 5 hours i have for nightly maintenance/backups. Currently > it's possible by recycling directories... To cover that many files in that much time you will require a high speed system. Just another thought. P

Re: rsync --link-dest won't link even if existing file is out of date

Hi Ken, You may wish to take a quick look at LBackup (disclaimer I am a developer on this project) which is a wrapper to rsync ; designed for reliable user data backups. LBackup always starts a new backup snapshot with an empty directory. I have been looking at extending --link-dest options

Re: rsync --link-dest won't link even if existing file is out of date

On Sun, Apr 5, 2015 at 10:51 PM, Ken Chase wrote: > Feature request: allow --link-dest dir to be linked to even if file exists > in target. > >From the release notes for 3.1.0: - Improved the use of alt-dest options into an existing hierarchy of files: If a match is found i

Patch for rsync --link-dest won't link even if existing file is out of date (fwd)

: 1. rsync_link_dest improvement by Bryant Hansen Normally, existing files in destination are never updated from link-dest but are transferred over the wire. This patch changes that behaviour to use link-dest instead, which is a major performance enhancement in our environment

Re: rsync --link-dest won't link even if existing file is out of date

t; you are in an environment with millions of files I highly recommend > that you move to ZFS storage and use ZFS's subvolume snapshots > instead of --link-dest. It is much more space efficient, rsync run > time efficient, and the old backups can be deleted in seconds. > Rsync doesn&

Re: rsync --link-dest won't link even if existing file is out of date

ent with millions of files I highly recommend > that you move to ZFS storage and use ZFS's subvolume snapshots > instead of --link-dest. It is much more space efficient, rsync run > time efficient, and the old backups can be deleted in seconds. > Rsync doesn't have to underst

Re: rsync --link-dest won't link even if existing file is out of date

ince you are in an environment with millions of files I highly > >recommend that you move to ZFS storage and use ZFS's subvolume > >snapshots instead of --link-dest. It is much more space efficient, > >rsync run time efficient, and the old backups can be delete

Re: rsync --link-dest won't link even if existing file is out of date

ZFS storage and use ZFS's subvolume >snapshots instead of --link-dest. It is much more space efficient, >rsync run time efficient, and the old backups can be deleted in >seconds. Rsync doesn't have to understand anything about ZFS. You >just rsync to the same directory e

Re: rsync --link-dest won't link even if existing file is out of date

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Since you are in an environment with millions of files I highly recommend that you move to ZFS storage and use ZFS's subvolume snapshots instead of --link-dest. It is much more space efficient, rsync run time efficient, and the old backups c

rsync --link-dest won't link even if existing file is out of date

Feature request: allow --link-dest dir to be linked to even if file exists in target. This statement from the man page is adhered to too strongly IMHO: "This option works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so it never looks i

Re: Link-dest thinks file is newly created, but it isn't

I suspect that I've found the problem. It appears that the permissions for these directories have been turned off at some point for these directories. This might be causing the problem with --link-dest. When it tried to stat the file on the backup it couldn't and therefore did not exist

Re: Link-dest thinks file is newly created, but it isn't

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 If it seeing the files as new then I agree that stat won't help. It might have explained some other itemize output. Since it is seeing the files as new then they must not be where it is looking. Meaning that your link-dest parameter must n

Re: Link-dest thinks file is newly created, but it isn't

with a command like the following: > > > > /usr/bin/rsync -avz *--itemize-changes *--dry-run --no-o --no-g > > --no-p --exclude-from=/cygdrive/c/Users/clint/.rsync/exclude > > --delete --delete-excluded --link-dest=../latest > > /cygdrive/c/Users/clint /cygdrive/c/Users/nancy >

Re: Link-dest thinks file is newly created, but it isn't

uple years now. Unfortunately, I've > made some changes on both ends, so it's unclear what could be the > culprit. I make extensive use of --link-dest to provide a cheap > "Time Machine"-like backup for a Windows machine. > > Source: Windows 7 running Cygwin (CYGWIN_N

  1   2   3   4   5   6   7   8   >