On 5/25/2010 6:41 AM, Mag Gam wrote:
I know rsync can do many things but I was wondering if anyone is using
it for data deduplication on a large filesystem. I have a filesystem
which is about 2TB and I want to make sure I don't have the same data
in a different place of a filesystem. Is there an algorithm for that?
While rsync is not an appropriate tool for this, I have successfully
used dupseek in the past.
http://freshmeat.net/projects/dupseek/
It is a perl script, so I expect you should be able to use it on any
platform you need. It show support for POSIX/Linux, but I expect it can
run under Windows as well if you are comfortable with Cygwin.
I'm sure there are many more tools like this. I used this one because
it was optimized for large files.
-Ben
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
