On Jul 31, 2012, at 10:07 AM, Nigel W wrote:

> On Tue, Jul 31, 2012 at 9:36 AM, Ray Arachelian <r...@arachelian.com> wrote:
>> On 07/31/2012 09:46 AM, opensolarisisdeadlongliveopensolaris wrote:
>>> Dedup: First of all, I don't recommend using dedup under any
>>> circumstance. Not that it's unstable or anything, just that the
>>> performance is so horrible, it's never worth while. But particularly
>>> with encrypted data, you're guaranteed to have no duplicate data
>>> anyway, so it would be a pure waste. Don't do it.
>>> _______________________________________________ zfs-discuss mailing
>>> list zfs-discuss@opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>> One thing you can do is enable dedup when you copy all your data from
>> one zpool to another, then, when you're done, disable dedup.  It will no
>> longer waste a ton of memory, and your new volume will have a high dedup
>> ratio. (Obviously anything you add after you turn dedup off won't be
>> deduped.)  You can keep the old pool as a backup, or wipe it or whatever
>> and later on do the same operation in the other direction.
> Once something is written deduped you will always use the memory when
> you want to read any files that were written when dedup was enabled,
> so you do not save any memory unless you do not normally access most
> of your data.
> Also don't let the system crash :D or try to delete too much from the
> deduped dataset :D (including snapshots or the dataset itself) because
> then you have to reload all (most) of the DDT in order to delete the
> files.  This gets a lot of people in trouble (including me at $work
> :|) because you need to have the ram available at all times to load
> the most (>75% to grab a number out of the air) in case the server
> crashes. Otherwise you are stuck with a machine trying to verify its
> filesystem for hours. I have one test system that has 4 GB of RAM and
> 2 TB of deduped data, when it crashes (panic, powerfailure, etc) it
> would take 8-12 hours to boot up again.  It now has <1TB of data and
> will boot in about 5 minutes or so.

I believe what you meant to say was "dedup with HDDs sux." If you had
used fast SSDs instead of HDDs, you will find dedup to be quite fast.
 -- richard

ZFS Performance and Training

zfs-discuss mailing list

Reply via email to