On Jul 31, 2012, at 10:07 AM, Nigel W wrote:
> On Tue, Jul 31, 2012 at 9:36 AM, Ray Arachelian <r...@arachelian.com> wrote:
>> On 07/31/2012 09:46 AM, opensolarisisdeadlongliveopensolaris wrote:
>>> Dedup: First of all, I don't recommend using dedup under any
>>> circumstance. Not that it's unstable or anything, just that the
>>> performance is so horrible, it's never worth while. But particularly
>>> with encrypted data, you're guaranteed to have no duplicate data
>>> anyway, so it would be a pure waste. Don't do it.
>>> _______________________________________________ zfs-discuss mailing
>>> list email@example.com
>> One thing you can do is enable dedup when you copy all your data from
>> one zpool to another, then, when you're done, disable dedup. It will no
>> longer waste a ton of memory, and your new volume will have a high dedup
>> ratio. (Obviously anything you add after you turn dedup off won't be
>> deduped.) You can keep the old pool as a backup, or wipe it or whatever
>> and later on do the same operation in the other direction.
> Once something is written deduped you will always use the memory when
> you want to read any files that were written when dedup was enabled,
> so you do not save any memory unless you do not normally access most
> of your data.
> Also don't let the system crash :D or try to delete too much from the
> deduped dataset :D (including snapshots or the dataset itself) because
> then you have to reload all (most) of the DDT in order to delete the
> files. This gets a lot of people in trouble (including me at $work
> :|) because you need to have the ram available at all times to load
> the most (>75% to grab a number out of the air) in case the server
> crashes. Otherwise you are stuck with a machine trying to verify its
> filesystem for hours. I have one test system that has 4 GB of RAM and
> 2 TB of deduped data, when it crashes (panic, powerfailure, etc) it
> would take 8-12 hours to boot up again. It now has <1TB of data and
> will boot in about 5 minutes or so.
I believe what you meant to say was "dedup with HDDs sux." If you had
used fast SSDs instead of HDDs, you will find dedup to be quite fast.
ZFS Performance and Training
zfs-discuss mailing list