On Wed, 17 Oct 2012, Craig Sanders wrote: > > The headers are essentially impossible to dedup as they differ in > > the final stage of delivery even if a single SMTP operation was used > > to send to multiple local users. Deduping the message body seems > > unlikely to provide a significant benefit as there usually aren't > > that many duplicates, not even when you count spam and jokes - > > the scenario I was thinking of was internal email memos sent to "all > staff", with a stupidly large word .doc or .pdf file attached. > > for an ISP mail server, de-duping isn't likely to help much (if at all). > For a small-medium business or corporate mail server, it could help a > lot. > > > > I'm assuming that ZFS is even capable of deduplicating files which > > have the duplicate part at different offsets, but I don't care enough > > about this to even look it up. > > zfs de-duping is done at block level. if a block's hash is an exact > match with another block's hash then it can be de-duped.
And guess what happens when 200 bytes into the message, Delivered-To: changes from [email protected] to [email protected]? Every subsequent byte is out by one and no subsequent block looks the same. > Editing large video files, perhaps. multiple cycles of edit & versioned > save would use not much more space than the original file + the size of > the diffs. Would multiple large video edits that insert/delete a frame here or there result in a non-integer amount of filesystem blocks to be inserted? I can't imagine things lining up on filesystem blocks sizes neatly like that. > VMs are quite often touted as a good reason for de-duping - hundreds > of almost identical zvols. I remain far from convinced that de-duping > is the best use of available RAM on a virtualisation server, or that > upgrading/adding disks wouldn't be better. We don't do it. Meh, 20GB per VM common, when the rest of the 500GB-20TB is unique on each system. I get the feeling parts of our SAN are zfs underneath, but the SAN controller is going to have a bit of trouble keeping track of cache for hundreds of TB of disk. -- Tim Connors _______________________________________________ luv-main mailing list [email protected] http://lists.luv.asn.au/listinfo/luv-main
