On Thu, Jul 12, 2012 at 9:14 AM, Edward Ned Harvey < opensolarisisdeadlongliveopensola...@nedharvey.com> wrote:
> > From: Jim Klimov [mailto:jimkli...@cos.ru] > > Sent: Thursday, July 12, 2012 8:42 AM > > To: Edward Ned Harvey > > Subject: Re: [zfs-discuss] New fast hash algorithm - is it needed? > > > > 2012-07-11 18:03, Edward Ned Harvey пишет: > > >> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > > >> boun...@opensolaris.org] On Behalf Of Sašo Kiselkov > > >> > > >> As your dedup > > >> ratio grows, so does the performance hit from dedup=verify. At, say, > > >> dedupratio=10.0x, on average, every write results in 10 reads. > > > > > > Why? > > > If you intend to write a block, and you discover it has a matching > checksum, > > > you only need to verify it matches one block. You don't need to > verify it > > > matches all the other blocks that have previously been verified to > match > > > each other. > > > > As Saso explained, if you wrote the same block 10 times > > and detected it was already deduped, then by verifying > > this detection 10 times you get about 10 extra reads. > > (In this case, Jim, you wrote me off-list and I replied on-list, but in > this case, I thought it would be ok because this message doesn't look > private or exclusive to me. I apologize if I was wrong.) > > I get the miscommunication now - > > When you write the duplicate block for the 10th time, we all understand > you're not going to go back and verify 10 blocks. (It seemed, at least to > me, that's what Saso was saying. Which is why I told him, No you don't.) > > You're saying, that when you wrote the duplicate block the 2nd time, you > verified... And then when you wrote it the 3rd time, you verified... And > the 4th time, and the 5th time... By the time you write the 10th time, > you have already verified 9 previous times, but you're still going to > verify again. > > Normally you would expect writing duplicate blocks dedup'd to be faster > than writing the non-dedup'd data, because you get to skip the actual > write. (This idealistic performance gain might be pure vapor due to need > to update metadata, but ignoring that technicality, continue > hypothetically...) When you verify, supposedly you're giving up that > hypothetical performance gain, because you have to do a read instead of a > write. So at first blush, it seems like no net-gain for performance. But > if the duplicate block gets written frequently (for example a block of all > zeros) then there's a high probability the duplicate block is already in > ARC, so you can actually skip reading from disk and just read from RAM. > > So, the "10 extra reads" will sometimes be true - if the duplicate block > doesn't already exist in ARC. And the "10 extra reads" will sometimes be > false - if the duplicate block is already in ARC. Sasso: yes, it's absolutely worth implementing a higher performing hashing algorithm. I'd suggest simply ignoring the people that aren't willing to acknowledge basic mathematics rather than lashing out. No point in feeding the trolls. The PETABYTES of data Quantum and DataDomain have out there are proof positive that complex hashes get the job done without verify, even if you don't want to acknowledge the math behind it. --Tim
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
