Jeff Bonwick wrote:
>>> First, wrt. fletcher2 ... I don't think it is a good idea to support the
>>> algorithm if it has these undesirable characteristics. Its not the case
>>> that you need to support it for compatibility.
>>>
>
> I'm fine with that. fletcher4 is almost as fast and far more robust.
> As you surmise, it's all about computation speed vs. sha256, which is
> an order of magnitude slower.
>
>
>> The idea is that blocks with fewer than 100 references aren't as
>> critical and thus we don't need to make an extra copy. It boils down to
>> a policy decision between safety vs. space savings.
>>
>
> There are a couple of factors that lead to this particular number:
>
> - We always ditto indirect blocks, which represent up to 128
> data blocks. The rationale is that loss of an indirect block
> would imply loss of 128 data blocks, so it's worth the cost.
> The same reasoning applies to a block that's deduped and
> referenced 128 times. We're rounding to 100 to be less geeky.
>
> - Unlike all other user data, dedupditto blocks are not billed
> to any particular user (because they do not belong to any
> particular dataset). Therefore, we would like to limit the
> maximum possible space overhead to something comparable to the
> overall metadata overhead, which is about 1%. Hence 100:1.
>
> I am open to the possibility of dedupditto being off by default, with
> 100 as the minimum legal value. If it's on by default, we can claim
> better data integrity out of the box, which is cool. However, it also
> means that when your build machine goes from 99 workspaces to 100,
> the space consumption suddenly doubles. That said, it's double the
> size of *one* workspace, not double the size of 99 workspaces, so it's
> no worse than if you simply added another workspace that wasn't deduped.
> (Of course, we could "fuzz" the refcnt threshold so that there is no sharp
> spike at the 100th copy -- which would be cool, but impossible to explain.)
>
If you set it at "2", then you don't have a sudden "oh usage doubled"
moment. It means that you immediately gain a redundancy advantage. The
drawback is the space savings of the ditto doesn't reap rewards until
you have the third reference, and of course for blocks that aren't
heavily duplicated, you do wind up using double the space than you would
without it. But the dedup is *still* a significant win over traditional
storage.
All that said, I'm ok with the rationale you've presented -- I think the
different values here are almost subject to administrator preference.
The case has my +1 in any event.
- Garrett
> Jeff
>
