> >First, wrt. fletcher2 ... I don't think it is a good idea to support the
> >algorithm if it has these undesirable characteristics. Its not the case
> >that you need to support it for compatibility.
I'm fine with that. fletcher4 is almost as fast and far more robust.
As you surmise, it's all about computation speed vs. sha256, which is
an order of magnitude slower.
> The idea is that blocks with fewer than 100 references aren't as
> critical and thus we don't need to make an extra copy. It boils down to
> a policy decision between safety vs. space savings.
There are a couple of factors that lead to this particular number:
- We always ditto indirect blocks, which represent up to 128
data blocks. The rationale is that loss of an indirect block
would imply loss of 128 data blocks, so it's worth the cost.
The same reasoning applies to a block that's deduped and
referenced 128 times. We're rounding to 100 to be less geeky.
- Unlike all other user data, dedupditto blocks are not billed
to any particular user (because they do not belong to any
particular dataset). Therefore, we would like to limit the
maximum possible space overhead to something comparable to the
overall metadata overhead, which is about 1%. Hence 100:1.
I am open to the possibility of dedupditto being off by default, with
100 as the minimum legal value. If it's on by default, we can claim
better data integrity out of the box, which is cool. However, it also
means that when your build machine goes from 99 workspaces to 100,
the space consumption suddenly doubles. That said, it's double the
size of *one* workspace, not double the size of 99 workspaces, so it's
no worse than if you simply added another workspace that wasn't deduped.
(Of course, we could "fuzz" the refcnt threshold so that there is no sharp
spike at the 100th copy -- which would be cool, but impossible to explain.)
Jeff
