On Wed, Nov 14, 2001 at 02:38:48PM -0600, thelema wrote:
> On Wed, 14 Nov 2001, Oskar Sandberg wrote:
>
> > On Wed, Nov 14, 2001 at 03:40:00PM -0500, Gianni Johansson wrote:
> > < >
> > Is 50% really necessary? I don't know what effect the graphs have under
> > this system, but it sounds like a lot (considering 10-20% is definitely
> > enough using a real IDA).
> >
> I agree that 50% is probably way too much. There isn't a graph per-se
> in the onion-FEC system[1], but the way it encodes, any collection of
> check blocks that's the same size as the original data can be used to
> recover the original data, unlike the tornado-style systems where
> there's the possibility of a space overhead.
Then 50% is definitely too much. My simulations usually have success
rates around 90%, and definitely above 80% - ideally, the algorithm
should start with a (configurable) expected per part failure, an (also
configurable) target failure rate, and calculate the redundancy from
that and the number of parts. The math for doing so is quite trivial
(binomial distribution).
> > > Once you get past the 16M data size, the only thing you are stressing is
> > > Freenet's ability to handle a large number of inserts, which everyone
> > > knows is
> > > pretty poor.
> >
> > Well, he did show that you were forgetting to use redirects when the
> > splitfile index grows above 32 kB.
> >
> True. Maybe the way to handle this is to have the file be a
> non-redundant splitfile over all these redundant sections (since that's
> actually what's going on under the surface). None of the sections
> should exceed 32K of info because of the limitations on the onion-FEC,
> and the non-redundant level only has to have one meta-block for every
> 16MB of data, so it's probably pretty good in terms of size. Even a
> 700MB file would require 44 chunks, and we can definitely fit 44
> references into 32K.
That sounds excessive, I think a single nother layer would be enough:
KSK Redirect
|
|__ SSK Redirect || SSK DBR
| |
| |__ SSK Redirect
| |
|____________________|__ CHK Splitfile index
|
|__ CHK Split part 1
|
|__ CHK Split part 2
|
(...)
|
|__ CHK Check block 1
|
|__ CHK Check block 2
|
(...)
(Note the importance of the first step whenever a KSK is used, it has to
be done so that any bookmarks made are to the SSK - not the KSK.)
We could of course also change the 32 kB limit.
>
> Thelema
> --
> E-mail: thelema314 at bigfoot.com If you love something, set it free.
> GPG 1536g/B9C5D1F7 fpr:075A A3F7 F70B 1397 345D A67E 70AA 820B A806 F95D
--
Oskar Sandberg
oskar at freenetproject.org
_______________________________________________
Devl mailing list
Devl at freenetproject.org
http://lists.freenetproject.org/mailman/listinfo/devl
