Hi Scott, On Mon, 5 Nov 2001 16:26:25 -0500 Scott Young <scottyoung at adelphia.net> wrote:
> On Monday 05 November 2001 03:42 pm, you wrote: > > On Monday 05 November 2001 08:15, toad wrote: > > > On Mon, Nov 05, 2001 at 03:40:05AM -0500, Gianni Johansson wrote: > > >... > > > > > > Could you please make sure this uses full blocks (i.e. PO2 sizes) for all > > > the chunks (as many of them as possible anyway)? > > > > It already does this. Blocks are 256k, 512K or 1Mb depending on the file > > size. The final block may be zero padded if nescessary. > > Full blocks might not always be the best way to insert data. Let's say you > have file B which is an updated version of A. B might have its data shifted > slightly, so its blocks might not align, creating unnecessary redundency if > inserted into freenet even with the same block size. Upon inserting B, what > if FProxy could use the checksums on the segments of A, then calculate the > checksums at all offsets in B, and create the splitfile index of B using the > blocks on A that match B, and just insert blocks that fill in the changes and > the missing data? This would increashttp://samba.org/~tridge/phd_thesis.pdfe > reliability of the overlapping blocks, > and reduce the total file size on freenet. The only problem with this is > that it would make some blocks smaller than others, which would make FEC more > complicated. Maybe an FEC algorithm that can work well with variable-size > blocks can be used, and maybe it could have some abilities for use with > overlapping blocks. Or for simplicity with FEC, the partial blocks could be > padded and/or combined with other partial blocks and somehow be specially > indexed in the splitfiles' indexes. Something like this has been done. Perhaps you will find this link useful: http://samba.org/~tridge/phd_thesis.pdf Its Andrew Tridgell's PhD thesis. One of the chapters is about Rsync. Rsync uses a rolling CRC and a block CRC to allow just the sort of thing you describe, but in a different context. Still, it may be interesting reading for you. Keep well, Ray _______________________________________________ Devl mailing list Devl at freenetproject.org http://lists.freenetproject.org/mailman/listinfo/devl
