A FEC code, such as the one we use for redundant splitfiles, is a code which takes k blocks of input data and produces n blocks of output data (the first k of which are normally the input data).
Onion FEC's CPU usage is roughly quadratic with k. It takes 3.4 seconds on my box (XP 2800+) to encode one segment (after setup; 4MB input, 6MB output), with the current 128/192 code. This is the same as we used in 0.5. In 0.5, for a long time all encoding was done in pure java, which is many times slower, and all encoding had to be done at the start of the splitfile insert. In the 0.7 code, we start inserting the data blocks immediately while running the splitfile encode, segment by segment, in parallel, and adding the new check blocks when they are done (starting with the last, shortest, segment). Therefore I suggest that it may be sensible to consider some other configurations. Issues: - Speed reduces by roughly a factor of 4 if either n or k is over 256, because we have to use a 16-bit code. - At present we use 50% redundancy. This is probably not necessary. PROJECTED ENCODE RATES n k segsize time rate 128 192 4MB 3.48s 1200kB/sec 170 256 5.4MB 6.14s 880kB/sec 192 256 6MB 7.83s 780kB/sec 256 384 8MB 55s ~= 1m 147kB/sec 512 768 16MB 222s ~= 3m 74kB/sec 1024 1536 32MB 890s ~= 15m 37kB/sec 2048 3072 64MB 3560s ~= 1h 18kB/sec 4096 6144 128MB 14250s ~= 4h 9.2kB/sec 8192 12288 256MB 57016s ~= 16h 4.6kB/sec 16384 24576 512MB 228ks ~= 63h 2.3kB/sec Encode rates are relative to the original data. Since we start inserting the original data blocks immediately, a codec will be acceptable if its encode rate is over our expected insert rate. Actually we have some leeway beyond that for the later segments, as we have to insert the check blocks too, but lets assume single segment of the maximum size. Note that the memory usage is 1/8th of the segment size with 4kB stripes. IMHO very few nodes will be able to insert at much over 37kB/sec. But some will, and there is CPU usage to consider too... do we want fast encodes so that people can run games etc at the same time as inserts? Realistically people are going to turn Freenet off while gaming... at least the hardcore gamers are. Even if they had loads of RAM and could nice it, they'd want to have really low bandwidth usage. Which buys us time, because it slows down the insertion of blocks, and therefore we don't need to have such a fast encode. 16 or 32MB segments seem to make sense to me. Does this make sense? I can make segment size a parameter otherwise... -- Matthew J Toseland - toad at amphibian.dyndns.org Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20051111/10bcd6c4/attachment.pgp>
