I have a big problem with compression at the FCP level. Leave it in client/, where it belongs.
On Fri, Aug 01, 2003 at 02:43:49PM +0200, [EMAIL PROTECTED] wrote: > >> if it's the compressed hash, then > >> + the freenet protocol would know where to look for the data in the > >> freenet, grab it, detect it's precompression, extract the original data and > >> return the data just like it went into the insertion tool > >> - 3rd party tools > >> would have to emulate the node's compression step, killing portability and > >> future compatibility, if they want to calculate the CHK hash by themselves > > > >Is there any particular reason why the tool would do that on it's own? The > >node already has to do it regardless (am I correct in thinking that?), so > >doing it twice is wasteful. > > adding a hypothetical FCP parameter like "tryToCompress=false" would force the node > to not try to compress the stuff and thus tools could compress the data by > themselves using faster and/or native > routines/hardware. > IIRC FUQID uses it's *own*, faster and non-localhost-network-looped-back FEC > splitfile routines, without using FRED's FCP FEC functions. lets hope they are > compatible (unless the implementations diverge at > some time in the future) > so the here discussed insertion compression might be done by some weird insertion > tool by itself, bypassing the node's implementaion, if the tool author sees some > gain with this behaviour. > > also, 3rd party *nodes* will have to reimplement at least the decompression side to > allow the user to use freenet. > this might not be a big problem if we try to use widely spread algorithms like > standard zip, which has a library at hand in nearly every computer language. try to > use an algorithm that is available for the mayot > programming languages like c, cpp, java, delphi, python, .... i myself have not > looked for support of bzip2/... in these languages, but i'll bet the chances are > better that standard zip could be found within a lib. > > another topic is the hash of the compressed data. > even if we can agree on the used compression algorithm, the *implementation* of this > alg might differ between languages and even language versions. > this might lead to the effect, that compressing the *same data* leads to *different > archives*, which will of course produce different hash keys for the compressed data > (try to zip a file with winzip, then with winrar. > they're different. or imagine using a different compression level...!) > so the insertion of a piece of data produces a hash of the compressed data which is > effectively not predictable and not consistent. > this will lead to the problem, that you have to reinsert your data from the same > node as you always do, because if the hash of the inserted compressed data is > different between the nodes (or tools), you get no key > collision; resulting in wasted store space and wasted time. maybe the insertion tool > will remember an already inserted file (like FIW does) and does not reinsert the > data but uses the previous hash. this will lead to > different compression algorithms used for one single freesite: some files are > inserted using algorithm A, re-insertion with a node or tool that produces different > hashes will partially upload the new files with algorithm > B. the resulting mapfile for the site will then contain some files compressed with A > and some with B. so if you're unlucky and use something (node/tool) that is > incompatible with B, you only get a crippled site or data > (splitfile blocks can theoretically be inserted compressed, if they compress > well)...... if any data at all.... > > > just my 2 eurocent, maybe i'm worrying too much... > > > > > _______________________________________________ > devl mailing list > [EMAIL PROTECTED] > http://hawk.freenetproject.org:8080/cgi-bin/mailman/listinfo/devl -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so.
pgp00000.pgp
Description: PGP signature
