On Friday 01 Aug 2003 13:43, [EMAIL PROTECTED] wrote: > >Is there any particular reason why the tool would do that on it's own? The > >node already has to do it regardless (am I correct in thinking that?), so > >doing it twice is wasteful. > > adding a hypothetical FCP parameter like "tryToCompress=false" would force > the node to not try to compress the stuff and thus tools could compress the > data by themselves using faster and/or native routines/hardware.
Hmm... I am not sure that allowing such deviation from the norm would be a good idea at the moment. Things like this should be left in the black-box that is the node. IMHO, authoring tools should not be trying to outsmart the node. If they work, then fine, I guess that makes it OK, but I don't think this sort of approach should be entertained with support at the current stage of development. > IIRC FUQID uses it's *own*, faster and non-localhost-network-looped-back > FEC splitfile routines, without using FRED's FCP FEC functions. lets hope > they are compatible (unless the implementations diverge at some time in the > future) Exactly the point I was about to make. There has to be a standard way for handling such things. I believe that such things are best left to the node. > so the here discussed insertion compression might be done by some weird > insertion tool by itself, bypassing the node's implementaion, if the tool > author sees some gain with this behaviour. The author of such a tool may well be better off applying the improvement to the node code, rather than applying it in a non-standard way externally and then trying to hack past the standard protocol. > also, 3rd party *nodes* will have to reimplement at least the decompression > side to allow the user to use freenet. this might not be a big problem if > we try to use widely spread algorithms like standard zip, which has a > library at hand in nearly every computer language. try to use an algorithm > that is available for the mayot programming languages like c, cpp, java, > delphi, python, .... i myself have not looked for support of bzip2/... in > these languages, but i'll bet the chances are better that standard zip > could be found within a lib. Zlib has been around for years, and has been ported to just about every platform under the sun. Bzip2 has't been around for as long, but at the very least Java and C libraries exist, which should be enough for most things. > another topic is the hash of the compressed data. > even if we can agree on the used compression algorithm, the > *implementation* of this alg might differ between languages and even > language versions. this might lead to the effect, that compressing the > *same data* leads to *different archives*, which will of course produce > different hash keys for the compressed data (try to zip a file with winzip, > then with winrar. they're different. or imagine using a different > compression level...!) so the insertion of a piece of data produces a hash > of the compressed data which is effectively not predictable and not > consistent. this will lead to the problem, that you have to reinsert your > data from the same node as you always do, because if the hash of the > inserted compressed data is different between the nodes (or tools), you get > no key collision; resulting in wasted store space and wasted time. Hmm... That usually implies a bug in the algorithm somewhere. Applying the same algorithm to the same data should yield the same output, regardless of language or implementation. Everything else should be considered a bug. > maybe > the insertion tool will remember an already inserted file (like FIW does) > and does not reinsert the data but uses the previous hash. this will lead > to different compression algorithms used for one single freesite: some > files are inserted using algorithm A, re-insertion with a node or tool that > produces different hashes will partially upload the new files with > algorithm B. the resulting mapfile for the site will then contain some > files compressed with A and some with B. so if you're unlucky and use > something (node/tool) that is incompatible with B, you only get a crippled > site or data (splitfile blocks can theoretically be inserted compressed, if > they compress well)...... if any data at all.... See above. Under no sane protocol should algolrithm(data) yield variable results when compression is concerned. Remember that the node software is the only thing guaranteed to be the same on all nodes. This means that all tools MUST comply with the standard dictated by the nodes. Otherwise, compatibility issues happen. > just my 2 eurocent, maybe i'm worrying too much... I think you are worrying too much. :-) Gordan _______________________________________________ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org:8080/cgi-bin/mailman/listinfo/devl
