On Wed, Dec 16, 2009 at 8:12 PM, Florent Daigniere <nextgens at freenetproject.org> wrote: > * Gregory Maxwell <gmaxwell at gmail.com> [2009-12-15 18:39:24]: > >> On Mon, Dec 14, 2009 at 11:51 AM, Florent Daigniere >> <nextgens at freenetproject.org> wrote: >> > Modern compression algorithms allow FAST decompression. We are talking >> > 10 to 20 times faster here! >> > >> > http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm >> > # Compression speed: approximately 1 MiB per second on a 2 GHz CPU >> > # Decompression speed: 10-20 MiB per second on a 2 GHz CPU >> > >> > Anyway, the assumption is and has always been that CPU cycles are cheap >> > contrary ?to network traffic. Moore's law doesn't apply to networks. >> >> It does, in fact. ? Networks are just a combination of processing, >> storage, and connectivity. Each of those is itself a combination of >> processing, storage, and connectivity. At the limit of this recusion >> the performance of all of these, except the connectivity, is driven by >> transistor density??? Moore's law. >> > > The keyword here is "except". What characterize a bottleneck is the weakest > part of the chain... > >> There is a *ton* of available bandwidth on optical fibers. For the >> communication part we have Butter's law: "Butters' Law says the amount >> of data coming out of an optical fiber is doubling every nine >> months"[1] >> > > Very informative... except you're comparing transit connectivity. Here we are > writing a p2p software, what matters is what the average user has on his local > loop. > > ADSL for most of them... or worst. If Freenet was run from servers with fiber > connectivity and high uptimes it would perfom much better. > >> It took me a day to find a graph of historical wholesale internet transit >> prices: >> >> http://www.drpeering.net/a/Peering_vs_Transit___The_Business_Case_for_Peering_files/droppedImage_1.png >> (In fact, this graph appears to be overstating the current cost for >> bulk rate transit. Advertised pricing at the gbit port level is down >> to $2/mbit/sec/month from some cut-rate providers; negotiated prices >> can be lower still) >> >> Of course, Freenet does a lot of network criss-crossing... this shifts >> the balance in favour of stronger compression but that doesn't >> magically make compression that only gives a 1% reduction a win. >> >> [1] http://www.eetimes.com/story/OEG20000926S0065 > > Like often we are arguing over not much: Freenet does heavy encryption and FEC > anyway... Adding compression to the mix is not much of an overhead compared > to the rest. > > Actually I'm surprised no one suggested getting rid of encryption > altogether; it would be waaaaaayyyy faster for sure. > > All I gathered from Ian's report is that we probably shouldn't have a > COMPRESSION stage... on the user interface. Users obviously know better but > they still don't understand what is being achieved here and press cancel. > If we weren't saying what we are doing they would just complain about speed, > which we are used to. > > It's not like this is the first time we argue over it: > Feb 11 Todd Walton ? ? ( ?24) [Tech] CHKs, Metadata, Encryption, Compression, > Hashing > Aug 12 Matthew Toselan ( ?55) [Tech] Should we try to compress all files? > => that one is worth reading, Ian already says he want the client to choose > whether to compress or not... and we already argued over recompressing video > http://archives.freenetproject.org/message/20060812.161518.1148b5c5.en.html > Jun 03 Matthew Toselan ( ?50) [freenet-dev] Memory overhead of compression > codecs >
Personally, I prefer a very light compression (e.g. gzip -2). Most of the media files are compressed anyway, using more cpu for compression does not help much and degrade user experience. For other files, they are mostly text which compress well on gzip -2. LZMA and similar algorithm take memory (>32MiB). If we want to target lower-end computer (i think we do, according to the recent discussion on Chinese user), the LZMA choice does not make sense.
