On Thu, Oct 23, 2008 at 6:49 AM, NextGen$ <[EMAIL PROTECTED]> wrote: > * Matthew Toseland <[EMAIL PROTECTED]> [2008-10-22 20:48:24]: > >> On Wednesday 22 October 2008 01:09, NextGen$ wrote: >> > * Matthew Toseland <[EMAIL PROTECTED]> [2008-10-21 20:53:51]: >> > >> > > On Tuesday 21 October 2008 16:24, [EMAIL PROTECTED] wrote: >> > > > Author: nextgens >> > > > Date: 2008-10-21 15:24:47 +0000 (Tue, 21 Oct 2008) >> > > > New Revision: 23014 >> > > > >> > > > Modified: >> > > > trunk/freenet/src/freenet/client/ArchiveManager.java >> > > > trunk/freenet/src/freenet/client/ArchiveStoreContext.java >> > > > trunk/freenet/src/freenet/client/ClientMetadata.java >> > > > trunk/freenet/src/freenet/client/HighLevelSimpleClientImpl.java >> > > > trunk/freenet/src/freenet/client/Metadata.java >> > > > trunk/freenet/src/freenet/client/async/ClientPutter.java >> > > > trunk/freenet/src/freenet/client/async/SimpleManifestPutter.java >> > > > trunk/freenet/src/freenet/client/async/SingleFileFetcher.java >> > > > trunk/freenet/src/freenet/client/async/SingleFileInserter.java >> > > > trunk/freenet/src/freenet/client/async/SplitFileInserter.java >> > > > trunk/freenet/src/freenet/clients/http/WelcomeToadlet.java >> > > > trunk/freenet/src/freenet/frost/message/FrostMessage.java >> > > > trunk/freenet/src/freenet/node/NodeARKInserter.java >> > > > trunk/freenet/src/freenet/node/TextModeClientInterface.java >> > > > trunk/freenet/src/freenet/node/fcp/ClientPut.java >> > > > trunk/freenet/src/freenet/node/fcp/DirPutFile.java >> > > > trunk/freenet/src/freenet/node/simulator/BootstrapPushPullTest.java >> > > > Log: >> > > > more work on bug #71: *** IT NEEDS TESTING! *** >> > > > It's still not backward compatible with stable but should be >> > > forward-compatible ;) >> > >> > >> > [...] see r23023 >> > >> > > Do we attempt to compress all files with bzip2 as well as gzip now? >> Shouldn't >> > > there be a max size configuration above which we don't try bzip2, perhaps >> > > unless asked to via FCP? bzip2'ing ISOs could take a really long time ... >> > >> > I don't think we need one. Big files will take long to compress but will >> take >> > long to insert too. I think it's worth spending a few more CPU cycles to >> > spare the insertion of a few blocks (plus their FEC blocks). >> >> I'm not convinced that this is acceptable from a usability point of view. >> Maybe we can provide a progress bar within the compression phase? On the new >> UI it is proposed to separate downloads which are not yet finalised (i.e. >> haven't fetched the last lot of metadata) from downloads that are... we could >> do something similar with inserts in compression. >> > > Have a look to what I have commited. From now on the compression is fully > serialized... We have one mutex, and only one compression job (just like we > do for FEC encoding in fact) which means a even higher latency.
It is feasible to insert some blocks of data while compressing? Gzip, bzip2 and lzma all support streams. We can collect the output data as we feed data to them. As soon as we get enough compressed data for FEC, we can insert them. This would be a great preformance improvement for large file on SMP. It this doable without changing the data format? >> Also, how long does it take to bzip2 a DVD ISO? Versus how long does it take >> to insert one? > > Way longer to insert the few additionnal blocks plus their FEC blocks than to > compress the ISO in any case. _______________________________________________ Devl mailing list [email protected] http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
