On Wed, 2003-03-05 at 23:43, Frank v Waveren wrote: > On Wed, Mar 05, 2003 at 08:39:37PM -0800, Ian Clarke wrote: > > Yes they should, good observation. We do, however, need to be careful > > about memory usage (and, to a lesser extent, disk usage) when sewing the > > segments back together (it would be tempting to load them all into RAM > > at once and stick-em together, but that would clearly be a bad thing). Actually, my processor (Athlon XP 1800) seems to be the major bottleneck when decoding FEC blocks, so loading into RAM wouldn't be that much of a performance boost. Plus, I don't think many JVMs would like to store and decode 128 1 MB segments in memory.
> Why? There's no magic involved in the joining of segments, so just > store them all to tempfiles and then read through them sequentially? If the segments are all stored in temp files, then sewn together, then twice as much free space as the file itself (unless the temp files are deleted as they are "sewn" into the file, which seems to not be happening). This is actually three times, if you include the space the blocks take up in the DataStore, or four, if your web browser makes a copy on a different partition first. This doesn't even include the final copy of the decoded file. I don't want to have to have 4 GB of free space just to decode 1 GB of data. I would much rather give the extra 3 GB to my DataStore. I was thinking it would be better to do what other programs do for partial downloads: .dat files. An in-place reconstruction of the file could take place after enough data for each segment is downloaded. If the user can tell Freenet where they want to store the file after it isfile downloaded, the dat file could be constructed in the same place so that the file doesn't need to be copied across partitions in the end. The DAT file would also be able to store data that is deleted from the node, so that the data store could actually be smaller than the size of the SplitFile being downloaded. Resuming downloads would also be quicker because FEC wouldn't have to be run on already-decoded segments again. The ultimate way of doing this would be to allow a user to transparently mount the data store and access a SplitFile directly. This way, a user wouldn't have to give up extra space on their drive (which could be used by the data store itself, anyway), AND the node would have a cached copy of the blocks to use. This has obvious problems though. _______________________________________________ devl mailing list devl at freenetproject.org http://hawk.freenetproject.org:8080/cgi-bin/mailman/listinfo/devl
