Daniel Grace wrote: > On Apr 1, 2005 6:42 PM, Christian Biere <[EMAIL PROTECTED]> wrote: > > There a definitely some issues in Gtk-Gnutella that harm its > > download performance:
> > 1. Missing support for Tigertree hashes (TTH) > > 2. Bad sources (uploading damaged data, intented or not) are not removed > > due to (1). > > 3. Gtk-Gnutella accepts arbitrary sized chunks as long as it's free > 3 is obvious. I will look for documentation on tiger tree hashes and > the download mesh when I get off of work. 2 is obvious except for how > 1 causes it. So far Gtk-Gnutella can only perform an overlap check to verify the consistency of partial downloads and the final SHA1 check. With TTH we could verify the consistency of each single chunk and therefore drop bad ones and recognize bad sources. > How would an "ideal" sized chunk be determined? Dunno. This needs to be evaluated. > Could be simply size chunks based on past performance of source, merge > any chunks that are below the minimum size (assuming they are > contiguous), then let each source have its own chunk? Sounds reasonable. If you peek at Bittorrent, the optimum size is probably somewhere between 16-256 KiB. Of course, BT uses pipelining so that smaller chunks don't cause a latency pause. > > [1] This might even allow a minor(?) DoS. It would probably be > > better to use a tree instead of a list to handle the chunks but > > memory is limited as well, of course, so there must be an upper > > limit of chunks. That probably means, as soon as we approach the > > limit we must stop creating new chunks and only fill existing > > gaps. > Am I correct in assuming that a) this tree would only need to be > calculated for _downloading files_ since a remote requester would just > tell us what range they want and b) the tree only needs to be > calculated for downloads that have moved out of the queue and started > at least _trying_ to download? > If so, I don't see memory being a huge concern except in very odd > situations. That might have been an exaggeration. I didn't really do the math, I just considered that a huge file could be splitted into thousands and millions of chunks but that should be easy to prevent - if it isn't already. > Are there that many people that start a huge number of > downloads that aren't almost instantly finished? I'd imagine that the > only people who actually _start_ a huge number are those that can > finish a huge number in short order. The slow-computer-dial-up users > only start three or so at a time, right? Well, that's another problem in Gtk-Gnutella. Queued downloads and active ones are subject to the same download slot count. So people tend to allow dozens or hundreds of downloads at a time to prevent having nothing but idle queued downloads. It's also not really possible to manually queue/preference downloads. Keep in mind that sources need to be kept fresh, so it's cannot be as blatantly simply as just pausing them. > > If you have some time to work on Gnutella, I'd rather encourage > > you to try getting TTH support to fly. > Point me at documentation if you have it sitting around. If not I'll > google after work. A lot of necessary code is already there - albeit not test, AFAIK. The next step would be wiring this code i.e., finding the right places and adapting the current download logic - which is unfortunately comparatively complex. Some documentation is here: THEX: http://www.open-content.net/specs/draft-jchapweske-thex-02.html Partial File Sharing Protocol: doc/gnutella/PFSP I'm not sure what's the best place to put the tigertree. One possibility is to use the fileinfo infrastructure and just put it into the partial file. Otherwise, the tigertree needs to put into a separate file which is somewhat uglier to handle. One further issue is that Gtk-Gnutella cannot download files without knowing the filesize in advance. There were some half-baken attempts to fix this but it's very bug at the moment. > > At the moment, I suppose doing it vice-versa would be more effective > > i.e., use the fast sources for random chunks. Another thing that > > would speed thing up is pipelining. > That seems illogical to me, so I'm sure that I'm missing something. I > see no way in which it is better for the individual user to have the > random chunks downloaded faster. I see a potential tiny improvement to > the network as a whole, but not one that I would even begin to > classify as more important than the benefit to the user. Well, yes, I was thinking about the network as a whole. I don't have hard facts on this, though. In general, what's good for the network should be good for the user, the reverse does not apply. -- Christian
pgptqeQCHoigh.pgp
Description: PGP signature
