On Apr 1, 2005 6:42 PM, Christian Biere <[EMAIL PROTECTED]> wrote: > Daniel Grace wrote: > > the download doesn't seem very random at all.
Yes, I'm replying to myself. Further watching does reveal that at least in some cases the downloads are rather random. I'm unclear which case is more common, but I'm going to assume that it's as random as it can be, when situations permit. For some reason the first two or three files that I watched completely completely linearly, with the exceptions of chunks being skipped when a source flaked out. > Well, we can only download ranges that are available. If it tends be > heavily contiguous then [snip] > - other clients do not randomize chunks (maybe to enable streaming) This could very well be the reason. There were a small number of sources available. If every one of those didn't support randoms that could very well be the cause. > I also suspect that the download mesh isn't utilized and updated as > it should - in all Gnutella clients, not just Gtk-Gnutella. If properly > implemented it should be almost as good as the Bittorrent approach. > > There a definitely some issues in Gtk-Gnutella that harm its > download performance: > > 1. Missing support for Tigertree hashes (TTH) > 2. Bad sources (uploading damaged data, intented or not) are not removed > due to (1). > 3. Gtk-Gnutella accepts arbitrary sized chunks as long as it's free 3 is obvious. I will look for documentation on tiger tree hashes and the download mesh when I get off of work. 2 is obvious except for how 1 causes it. > I think arbitrary small chunks (unless the gap is really that small) > shouldn't be accepted as it causes a lot overhead, wasted time and > increases the amount of chunks to handle[1] and tiny gaps that have to > be resumed later. If there are a lot of tiny gaps scattered over a > file there'll be a massive HTTP protocol overhead to fill them. That makes sense, and I've seen at least one place in the code that causes this. Re-writing it to make DoS unlikely would be easy. Re-writing to be idea would take some math that I'm unsure how to do. How would an "ideal" sized chunk be determined? Could be simply size chunks based on past performance of source, merge any chunks that are below the minimum size (assuming they are contiguous), then let each source have its own chunk? > [1] This might even allow a minor(?) DoS. It would probably be > better to use a tree instead of a list to handle the chunks but > memory is limited as well, of course, so there must be an upper > limit of chunks. That probably means, as soon as we approach the > limit we must stop creating new chunks and only fill existing > gaps. Am I correct in assuming that a) this tree would only need to be calculated for _downloading files_ since a remote requester would just tell us what range they want and b) the tree only needs to be calculated for downloads that have moved out of the queue and started at least _trying_ to download? If so, I don't see memory being a huge concern except in very odd situations. Are there that many people that start a huge number of downloads that aren't almost instantly finished? I'd imagine that the only people who actually _start_ a huge number are those that can finish a huge number in short order. The slow-computer-dial-up users only start three or so at a time, right? > If you have some time to work on Gnutella, I'd rather encourage > you to try getting TTH support to fly. Point me at documentation if you have it sitting around. If not I'll google after work. > > Now, the compromise. Things would be much better for me, without > > actually hurting the download pool if the following two rules were > > followed: > > > 1) New rules only kick in when there is a gap followed by a "large" > > (yet to be defined) chunk that is finished > > 2) Everything but the fastest chunk historically ignores the chunks, > > the fastest source will go back to get the chunk > > > That way the slower sources will still grab random chunks and preserve > > the randomness of the pool. Would that be acceptable? > > At the moment, I suppose doing it vice-versa would be more effective > i.e., use the fast sources for random chunks. Another thing that > would speed thing up is pipelining. That seems illogical to me, so I'm sure that I'm missing something. I see no way in which it is better for the individual user to have the random chunks downloaded faster. I see a potential tiny improvement to the network as a whole, but not one that I would even begin to classify as more important than the benefit to the user. Are we prioritizing differently, or is there an extra factor or two that I'm just not seeing? > -- > Christian ~Daniel ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Gtk-gnutella-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/gtk-gnutella-devel
