Zooko O'Whielacronx wrote: > > Since we stream the file from all servers in lock-step parallel, your > download bandwidth is equal to the bandwidth of your worst server > times the number of servers (ignoring the possibility of > multi-share-per-server).
Hm, it would be awfully cool to have a tool that visualized this sort of thing in realtime. Like, given a list of who claims to have the shares, and a stream of information about which shares are arriving with what sort of latency from whom, have a display which shows: * overall transfer rate, graphed versus time * estimated bandwidth per server, graphed versus time * which server is the bottleneck, at any given moment * which other servers might be useful Part of the Downloader overhaul I want to do (mainly to tolerate stalled/lost servers) would have code to switch to alternate servers in response to seeing slow responses. (this is super-expensive for upload, since you basically have to start all over again, but fairly cheap for download, since you only have to fetch the pieces of the merkle trees that you didn't already have). This could have a nice side-effect of switching to faster servers as they are discovered, thus moving away from that "worst server" over time. A visualization like this might make it easier to see how such a Downloader was behaving. There will be lots of questions about when it is appropriate to speculatively pull a block from an as-yet-untested server, in the hopes that you'll find them to serve faster than one of your active set, and if it's worth pulling some redundant block (taking a short-term overhead hit) to gain long-term speedups of the overall process. OTOH, since all that data is funneling through the same downstream pipe, measuring latency may be a pretty bad proxy for downstream bandwidth. It might be useful for the downloader to start fetching blocks one-server-at-a-time to estimate the available bandwidth, and/or to remember information from previous operations to build up a profile of how different servers respond. Maybe every once in a while it can pull a single block, while nothing else is going on, and throw the speed results in a long-term database, and analyze it later. Also, if a file winds up with multiple shares-per-server, the Downloader might be able to pull multiple blocks from the high-bandwidth servers, and fewer blocks from the slower ones. cheers, -Brian _______________________________________________ tahoe-dev mailing list [email protected] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
