Spinning this the opposite way: Assume you administer a project and the load from clients gets extreme.
Would you like to have a way to tell the clients to slow down, or would you rather just batten down the hatches and weather the storm? Mark Pottorff wrote: > In the first 10 minutes of being back up and running, I would say most > projects could respond to every attached client, so long as the protocol and > scoring system are sufficiantly simple. > > 10 minutes = 600 seconds, say 10 interactions per second? 6000 hosts. Which > for many tiny budget projects is all they will have. For a larger project > with 60,000 hosts, it is still reasonable to expect you would only have hits > from 10% of them in the first 10 minutes. So, you've still kept up with the > workload. ...and if not, that's why the client does retries. > > > Running Microsoft's "System Idle Process" will never help cure cancer, > AIDS nor Alzheimer's. But running rose...@home just might! > http://boinc.bakerlab.org/rosetta/ > > > --- On Sat, 7/11/09, Lynn W. Taylor <[email protected]> wrote: > >> From: Lynn W. Taylor <[email protected]> >> Subject: Re: [boinc_dev] Optimizing uploads..... >> To: [email protected] >> Cc: "BOINC dev" <[email protected]> >> Date: Saturday, July 11, 2009, 5:33 PM >> Mark, >> >> What if the server is so busy you can't connect to it to >> get that message back? >> >> We have three cases: >> >> 1) Everything is happy. >> >> 2) Things are really busy. >> >> 3) A million pounds of (fill in whatever you'd like) in a >> five pound bag. >> >> Things can go from #1 to #3 quickly if the project is big, >> and funding is low (which is the goal, to bring distributed >> computing to projects with tiny funding). >> >> So, while it would be ideal for the servers to be able to >> say "no, and please slow way, way down" I think we have to >> assume that all of the project's servers are unreachable >> under an extreme load. >> >> That is when the need is most extreme, and the hardest to >> deal with, so that's the only one I'm really thinking >> about. >> >> -- Lynn >> >> Mark Pottorff wrote: >>> It would seem that having the server be able to accept >> a connection, and then direct the client on any preferred >> backoff would be ideal. The server is in the best position >> to know how long it was down, how long it will take to >> recover to normal bw levels, how many active connections it >> already has, whether deadlines have been extended etc. etc. >>> The client is in the best position to know how >> frequently it has an internet connection available, what the >> available bw to the server tends to be, how much data it has >> to send, etc. >>> If the upload server could respond to requests with a >> message, to the effect of "I'm trying to focus only on >> uploads that are within 12 hours of deadline" and negotiate >> with the client as to whether the uploads are within that >> objective, whether it expects to have an internet connection >> available later, number of files to upload etc. then it >> should be possible to arrive at a single backoff to a point >> in time that the server will be available to handle the >> request at full speed. "OK, we'll upload one file now and >> hit you in 3hrs 12 minutes with the other 4." >>> As the server gets caught up, and bw utilization >> begins to drop, or the server sees holes in the commitments >> it has made, the window extends on the uploads out to 24hrs >> from deadline etc. >>> When an upload completes, a confirmation is made that >> it arrived with all of the data. It would seem an ideal time >> for an overloaded server to convey that it would prefer to >> delay any further uploads. Or establish a simple 1-5 system >> expressing how busy it is. The client then weighs time to >> deadline against likelihood of future internet connection to >> assess whether it has an upload of sufficient priority or >> not. And if not, it delays starting next upload. >>> Really the objective is that all result files across >> the user base get uploaded in priority order. Where priority >> takes in to account bandwidth user and project have >> available, deadline, file size, etc. >>> Same is true for downloads. If server is drowning in >> requests, you want to be servicing the starving clients >> first, not those with full-time networks and fat work >> caches. And only to the extent that they aren't starving >> anymore, until the contention is resolved. >>> Having a machine sitting idle waiting for downloads >> can occur, and the order of downloads isn't optimized >> either. One more file and I'll be able to start processing a >> task, but that's not necessarily the file being downloaded >> right now. Just depends on the backoffs it was assigned etc. >> as retries occurred and connections were satisfied. >>> >>> Running Microsoft's "System Idle Process" will never >> help cure cancer, >>> AIDS nor Alzheimer's. But running rose...@home >> just might! >>> http://boinc.bakerlab.org/rosetta/ >>> >>> >>> >> _______________________________________________ >>> boinc_dev mailing list >>> [email protected] >>> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev >>> To unsubscribe, visit the above URL and >>> (near bottom of page) enter your email address. >>> > > > > _______________________________________________ > boinc_dev mailing list > [email protected] > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev > To unsubscribe, visit the above URL and > (near bottom of page) enter your email address. > _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
