On Mon, Jul 31, 2000 at 04:12:53PM -0400, Richard J. Sexton wrote:
> >Nobody set a hard limit.
>
> I dunno Kent, when it says "the database is overloaded" it sure seems
> to me the software ran up agaisnt a hard limit. No?
Hmm. I thought you were a technical person, and knew about things like
"thrashing". Pardon me for the length of this little essay:
Database overload doesn't imply anything at all about descernible "hard
limits". Here's a scenario:
If all the database accesses complete on average in less than the
arrival time of requests, then everything is cool. However, knowing
those averages is very difficult: 1) the database accesses vary with the
different kinds of incoming requests; 2) the database accesses interact
with each other -- have to lock access to certain items, but not to
others (a difficult to analyze interaction); 3) the number of
simultaneously active processes varies considerably, because you have
multiple instantiations of the web server, cgi scripts, subsidiary
processes called by the scripts, and so on, and this impacts the overall
load average on the system.
As the system gets loaded these various effects interact in bad ways --
this is common knowledge to anyone with any experience at all in
performance tuning, though knowing what to do about it in any
particular case can be very difficult.
All the above is purely internal. However, there are external
interactions that have even more impact: when the system starts to slow
down, people get impatient, and start clicking "back" and "submit" to
resubmit their entry. Clicking "back" doesn't make the process(es) on
the server side go away immediately; clicking "submit" starts another
instance of the cgi script, which adds to the load on the system. This
slows things down more, more users start retrying; and things go down in
a fast decaying spiral which isn't broken until either 1) *lots* of
people give up, or 2) the system crashes. People desperate to register
don't give up easily. Note that it only takes a short spike of heavy
load to send the server into a much longer term slowdown -- it is very
much like what happens in heavy automobile traffic -- conditions
persist.
As you might imagine, the science of predicting when people will start
trying to do retries is not well understood, and certainly the ICANN
staff is in no position to make such detailed analyses -- they are in
the swamp, fighting off alligators.
The internal issues mentioned above cause things to slow down under
load; the external positive feedback loop in the retries causes things
to go seriously to hell. Under these circumstances, essentially no
registrations would get processed. The goal is maximum average throughput,
and if you let the feedback loop hit, you get zero throughput. So you
try to throttle the input to just below the level that the maximum
throughput can be sustained. That is, you pick an arbitrary fixed
limit that is just below what you think your system as a whole can
sustain continuously.
But any throttle mechanism itself uses some processing power --
schematically, if the throttle takes 10% of the processing that a normal
request takes, the system will still keel over when the load get's a
factor of 10 higher. Given the positive feedback loop involved, and the
fact that people are retrying like mad, it just takes a short term heavy
duty spike to get over the threshold into a deadly downward spiral.
Sometimes you get a big win by recoding some critical component, or by
altering the way the database is accessed or structured. Sometimes the
thing to do is just throw more hardware at the problem -- but moving a
running application onto new hardware in midstream is a very tricky
operation.
> >> At the end of the day, planning for 10,000 anf getting 143,000
> >> gets you in trouble. Planning for a million and getting 143,000
> >> doesn't.
> >
> >Sure it does. You've wasted money, lots of it.
>
> Sure. Waste money and it'll work or don't and might fail.
Corporate officers have a fiduciary responsibility not to waste money...
> >> As for the money, Becky Burr/NTIC/DoC has stated in open fora
> >> that if it came right down to it and ICANN couldn't afford
> >> to do what it was doing, DoC would not let it fail because
> >> of money. I have no reason to believe she was lying.
> >
> >Of course! It's so simple! ICANN can spend as much money as it wants
> >because the DOC will bail them out.
>
> That's what Becky said.
Sorry, you are quite incorrect. Becky never at ANY time said "ICANN can
spend as much money as it wants and DOC will bail them out."
"Not let it fail because of lack of money" is a vastly different
statement. Please note: registering 10-20 times the expected number
does not constitute a failure. ICANN made guesses that seemed very
reasonable at the time, and actually handled an order of magnitude more.
Please remember back when: there were multiple complaints that the ICANN
requirement of 5000 registered voters before an election was held was a
cynical ploy by ICANN to eliminate the possibility of elections. There
were lots of people who thought that 5000 was an impossibly high target.
Now, of course, all such thoughts have vanished from the fickle minds of
the critics, and the shrill chorus is "*anyone* could have anticipated
the load!!". These guys are just too funny, you know :-)
> Markle has also said, in public, "if you need more money it's here".
Please note that
1) Talk is cheap.
2) Sometimes money just can't solve the problem.
You are surely familiar with Fred Brooks' "The Mythical Man-Month"?
> >> Alternatives are always an option. If people were asekd "do
> >> you want only the first 10,000 to be able to vote or
> >> do you mind if we ask you to send a self addresses stamped
> >> envelope or a doller or two if you're outside the US" my
> >> off the wall guess is people would pick the latter.
> >
> >Hindsight is wonderful. Planning with off the wall guesses is a
> >breeze.
>
> All we hae is hindsight cause they never made it clear what they
> were doing. Now that we know....
Nothing has changed. You would be finding anything you could to
criticize, regardless of the outcome.
--
Kent Crispin "Do good, and you'll be
[EMAIL PROTECTED] lonesome." -- Mark Twain
