Well,
I'm finally getting back to answering those who kindly posted some ideas,
here. It's been the week from hell. Every day a new
crisis.
It
started on Canadian Thanksgiving weekend when our client sent out millions of
emails to their opt-in list, with a special order offer, to come to the
store. We haven't completely analyzed the log files yet, but my
staff tell me that, in one 20-minute period of time, we had many
thousands of people come to the site. This is a bit more of a "burst" than
I have been accustomed to in all my years doing this work.
Today,
we have:
--
moved our database to a brand new and powerful box, to be certain of removing
that as a bottleneck;
--
increased our Witango threadpool size from 15 to 25 on each
webserver;
Both
those things have helped, me thinkst. That said, we did some
other things that relate to good webserver hygiene, that I think will help far
more. We:
--
re-wrote the site home page to be what I would call "quasi-dynamic," meaning,
it's a static page that gets re-written every few minutes. Tons of
people land there and perhaps go no further. Huge off-loading of the
Witango service for that alone;
--
vastly reduced database lookups through the use of domain-scoped variables that
have short time-outs, in the range of 10 minutes;
--
optimized the remaining necessary queries, to cut the time waiting for database
results dramatically; and
-- on
the user-interface side, extended our use of _javascript_ to prevent
multiple clicks on key links. This prevents a server that is a bit slow
while under dramatic load .. from cascading down and completely tanking as 5-10%
more eyeballs translate into a 100-300% increase in traffic due to impatient
repeat clicks.
Finally, we are increasing our customer-facing servers by one yet this
evening, to make this a 7-server product. It may be a bit overkill,
but we have three landing page servers that are primarily content, and
pre-qualifying clients ... moving on to more "relationship" and point of
sale.
We
figure that our application was in over its head by a load factor of maybe ten
times what it could handle. We'll see in the coming days if our
strategy of handling the load (divide, qualify and conquer) will be sufficient
to take the heat.
Alan
... I really like your idea of capturing the @SERVERSTATUS info into the
database every so often, and tracking the performance of the Witango processes
on the boxes over time. Such a great idea. Thanks
..!!
Robert
G ... your point regarding optimizing queries was one that we knew we had to do,
but your reinforcing the point pretty much drove it home for us. We found
some that were using "views" within MS SQL Server, that my developer tells me
were taking far too long because the view was recalculating the entire view when
all we needed was one small aspect of it. Not sure I've got that
correct, but "views" were consuming resources unnecessarily in this
case.
As for
the number of threads, addressed by Robert S, my sense is that we could increase
our number up from 25 ... but I don't know how many. We have plenty of RAM
.. all the more now that we're using domain-scoped variables for some of this --
interestingly, that move alone dramatically reduced our RAM usage ... and a very
fast database server. So, I'm still struggling to learn what might
be optimal, here, and now, with all this new hardware thrown at the problem,
what is my upper end and where will the new constraints be found?
So, we
have a much more robust application (amazing what can be done in a day when the
mind is focused due to fire-breathing clients) ... and we can no doubt handle a
far higher through-put ... but ... the root question remains: what are the
optimal settings for some of these things?
Serious thanks to the list for responding with both ideas and moral
support. It was a tough day, but I am grateful for you and for my
team.
Sincerely,
Ian
Ian
-----Original Message-----
From: Alan Wolfe [mailto:[EMAIL PROTECTED]
Sent: Friday, October 13, 2006 9:09 AM
To: [email protected]
Subject: Re: Witango-Talk: Urgent Question re Thread Count -- Quick Answers, Please ..I had a look at our 2003 servers and they are at 15 too, sorry can't help there ):did your servers all of a sudden get overloaded?If so maybe something else is to blame such as windows update screwing a config item up (a windows update once turned off our http compression and cost us a few hundred dollars) or some new software? Or perhaps you should check your sql query execution times to see if maybe you just need to index some tables better or optomize a query, or something like that.It sounds pretty urgent though so you may not have the option of investigation eh? :Pif you do have some time, one thing we do on our servers is have a crontab event that fires every 15 minutes and logs the values of <@serverstatus> as well as datasource statistics and this lets us view trends on the server (especially when graphed).That combined with a user activity log and maybe even some benchmark logs for involved processes such as reports, you can get some mean research done and find your bottlenecks.
On 10/13/06, Ian Daniel <[EMAIL PROTECTED]> wrote:Ok .. we have some seriously overloaded servers.The Witango 5.x threadcount is by default set at 15.On a Win2k3 server with 2 Gb of ram, what is a safe maximum to adjust this to?Ian Daniel, A.Sc.T., President
New Creation Consulting & Information Management Inc.
# 505, 889 West Pender Street, Vancouver, BC V6C 3B2 Canada
======================================================
[EMAIL PROTECTED] http://www.ncol.comPh: (604) 806-6265 ext 110 Fax: (604) 806-6269________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
