| A couple more things that might help, after dealing with this stuff a lot.
I see a lot of code, where people do many unecessary loops in witango. Avoid loops at all costs. Look for ways to use things like filter, and distinct and ways to parse/filter arrays, rather than looping and evaluating. Some of my developers have poked fun at how much I push this, but I push it hard. If someone wants to use a while loop, for instance, they almost have to submit a request in writing, and must have a REAL good reason, and there must not be another way. I cannot stress enough, avoid looping as much as possible in witango. This is an issue in any language, but much more of an issue in witango.
I have big deployments, that hook to PrimeBase, MS SQL, and Oracle. And all of my stuff, eventpix, etc, all uses MYSQL. I can't tell you the difference we saw when we moved to MYSQL 5 on linux as the DB. To put it bluntly, it has been amazing how LITTLE our witango servers and php servers work, with the amazing fast results from mysql. And it is not just the result speed, but the speed and overhead of opening and closing connections. I have not seen mysqls equal when it comes to the speed of opening connections through MYSQL ODBC 3.5+ or mysqli in php.
Also, we setup slave MYSQL servers, and I have done replication in several other environments. MYSQL was not necessarily the easiest, but def has been the most trouble free, and rock solid. All of our inserts and updates etc go to main server, but all strictly read functions get spread to the slaves. We have a layer of code that spreads the reads, that we can alter on the fly from a central console if there is a failure. It works well.
Admittedly, we had to learn, and learn some more about the best ways to setup and tweak mysql, but even though all of our dbs are inno_db for transactions, they scream. Oh yeah, and its free.
The caching of data in variables, is a great thing, we do that on all our busy apps, and its a huge benefit to optimization. We don't time out so quick, but have triggers that may cause a cache to reload, and have a console to manually reload cache for an app.
Anyway, those 3 things are like 90% of the battle. You can add bigger processors, and it may help the looping issue, but it won't help you at all, if you are waiting for results. It seems to me that the behavior of a witango thread, is that it does a query to the db, and sits in a tight loop waiting for results. This makes sense, for best performance, if the results are quick, but causes a thread to hog cpu while it is waiting. This problem is greatly excagerrated on windows/witango servers, because the process scheduler in windows, is not as effective as linux, and causes the server to hang faster than it should.
HTH.
--
Robert Garcia President - BigHead Technology VP Application Development - eventpix.com 13653 West Park Dr Magalia, Ca 95954 ph: 530.645.4040 x222 fax: 530.645.4040 On Oct 13, 2006, at 11:20 PM, Ian Daniel wrote: Well, I'm finally getting back to answering those who kindly posted some ideas, here. It's been the week from hell. Every day a new crisis. It started on Canadian Thanksgiving weekend when our client sent out millions of emails to their opt-in list, with a special order offer, to come to the store. We haven't completely analyzed the log files yet, but my staff tell me that, in one 20-minute period of time, we had many thousands of people come to the site. This is a bit more of a "burst" than I have been accustomed to in all my years doing this work. Today, we have: -- moved our database to a brand new and powerful box, to be certain of removing that as a bottleneck; -- increased our Witango threadpool size from 15 to 25 on each webserver; Both those things have helped, me thinkst. That said, we did some other things that relate to good webserver hygiene, that I think will help far more. We: -- re-wrote the site home page to be what I would call "quasi-dynamic," meaning, it's a static page that gets re-written every few minutes. Tons of people land there and perhaps go no further. Huge off-loading of the Witango service for that alone; -- vastly reduced database lookups through the use of domain-scoped variables that have short time-outs, in the range of 10 minutes; -- optimized the remaining necessary queries, to cut the time waiting for database results dramatically; and -- on the user-interface side, extended our use of _javascript_ to prevent multiple clicks on key links. This prevents a server that is a bit slow while under dramatic load .. from cascading down and completely tanking as 5-10% more eyeballs translate into a 100-300% increase in traffic due to impatient repeat clicks. Finally, we are increasing our customer-facing servers by one yet this evening, to make this a 7-server product. It may be a bit overkill, but we have three landing page servers that are primarily content, and pre-qualifying clients ... moving on to more "relationship" and point of sale. We figure that our application was in over its head by a load factor of maybe ten times what it could handle. We'll see in the coming days if our strategy of handling the load (divide, qualify and conquer) will be sufficient to take the heat. Alan ... I really like your idea of capturing the @SERVERSTATUS info into the database every so often, and tracking the performance of the Witango processes on the boxes over time. Such a great idea. Thanks ..!! Robert G ... your point regarding optimizing queries was one that we knew we had to do, but your reinforcing the point pretty much drove it home for us. We found some that were using "views" within MS SQL Server, that my developer tells me were taking far too long because the view was recalculating the entire view when all we needed was one small aspect of it. Not sure I've got that correct, but "views" were consuming resources unnecessarily in this case. As for the number of threads, addressed by Robert S, my sense is that we could increase our number up from 25 ... but I don't know how many. We have plenty of RAM .. all the more now that we're using domain-scoped variables for some of this -- interestingly, that move alone dramatically reduced our RAM usage ... and a very fast database server. So, I'm still struggling to learn what might be optimal, here, and now, with all this new hardware thrown at the problem, what is my upper end and where will the new constraints be found? So, we have a much more robust application (amazing what can be done in a day when the mind is focused due to fire-breathing clients) ... and we can no doubt handle a far higher through-put ... but ... the root question remains: what are the optimal settings for some of these things? Serious thanks to the list for responding with both ideas and moral support. It was a tough day, but I am grateful for you and for my team. Sincerely,
Ian I had a look at our 2003 servers and they are at 15 too, sorry can't help there ): did your servers all of a sudden get overloaded? If so maybe something else is to blame such as windows update screwing a config item up (a windows update once turned off our http compression and cost us a few hundred dollars) or some new software? Or perhaps you should check your sql query execution times to see if maybe you just need to index some tables better or optomize a query, or something like that. It sounds pretty urgent though so you may not have the option of investigation eh? :P if you do have some time, one thing we do on our servers is have a crontab event that fires every 15 minutes and logs the values of <@serverstatus> as well as datasource statistics and this lets us view trends on the server (especially when graphed). That combined with a user activity log and maybe even some benchmark logs for involved processes such as reports, you can get some mean research done and find your bottlenecks.
On 10/13/06, Ian Daniel <[EMAIL PROTECTED]> wrote: Ok .. we have some seriously overloaded servers. The Witango 5.x threadcount is by default set at 15. On a Win2k3 server with 2 Gb of ram, what is a safe maximum to adjust this to? Ian Daniel, A.Sc.T., President
New Creation Consulting & Information Management Inc.
# 505, 889 West Pender Street, Vancouver, BC V6C 3B2 Canada
======================================================
[EMAIL PROTECTED] http://www.ncol.com Ph: (604) 806-6265 ext 110 Fax: (604) 806-6269
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
|
