I've been an avid reader of this and other lists for ages, but this is the first time I've needed to ask something.
A client has been having awful problems handling high loads. What happens is that out of the blue CFMX (which has been happily servicing the same level of load with c5 running requests) climbs up to maximum running requests (set to 96 on a quad processor) and then starts queuing requests. Things go dire for a while (sql response times increase, bytes output decreases, request response time obviously increases). Usually it will recover, in anything from 5 seconds to a minute and go back to normal. 99.9% of the time things are fine; it copes with the load with no problems. Maybe 5 times an hour we have a brief bottleneck (10 seconds or so, which clears before it starts queuing requests) and maybe 2 or 3 times a day we max out the running requests and start queuing. Sometimes I have to limit the number of connections, briefly, with IIS to get it to recover. It's an ecommerce site so we can't afford these overloads. What have we eliminated? Most things. 96 running requests is our best number after trial and error. It does allow cf running requests to climb, but not queue most of the time. It was set to 20 - but 96 means we get a lot fewer overloads and works best for us. We did have similar problems with CF 5, but never often enough to be worth investigating. CFMX was initially fine, but the site load has recently gone up by 30% or more. SQL is fine - we have loads of query caching, query of queries - sql response times are fine, < 20ms. I've fine tuned the app, hardware, os, network, IIS - the lot. Overloads happen, but a lot less often than they did before at the same load. Because of the way we have to talk to a legacy system there's an awfully high amount of disk i/o - creating / checking if the reply file exists / reading / deleting some 100,000 files a day. The create / read / delete calls are all cflocked. It's a single server system, with MS SQL server on a separate box. Win 2k server / IIS 5. Application and session variables are used but I believe they are locked adequately (writes are locked, app reads are locked, session reads are not). For the past 3 days I've been trying to catch an overload situation so that I could do a thread dump, but the first successful dump I got wasn't until today. The dump shows that when the site is overloaded the vast majority of running threads are in this state - "jrpp-1068" prio=5 tid=0x256dca90 nid=0x1b40 waiting on monitor [0x2ac4f000..0x2ac4fdb8] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Unknown Source) at coldfusion.util.AbstractCache.fetch(Unknown Source) at coldfusion.util.SoftCache.get(Unknown Source) at coldfusion.runtime.TemplateClassLoader.findClass(Unknown Source) at coldfusion.runtime.RuntimeServiceImpl.checkExists(Unknown Source) at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(Unknown Source) at coldfusion.runtime.TemplateProxy.resolveFullName(Unknown Source) at coldfusion.runtime.TemplateProxy.resolveName(Unknown Source) at coldfusion.runtime.CFPage.CreateObject(Unknown Source) at coldfusion.tagext.lang.ObjectTag.doStartTag(Unknown Source) at coldfusion.runtime.CfJspPage._emptyTag(Unknown Source) at cfenq2ecfm1767146982.runPage(E:\www\xx\xxx\enj.cfm:21) etc Line 21 of that module is the line after a <cfobject> call to load a cfc. The module referenced varies but the line referenced either points to the line after a cfc cfobject call, or points to a custom tag call. I gave up analysing the dump in detail about half way through, but by that time I'd found 37 threads all in the same wait state. Only a very few active threads were apparently actually running - doing something - 4 in fact. To me this indicates some sort of bottleneck in handling the template cache. But what sort of bottleneck, and how do I cure it? Regards, Alan Ford ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Archives: http://www.houseoffusion.com/cf_lists/index.cfm?forumid=4 Subscription: http://www.houseoffusion.com/cf_lists/index.cfm?method=subscribe&forumid=4 FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
