It sounds like your going through what we did a few yeas ago. While the monitoring is nice, it only points to some of the problems. Get the JVM tuned. I've written a few articles at my blog about how we tuned our JVM for high availability. Your right to use Fusion Reactor (or Seefusion) over CF8's heavy monitoring tools.
I also understand the code changes in large multiple apps taking time. We've been working on the "re-write" plan for a few years. So we may even start writing new code. In the mean time we've shored things up with lots of JVM, DB, apache and filesystem tuning. We've fixed the worst parts of the code and occasionally we are blind sided by some bit of bad code that has escaped being under load. You'll need to use the jdbc wrapping of your data sources to get the full details on the rouge queries. Next you should make sure all calls to external dependencies have time out's set and ways to gracefully catch them. CF will kill itself waiting for an external dependency. This happens because the thread gets locked and then another users tries the same thing and their thread gets lock and soon enough your out of threads. By far learning to tune the JVM was our hardest part because at the time not many people were writing about this subject. Now many people are writing about these subjects including myself. After you've made your code and tuning changes you'll want to do some load testing to see how much load your setup can handle. This way with normal monitoring you'll know when your getting close to the limits and have time to plan ahead to increase your servers capabilities. Wil Genovese http://www.trunkful.com One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Jan 15, 2009, at 9:52 PM, Craigsell wrote: > I said not to ask..... > > Just to answer what the heck are we doing --We are looking at all the > possibilities. The instance in question has thousands of pages on it > running all sorts of bad code built over10 years by a host of > developers. > When activity recently picked up on this server and the performance > went > bad, we got a team together and started digging. And, frankly, I was > appalled at what I found. It's not just one query on a page -- we > could > handle that easy. > > We just installed Fusion Reactor and are using it to identify > problem areas, > help with crash protection, etc. Our fixes includes re-writing > code, row > limiting, timeouts, query optimization, and other solutions for > reports. > I've just re-written our Excel POI module to be more efficient to > ease our > report problems temporarily. Our DB is definitely not > underpowered. We > will be tuning the JVM, look at CF defaults, and a host of other > settings. > We have killed threads when needed and are finding that isn't even > enough -- > we are re-starting CF almost every day. > > It takes time to change production code in a large corporate > environment, > especially when there are so many issues. So we have to deal with > these > long running threads without sockets because they aren't going away > any time > soon. Adobe support didn't have a solution for it but recommended > we do > something about it. I just don't know enough about sockets or > monitoring > them to really understand what's involved. I was hoping someone in > this > forum may have dealt with this before and thus save me some time. I > will > see if I can find an expert in our company who can help. > > Yes, Matt, I agree completely -- having a server monitor is a > necessity. I > am amazed at what I am learning about the server and my code. We > never had > Fusion Reactor until now -- we are impressed enough to buy copies > for all > our CFMX7 servers. I like it better than the CFMX8 Server Monitor > but we > won't be doing than for cost reasons (hard to beat free). > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:318046 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
