Re: Why is my CF8 server constantly hanging?
1. I increased the Java heap to 512k. 2. I lowered the number of allowed concurrent requests from 12 to 8 3. I added the TIMEOUT param to all my cfhttp and cfpop tags... I realized some of them were hanging. In the future I won't consider that param optional. 4. I configured CF8 Enterprise's auto self-healing stuff So far so good...! On Wed, Dec 3, 2008 at 10:39 PM, Wil Genovese [EMAIL PROTECTED] wrote: Philip, Glad to hear things are getting better. Please do tell us what changes you made that worked. And if something didn't work let us know that also. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 4:17 PM, Philip Kaplan wrote: You are all awesome. Took many of your suggestions, haven't had a crash yet today. I'm crossing my fingers. On Wed, Dec 3, 2008 at 2:05 PM, Jochem van Dieten [EMAIL PROTECTED] wrote: On Wed, Dec 3, 2008 at 7:33 AM, Philip Kaplan wrote: At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Try jstack to get a stack trace: http://jochem.vandieten.net/2008/12/03/using-jdk-tools-to-analyze-coldfusion-hangs/ Jochem ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316370 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
Only because I have overlooked this option more than once, there are server and exception logs under the CF directory. I guess you solved your issue, but for anyone else, it's worth looking at those logs once in a while. I have been surprised at what we've found. David ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316377 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
Hi Philip, If you have so many pageviews/day, I would recommend you to finetune the JVM settings as one of the user already have suggested here. You would need to increase your JVM heapspace. CF's default settings is 512MB for this. Also, it would be a good idea to upgrade your 1Gb Ram to 2 Gb if possible, so that you can have more room for fune tuning the JVM settings. Incase, you are facing the problem at request level, I would also recommend you to go through by Blog entry http://askjayvir.blogspot.com/2008/07/finetune-coldfusion-server-performance.html Thanks Jayesh Viradiya Adobe CF Team Thanks for letting me email you. CF installation mode? Standard or Multi Multi. But I'm not clustering so I might as well have gone single. Can you post the data from the settings summary page here? Here's the top half (i didn't want to show my datasource names and so on). I think this is what you want: http://1.mobog.com/cf-talk/settings_summary.html Can you post your jvm.config file so I can help you tune that? Is this what you're looking for? http://1.mobog.com/cf-talk/jvm-config.html What are your server's hardware specs? 3ghz cpu, 1 processor, 1GB ram. I'm assuming Windows and IIS? Correct. Thanks much, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316173 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Might want to ping Charlie Aerhart as this is an area of his expertise. Dan ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316177 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
Fine tuning the JVM really does fix a lot of issues. We used to get crashes a lot back in the day when I didn't know much about JVM tweaking, especially the heap size stuff. Ever since I updated the JVM and tuned it we rarely ever have a crash anymore. We used to run into the same exact thing by the way. The processes were maxed out, with no easily visible cause for it. ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316192 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
Philip,
Based on the load you mentioned earlier and the other data you've
given me here is what I'd do.
1. upgrade the JVM to 1.6_10
2. backup my jvm.config and then edit the java.args line to this.
java.args= -server -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -
Xmx512m -Xms512m -XX:MaxPermSize=128m -XX:PermSize=128m -XX:
+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=60 -
Dsun.rmi.dgc.server.gcInterval=60
Dcoldfusion.rootDir={application.home}/
3. next take several full stack dumps (not just a single part of one)
and run it through seestack. It will give you comparisons between
each stack dump and identify threads that are present from stack dump
to stack dump. These are the hangers or slow threads.
4. since your traffic isn't so great that debugging will hurt
performance, I would then turn on debugging (based on your IP address)
and then analyze every query that is running per request.
5. use fusion reactor alerts to alert you to a slow running thread. FR
has some good tools to help you see what is happening with the thread.
Typical points of pain I've seen
1. JVM config not right - not enough memory heap, not enough perm
memory, GC interval happening to often or not often enough.
2. Queries returning more results than you expected. Many times
people put CFIF statements in their queries and not all conditions are
accounted for thus an open ended where clause may try to return all
the rows of a table. This can cause issues for large tables.
3. critical external dependancies are slow or failing. Besides DB
access (check this also) , things like CFHTTP, external web services,
mail server (CFPOP) and file system access can cause problems if your
not setting a time out and not gracefully handling the times when the
external dependancy is unavailable.
4. excessive looping or external dependancies can cause issues even if
those dependancies are fast. Looping ten times over a resource call
that takes 1/2 second to run still give a thread time of 5 seconds
plus the time to process the rest of the thread.
This is just the start - if none of the above solves the problems (or
at least make things better) you may have to get in deeper. There is
always the option of doing a full code review. settings up a second
server with nothing but CF on it and see if you can duplicate the
problems. If not then you may be looking at the server itself more
than CF and code. If yes, then use this test server to hammer upon.
Wil Genovese
Sr. Web Application Developer
One man with courage makes a majority.
-Andrew Jackson
A fine is a tax for doing wrong. A tax is a fine for doing well.
~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to
date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316193
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
I finally found it.
http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_18339sliceId=1
This KB explains thread dumps completely. Once you do your set of 3
or more thread dumps then run all the out put from those through
Seestack to help you identify the problems.
Wil Genovese
One man with courage makes a majority.
-Andrew Jackson
A fine is a tax for doing wrong. A tax is a fine for doing well.
On Dec 3, 2008, at 9:53 AM, Wil Genovese wrote:
Philip,
Based on the load you mentioned earlier and the other data you've
given me here is what I'd do.
1. upgrade the JVM to 1.6_10
2. backup my jvm.config and then edit the java.args line to this.
java.args= -server -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -
Xmx512m -Xms512m -XX:MaxPermSize=128m -XX:PermSize=128m -XX:
+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=60 -
Dsun.rmi.dgc.server.gcInterval=60
Dcoldfusion.rootDir={application.home}/
3. next take several full stack dumps (not just a single part of one)
and run it through seestack. It will give you comparisons between
each stack dump and identify threads that are present from stack dump
to stack dump. These are the hangers or slow threads.
4. since your traffic isn't so great that debugging will hurt
performance, I would then turn on debugging (based on your IP address)
and then analyze every query that is running per request.
5. use fusion reactor alerts to alert you to a slow running thread. FR
has some good tools to help you see what is happening with the thread.
Typical points of pain I've seen
1. JVM config not right - not enough memory heap, not enough perm
memory, GC interval happening to often or not often enough.
2. Queries returning more results than you expected. Many times
people put CFIF statements in their queries and not all conditions are
accounted for thus an open ended where clause may try to return all
the rows of a table. This can cause issues for large tables.
3. critical external dependancies are slow or failing. Besides DB
access (check this also) , things like CFHTTP, external web services,
mail server (CFPOP) and file system access can cause problems if your
not setting a time out and not gracefully handling the times when the
external dependancy is unavailable.
4. excessive looping or external dependancies can cause issues even if
those dependancies are fast. Looping ten times over a resource call
that takes 1/2 second to run still give a thread time of 5 seconds
plus the time to process the rest of the thread.
This is just the start - if none of the above solves the problems (or
at least make things better) you may have to get in deeper. There is
always the option of doing a full code review. settings up a second
server with nothing but CF on it and see if you can duplicate the
problems. If not then you may be looking at the server itself more
than CF and code. If yes, then use this test server to hammer upon.
Wil Genovese
Sr. Web Application Developer
One man with courage makes a majority.
-Andrew Jackson
A fine is a tax for doing wrong. A tax is a fine for doing well.
~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to
date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316196
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: Why is my CF8 server constantly hanging?
At my previous job, after switching to CF8, we had a couple of totally
inexplicable crashes when we were using the CF8 server monitor in
production. No hung threads, no pegged cpu, just kablooey and no evidence
before the crash that anything was amiss. Once we switched it off, the
crashes stopped happening.
I think I remember earlier in the thread you mentioned something about the
server monitor. Not saying it's causing your problem, but it's worth a try
to switch it off if you haven't already.
-- Josh
-Original Message-
From: Wil Genovese [mailto:[EMAIL PROTECTED]
Sent: Wednesday, December 03, 2008 7:53 AM
To: cf-talk
Subject: Re: Why is my CF8 server constantly hanging?
Philip,
Based on the load you mentioned earlier and the other data you've
given me here is what I'd do.
1. upgrade the JVM to 1.6_10
2. backup my jvm.config and then edit the java.args line to this.
java.args= -server -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -
Xmx512m -Xms512m -XX:MaxPermSize=128m -XX:PermSize=128m -XX:
+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=60 -
Dsun.rmi.dgc.server.gcInterval=60
Dcoldfusion.rootDir={application.home}/
3. next take several full stack dumps (not just a single part of one)
and run it through seestack. It will give you comparisons between
each stack dump and identify threads that are present from stack dump
to stack dump. These are the hangers or slow threads.
4. since your traffic isn't so great that debugging will hurt
performance, I would then turn on debugging (based on your IP address)
and then analyze every query that is running per request.
5. use fusion reactor alerts to alert you to a slow running thread. FR
has some good tools to help you see what is happening with the thread.
Typical points of pain I've seen
1. JVM config not right - not enough memory heap, not enough perm
memory, GC interval happening to often or not often enough.
2. Queries returning more results than you expected. Many times
people put CFIF statements in their queries and not all conditions are
accounted for thus an open ended where clause may try to return all
the rows of a table. This can cause issues for large tables.
3. critical external dependancies are slow or failing. Besides DB
access (check this also) , things like CFHTTP, external web services,
mail server (CFPOP) and file system access can cause problems if your
not setting a time out and not gracefully handling the times when the
external dependancy is unavailable.
4. excessive looping or external dependancies can cause issues even if
those dependancies are fast. Looping ten times over a resource call
that takes 1/2 second to run still give a thread time of 5 seconds
plus the time to process the rest of the thread.
This is just the start - if none of the above solves the problems (or
at least make things better) you may have to get in deeper. There is
always the option of doing a full code review. settings up a second
server with nothing but CF on it and see if you can duplicate the
problems. If not then you may be looking at the server itself more
than CF and code. If yes, then use this test server to hammer upon.
Wil Genovese
Sr. Web Application Developer
One man with courage makes a majority.
-Andrew Jackson
A fine is a tax for doing wrong. A tax is a fine for doing well.
~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to
date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316198
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
On Wed, Dec 3, 2008 at 7:33 AM, Philip Kaplan wrote: At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Try jstack to get a stack trace: http://jochem.vandieten.net/2008/12/03/using-jdk-tools-to-analyze-coldfusion-hangs/ Jochem ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316215 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
You are all awesome. Took many of your suggestions, haven't had a crash yet today. I'm crossing my fingers. On Wed, Dec 3, 2008 at 2:05 PM, Jochem van Dieten [EMAIL PROTECTED] wrote: On Wed, Dec 3, 2008 at 7:33 AM, Philip Kaplan wrote: At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Try jstack to get a stack trace: http://jochem.vandieten.net/2008/12/03/using-jdk-tools-to-analyze-coldfusion-hangs/ Jochem ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316216 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
Philip, Glad to hear things are getting better. Please do tell us what changes you made that worked. And if something didn't work let us know that also. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 4:17 PM, Philip Kaplan wrote: You are all awesome. Took many of your suggestions, haven't had a crash yet today. I'm crossing my fingers. On Wed, Dec 3, 2008 at 2:05 PM, Jochem van Dieten [EMAIL PROTECTED] wrote: On Wed, Dec 3, 2008 at 7:33 AM, Philip Kaplan wrote: At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Try jstack to get a stack trace: http://jochem.vandieten.net/2008/12/03/using-jdk-tools-to-analyze-coldfusion-hangs/ Jochem ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316236 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Why is my CF8 server constantly hanging?
Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash protection and more, to no avail. It dies almost every day and the only way to fix it is to reboot the server (sometimes restarting the CF process works but not always). There's probably some stupid thread or script that's killing the server but I can't figure out which one. I've looked through thread dumps and stack traces but I don't really know how to read them and nothing really jumps out as me. Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Banging my head against a wall, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316140 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor. How many are there? How long have they been running? What are they doing? (stack trace) I'm more familiar with seefusion than reactor, but the stack trace has the stuff you are the most about on the top. It should give you the cf file being executed and the line number. It sounds like you might have an endless loop or something. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 12:07 AM Subject: Why is my CF8 server constantly hanging? Hi, I give up. ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316141 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
Hi there, I know the feeling. We've been there before at our company. Have you tried tuning the JVM yet? There are many items to look at in the JVM to help you handle larger amounts of traffic. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:07 AM, Philip Kaplan wrote: Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash protection and more, to no avail. It dies almost every day and the only way to fix it is to reboot the server (sometimes restarting the CF process works but not always). There's probably some stupid thread or script that's killing the server but I can't figure out which one. I've looked through thread dumps and stack traces but I don't really know how to read them and nothing really jumps out as me. Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Banging my head against a wall, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316142 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor Usually when Jrun is doing its 100% thing, FusionReactor won't load. I suppose the server is hogged. But there are never more than 12 requests running at a time (that's the max I set). How long have they been running? Whatever requests are running during the 100% thing, will continue to run forever until the server (or process) is restarted. What are they doing? (stack trace) I admit I don't really know how to effectively read a stack trace. There's a ton of stuff in there. Would it be useful or annoying if I pasted one to cf-talk, next time it happens? Philip On Tue, Dec 2, 2008 at 10:19 PM, Brad Wood [EMAIL PROTECTED] wrote: When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor. How many are there? How long have they been running? What are they doing? (stack trace) I'm more familiar with seefusion than reactor, but the stack trace has the stuff you are the most about on the top. It should give you the cf file being executed and the line number. It sounds like you might have an endless loop or something. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 12:07 AM Subject: Why is my CF8 server constantly hanging? Hi, I give up. ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316145 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
Have you tried tuning the JVM yet A little. Not a lot. At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Philip On Tue, Dec 2, 2008 at 10:16 PM, Wil Genovese [EMAIL PROTECTED] wrote: Hi there, I know the feeling. We've been there before at our company. Have you tried tuning the JVM yet? There are many items to look at in the JVM to help you handle larger amounts of traffic. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:07 AM, Philip Kaplan wrote: Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash protection and more, to no avail. It dies almost every day and the only way to fix it is to reboot the server (sometimes restarting the CF process works but not always). There's probably some stupid thread or script that's killing the server but I can't figure out which one. I've looked through thread dumps and stack traces but I don't really know how to read them and nothing really jumps out as me. Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Banging my head against a wall, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316146 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
You don't have to worry too much about trying to read the stack trace. Use this tool. http://www.seefusion.com/seestack/seestack.cfm It's called see stack. Dump your stack traces into the seestack tool and it will do the work for you and give you some meaningful information. Usually the things your looking for are threads that are present from trace to trace to trace. Those are usually the hung up or slow running threads. At that point the details of the thread should give you a clue as to what actions was being taken at the time. Also, you mentioned that no more than 12 requests at a time are allowed. Are you maxed out at 12, meaning there are normally threads in the que waiting to be processed? Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:31 AM, Philip Kaplan wrote: When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor Usually when Jrun is doing its 100% thing, FusionReactor won't load. I suppose the server is hogged. But there are never more than 12 requests running at a time (that's the max I set). How long have they been running? Whatever requests are running during the 100% thing, will continue to run forever until the server (or process) is restarted. What are they doing? (stack trace) I admit I don't really know how to effectively read a stack trace. There's a ton of stuff in there. Would it be useful or annoying if I pasted one to cf-talk, next time it happens? Philip On Tue, Dec 2, 2008 at 10:19 PM, Brad Wood [EMAIL PROTECTED] wrote: When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor. How many are there? How long have they been running? What are they doing? (stack trace) I'm more familiar with seefusion than reactor, but the stack trace has the stuff you are the most about on the top. It should give you the cf file being executed and the line number. It sounds like you might have an endless loop or something. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 12:07 AM Subject: Why is my CF8 server constantly hanging? Hi, I give up. ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316147 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
Use this tool. http://www.seefusion.com/seestack/seestack.cfm I will - thank you! Are you maxed out at 12, meaning there are normally threads in the que waiting to be processed? No. Normally there is rarely more than 1 thread running at a time. Except when a crash happens -- then it immediately becomes maxed out (because all threads become hung) with 12 threads. I look forward to another crash so I can use the SeeStack tool :-] On Tue, Dec 2, 2008 at 10:38 PM, Wil Genovese [EMAIL PROTECTED] wrote: You don't have to worry too much about trying to read the stack trace. Use this tool. http://www.seefusion.com/seestack/seestack.cfm It's called see stack. Dump your stack traces into the seestack tool and it will do the work for you and give you some meaningful information. Usually the things your looking for are threads that are present from trace to trace to trace. Those are usually the hung up or slow running threads. At that point the details of the thread should give you a clue as to what actions was being taken at the time. Also, you mentioned that no more than 12 requests at a time are allowed. Are you maxed out at 12, meaning there are normally threads in the que waiting to be processed? Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:31 AM, Philip Kaplan wrote: When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor Usually when Jrun is doing its 100% thing, FusionReactor won't load. I suppose the server is hogged. But there are never more than 12 requests running at a time (that's the max I set). How long have they been running? Whatever requests are running during the 100% thing, will continue to run forever until the server (or process) is restarted. What are they doing? (stack trace) I admit I don't really know how to effectively read a stack trace. There's a ton of stuff in there. Would it be useful or annoying if I pasted one to cf-talk, next time it happens? Philip On Tue, Dec 2, 2008 at 10:19 PM, Brad Wood [EMAIL PROTECTED] wrote: When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor. How many are there? How long have they been running? What are they doing? (stack trace) I'm more familiar with seefusion than reactor, but the stack trace has the stuff you are the most about on the top. It should give you the cf file being executed and the line number. It sounds like you might have an endless loop or something. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 12:07 AM Subject: Why is my CF8 server constantly hanging? Hi, I give up. ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316148 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
ok, lets got more basic. CF installation mode? Standard or Multi Can you post the data from the settings summary page here? Can you post your jvm.config file so I can help you tune that? I wrote a blog piece on JVM tuning. http://www.trunkful.com use with caution - editing your jvm.config is tricky and can be dangerous. Make a backup first. And always edit by hand with a plain text text editor. What are your server's hardware specs? This will help in deciding tuning options. I'm assuming Windows and IIS? If so, I can offer little help there. I do not run CF Server on Windows ever. And I never use IIS. Or if you don't want al that info public, just email me privately. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:33 AM, Philip Kaplan wrote: Have you tried tuning the JVM yet A little. Not a lot. At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Philip On Tue, Dec 2, 2008 at 10:16 PM, Wil Genovese [EMAIL PROTECTED] wrote: Hi there, I know the feeling. We've been there before at our company. Have you tried tuning the JVM yet? There are many items to look at in the JVM to help you handle larger amounts of traffic. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:07 AM, Philip Kaplan wrote: Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash protection and more, to no avail. It dies almost every day and the only way to fix it is to reboot the server (sometimes restarting the CF process works but not always). There's probably some stupid thread or script that's killing the server but I can't figure out which one. I've looked through thread dumps and stack traces but I don't really know how to read them and nothing really jumps out as me. Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Banging my head against a wall, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316149 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
What are your memory settings for the JVM (e.g. maximum heap, perm and stack)? What does the Fusion Reactor log say about the memory consumption just before the problem? What do the CF logs show for errors before the problem? mxAjax / CFAjax docs and other useful articles: http://www.bifrost.com.au/blog/ 2008/12/3 Philip Kaplan [EMAIL PROTECTED]: Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash protection and more, to no avail. It dies almost every day and the only way to fix it is to reboot the server (sometimes restarting the CF process works but not always). There's probably some stupid thread or script that's killing the server but I can't figure out which one. I've looked through thread dumps and stack traces but I don't really know how to read them and nothing really jumps out as me. Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Banging my head against a wall, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316151 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
I'm not sure about fusion-reactor, but the product SeeFusion does a pretty good job of staying de-coupled from Jrun and CF. Usually it will respond up until the JVM dies. Perhaps fusion reactor has a feature similar to seefusion's active monitor rules. you can have triggers shoot you an e-mail that show you all the running request and a stack dump at the time the server is slowing down. A full thread dump of your JVM can be a little crazy. Just stick to the threads named jrrp-xx (I think) Those are the threads related to actual web requests. Most of the threads won't be doing anything. The ones you are about are the ones processing files. That is most helpful at a time like yours is to see specifically what requests are running and exactly what those specific requests are doing. That will help you narrow down all the stack trace stuff. IF you have something like an endless loop, you will see the threads stuck on the same like of code over and over as you pull a stack trace. The trick to reading a full thread dump is usually just looking for patterns. Anytime you repeatedly see a file/function coming up during your thread dump that is a sign you need to inspect what that code is doing. Please paste a dump from a single thread that you don't know how to read and we will point out the important pieces. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 12:31 AM Subject: Re: Why is my CF8 server constantly hanging? When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor Usually when Jrun is doing its 100% thing, FusionReactor won't load. I suppose the server is hogged. But there are never more than 12 requests running at a time (that's the max I set). How long have they been running? Whatever requests are running during the 100% thing, will continue to run forever until the server (or process) is restarted. What are they doing? (stack trace) I admit I don't really know how to effectively read a stack trace. There's a ton of stuff in there. Would it be useful or annoying if I pasted one to cf-talk, next time it happens? ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316152 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
I look forward to another crash so I can use the SeeStack tool :-] You don't have to wait for a crash-- just pull some thread dumps of long running processes when your server is under load (or just any time really) and play around with looking through the stack to see what that process was doing. ~Brad ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316153 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. It is possible to get a thread dump from your JVM without seefusion or fusion reactor. The problem is, you have to have CF started manually from a DOS window and then you hit control-break. Not very optimal in a production environment. There really isn't any other way to see what is going on without the stack traces though. If you set up some rules to E-mail you the trace when the server first starts to bog down you should be able to get some usable stuff. ~Brad ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316155 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
Thanks for letting me email you. CF installation mode? Standard or Multi Multi. But I'm not clustering so I might as well have gone single. Can you post the data from the settings summary page here? Here's the top half (i didn't want to show my datasource names and so on). I think this is what you want: http://1.mobog.com/cf-talk/settings_summary.html Can you post your jvm.config file so I can help you tune that? Is this what you're looking for? http://1.mobog.com/cf-talk/jvm-config.html What are your server's hardware specs? 3ghz cpu, 1 processor, 1GB ram. I'm assuming Windows and IIS? Correct. Thanks much, Philip On Tue, Dec 2, 2008 at 10:46 PM, Wil Genovese [EMAIL PROTECTED] wrote: ok, lets got more basic. CF installation mode? Standard or Multi Can you post the data from the settings summary page here? Can you post your jvm.config file so I can help you tune that? I wrote a blog piece on JVM tuning. http://www.trunkful.com use with caution - editing your jvm.config is tricky and can be dangerous. Make a backup first. And always edit by hand with a plain text text editor. What are your server's hardware specs? This will help in deciding tuning options. I'm assuming Windows and IIS? If so, I can offer little help there. I do not run CF Server on Windows ever. And I never use IIS. Or if you don't want al that info public, just email me privately. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:33 AM, Philip Kaplan wrote: Have you tried tuning the JVM yet A little. Not a lot. At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Philip On Tue, Dec 2, 2008 at 10:16 PM, Wil Genovese [EMAIL PROTECTED] wrote: Hi there, I know the feeling. We've been there before at our company. Have you tried tuning the JVM yet? There are many items to look at in the JVM to help you handle larger amounts of traffic. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:07 AM, Philip Kaplan wrote: Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash protection and more, to no avail. It dies almost every day and the only way to fix it is to reboot the server (sometimes restarting the CF process works but not always). There's probably some stupid thread or script that's killing the server but I can't figure out which one. I've looked through thread dumps and stack traces but I don't really know how to read them and nothing really jumps out as me. Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Banging my head against a wall, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316157 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
WTF -Xmx200M your permgen space is 192 and I think that comes out of that. Why is your max JVM size so small? ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 1:10 AM Subject: Re: Why is my CF8 server constantly hanging? ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316158 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
A couple more notes: This probably won't fix your current problems, but I see you have yet to apply updater 1. Also, you are on the default JVM of 1.6. Does your application use CFC's heavily? ~Brad - Original Message - From: Brad Wood [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 1:19 AM Subject: Re: Why is my CF8 server constantly hanging? ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316160 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
Please paste a dump from a single thread that you don't know how to read and we will point out the important pieces. Thanks. Here's a random JRPP thread that's running right now. It seems to have been running for a while, even though the server seems healthy at the moment. I'd love to know what this all means and how it can help me. === *Thread Stack Trace* Trace Time: 23:16:49.775 02-Dec-2008 Thread ID: 0x7d2 (2002) Thread Name: jrpp-18 Priority: 5 Hashcode: 20602404 State: RUNNABLE jrpp-18 prio=5 runnable java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method] java.net.SocketInputStream.read(SocketInputStream.java:129) java.io.BufferedInputStream.fill(BufferedInputStream.java:218) java.io.BufferedInputStream.read1(BufferedInputStream.java:258) java.io.BufferedInputStream.read(BufferedInputStream.java:317) jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:581) jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:573) jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:591) jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:231) jrun.servlet.jrpp.JRunProxyService.swapRunnable(JRunProxyService.java:143) jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable(ThreadPool.java:410) jrunx.scheduler.WorkerThread.run(WorkerThread.java:76) On Tue, Dec 2, 2008 at 11:08 PM, Brad Wood [EMAIL PROTECTED] wrote: I'm not sure about fusion-reactor, but the product SeeFusion does a pretty good job of staying de-coupled from Jrun and CF. Usually it will respond up until the JVM dies. Perhaps fusion reactor has a feature similar to seefusion's active monitor rules. you can have triggers shoot you an e-mail that show you all the running request and a stack dump at the time the server is slowing down. A full thread dump of your JVM can be a little crazy. Just stick to the threads named jrrp-xx (I think) Those are the threads related to actual web requests. Most of the threads won't be doing anything. The ones you are about are the ones processing files. That is most helpful at a time like yours is to see specifically what requests are running and exactly what those specific requests are doing. That will help you narrow down all the stack trace stuff. IF you have something like an endless loop, you will see the threads stuck on the same like of code over and over as you pull a stack trace. The trick to reading a full thread dump is usually just looking for patterns. Anytime you repeatedly see a file/function coming up during your thread dump that is a sign you need to inspect what that code is doing. Please paste a dump from a single thread that you don't know how to read and we will point out the important pieces. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 12:31 AM Subject: Re: Why is my CF8 server constantly hanging? When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor Usually when Jrun is doing its 100% thing, FusionReactor won't load. I suppose the server is hogged. But there are never more than 12 requests running at a time (that's the max I set). How long have they been running? Whatever requests are running during the 100% thing, will continue to run forever until the server (or process) is restarted. What are they doing? (stack trace) I admit I don't really know how to effectively read a stack trace. There's a ton of stuff in there. Would it be useful or annoying if I pasted one to cf-talk, next time it happens? ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316161 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
JVM 1.6.0_01 - upgrade this to 1.6.0_10 Next your jvm.config needs help. Read what I posted at my blog to get help there. I don't want to re-type all of that here. A few points though. only 200Mb for the jvm heap? How much physical memory is there on this server? Also do you have multiple CPU's? Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 1:10 AM, Philip Kaplan wrote: Thanks for letting me email you. CF installation mode? Standard or Multi Multi. But I'm not clustering so I might as well have gone single. Can you post the data from the settings summary page here? Here's the top half (i didn't want to show my datasource names and so on). I think this is what you want: http://1.mobog.com/cf-talk/settings_summary.html Can you post your jvm.config file so I can help you tune that? Is this what you're looking for? http://1.mobog.com/cf-talk/jvm-config.html What are your server's hardware specs? 3ghz cpu, 1 processor, 1GB ram. I'm assuming Windows and IIS? Correct. Thanks much, Philip On Tue, Dec 2, 2008 at 10:46 PM, Wil Genovese [EMAIL PROTECTED] wrote: ok, lets got more basic. CF installation mode? Standard or Multi Can you post the data from the settings summary page here? Can you post your jvm.config file so I can help you tune that? I wrote a blog piece on JVM tuning. http://www.trunkful.com use with caution - editing your jvm.config is tricky and can be dangerous. Make a backup first. And always edit by hand with a plain text text editor. What are your server's hardware specs? This will help in deciding tuning options. I'm assuming Windows and IIS? If so, I can offer little help there. I do not run CF Server on Windows ever. And I never use IIS. Or if you don't want al that info public, just email me privately. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:33 AM, Philip Kaplan wrote: Have you tried tuning the JVM yet A little. Not a lot. At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Philip On Tue, Dec 2, 2008 at 10:16 PM, Wil Genovese [EMAIL PROTECTED] wrote: Hi there, I know the feeling. We've been there before at our company. Have you tried tuning the JVM yet? There are many items to look at in the JVM to help you handle larger amounts of traffic. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:07 AM, Philip Kaplan wrote: Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash protection and more, to no avail. It dies almost every day and the only way to fix it is to reboot the server (sometimes restarting the CF process works but not always). There's probably some stupid thread or script that's killing the server but I can't figure out which one. I've looked through thread dumps and stack traces but I don't really know how to read them and nothing really jumps out as me. Does anyone have any ideas? At this point I'm willing to pay someone to help. Ahh! Banging my head against a wall, Philip ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316162 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Why is my CF8 server constantly hanging?
I just discovered alerts in CF8 Monitor (the monitor that comes with CF8). Setting up crash protection there. On Tue, Dec 2, 2008 at 11:18 PM, Philip Kaplan [EMAIL PROTECTED] wrote: Please paste a dump from a single thread that you don't know how to read and we will point out the important pieces. Thanks. Here's a random JRPP thread that's running right now. It seems to have been running for a while, even though the server seems healthy at the moment. I'd love to know what this all means and how it can help me. === *Thread Stack Trace* Trace Time: 23:16:49.775 02-Dec-2008 Thread ID: 0x7d2 (2002) Thread Name: jrpp-18 Priority: 5 Hashcode: 20602404 State: RUNNABLE jrpp-18 prio=5 runnable java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method] java.net.SocketInputStream.read(SocketInputStream.java:129) java.io.BufferedInputStream.fill(BufferedInputStream.java:218) java.io.BufferedInputStream.read1(BufferedInputStream.java:258) java.io.BufferedInputStream.read(BufferedInputStream.java:317) jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:581) jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:573) jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:591) jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:231) jrun.servlet.jrpp.JRunProxyService.swapRunnable(JRunProxyService.java:143) jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable(ThreadPool.java:410 ) jrunx.scheduler.WorkerThread.run(WorkerThread.java:76) On Tue, Dec 2, 2008 at 11:08 PM, Brad Wood [EMAIL PROTECTED] wrote: I'm not sure about fusion-reactor, but the product SeeFusion does a pretty good job of staying de-coupled from Jrun and CF. Usually it will respond up until the JVM dies. Perhaps fusion reactor has a feature similar to seefusion's active monitor rules. you can have triggers shoot you an e-mail that show you all the running request and a stack dump at the time the server is slowing down. A full thread dump of your JVM can be a little crazy. Just stick to the threads named jrrp-xx (I think) Those are the threads related to actual web requests. Most of the threads won't be doing anything. The ones you are about are the ones processing files. That is most helpful at a time like yours is to see specifically what requests are running and exactly what those specific requests are doing. That will help you narrow down all the stack trace stuff. IF you have something like an endless loop, you will see the threads stuck on the same like of code over and over as you pull a stack trace. The trick to reading a full thread dump is usually just looking for patterns. Anytime you repeatedly see a file/function coming up during your thread dump that is a sign you need to inspect what that code is doing. Please paste a dump from a single thread that you don't know how to read and we will point out the important pieces. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 12:31 AM Subject: Re: Why is my CF8 server constantly hanging? When jrun is doing the 100% thing, look at the currently running requests in fusion-reactor Usually when Jrun is doing its 100% thing, FusionReactor won't load. I suppose the server is hogged. But there are never more than 12 requests running at a time (that's the max I set). How long have they been running? Whatever requests are running during the 100% thing, will continue to run forever until the server (or process) is restarted. What are they doing? (stack trace) I admit I don't really know how to effectively read a stack trace. There's a ton of stuff in there. Would it be useful or annoying if I pasted one to cf-talk, next time it happens? ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316164 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Why is my CF8 server constantly hanging?
Ditto. Wil's suggestion on the JVM version is basically what I was getting at when I asked if you use cfcs a lot. Java 1.6 has a classloader bug that will bring down your server if you have an app creating thousands of components. Updater 10 from Sun finally fixed it. To answer Wil's question, Philip said he has a gig. What needs to be answered here is what else dooes this machine do? SQL server? mail server? ftp server? etc. If this box does nothing but CF, you can probably get away with a max heap of 512. I really like to give Windows breathing room so you aren't rolling over to your swap disk and thrashing. On a server I'd hate to give windows much less than 500 megs. If you are running sql or anything else on this server, you are cutting yourself seriously thin on RAM. SQL server 2000 can easily suck 500 megs itself. How much RAM your app needs really depends on how much stuff you are caching among other things, but the max you can use with 32 bit CF is like 1.8 gigs or something per instance. In a production server with a healthy app I like to see CF have at least 512 megs to much around in. With a max of 200, I wouldn't be surprised if you are running out of memory. You need to find out how large the heap has grown to when you are slowing down. fusion reactor should show you this. Also, you have enteprise, so the server monitor is also available to you. secondly you need to see how windows is doing on memory as well. Pull open task manager, go to the performance tab and look at your memory usage. ~Brad - Original Message - From: Wil Genovese [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 1:19 AM Subject: Re: Why is my CF8 server constantly hanging? JVM 1.6.0_01 - upgrade this to 1.6.0_10 Next your jvm.config needs help. Read what I posted at my blog to get help there. I don't want to re-type all of that here. A few points though. only 200Mb for the jvm heap? How much physical memory is there on this server? Also do you have multiple CPU's? Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 1:10 AM, Philip Kaplan wrote: Thanks for letting me email you. CF installation mode? Standard or Multi Multi. But I'm not clustering so I might as well have gone single. Can you post the data from the settings summary page here? Here's the top half (i didn't want to show my datasource names and so on). I think this is what you want: http://1.mobog.com/cf-talk/settings_summary.html Can you post your jvm.config file so I can help you tune that? Is this what you're looking for? http://1.mobog.com/cf-talk/jvm-config.html What are your server's hardware specs? 3ghz cpu, 1 processor, 1GB ram. I'm assuming Windows and IIS? Correct. Thanks much, Philip On Tue, Dec 2, 2008 at 10:46 PM, Wil Genovese [EMAIL PROTECTED] wrote: ok, lets got more basic. CF installation mode? Standard or Multi Can you post the data from the settings summary page here? Can you post your jvm.config file so I can help you tune that? I wrote a blog piece on JVM tuning. http://www.trunkful.com use with caution - editing your jvm.config is tricky and can be dangerous. Make a backup first. And always edit by hand with a plain text text editor. What are your server's hardware specs? This will help in deciding tuning options. I'm assuming Windows and IIS? If so, I can offer little help there. I do not run CF Server on Windows ever. And I never use IIS. Or if you don't want al that info public, just email me privately. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:33 AM, Philip Kaplan wrote: Have you tried tuning the JVM yet A little. Not a lot. At risk of sounding like an idiot, is there a way to just figure out WHAT Jrun is working on when it's spiked to 100%? Assume CF Monitor and FusionReactor are unavailable, but I can usually get into the Windows box and mess around. Philip On Tue, Dec 2, 2008 at 10:16 PM, Wil Genovese [EMAIL PROTECTED] wrote: Hi there, I know the feeling. We've been there before at our company. Have you tried tuning the JVM yet? There are many items to look at in the JVM to help you handle larger amounts of traffic. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Dec 3, 2008, at 12:07 AM, Philip Kaplan wrote: Hi, I give up. I have a Windows CF8 server running a few moderately popular websites (around 250,000 pageviews/day). I've written code that I think is efficient. But it crashes a few times every day. I can't figure out why. Jrun CPU spiked at 100% and all processes are hung. I've tried FusionReactor crash
Re: Why is my CF8 server constantly hanging?
That thread actually isn't doing anything. It is waiting for an incoming request. ~Brad - Original Message - From: Philip Kaplan [EMAIL PROTECTED] To: cf-talk cf-talk@houseoffusion.com Sent: Wednesday, December 03, 2008 1:18 AM Subject: Re: Why is my CF8 server constantly hanging? Please paste a dump from a single thread that you don't know how to read and we will point out the important pieces. Thanks. Here's a random JRPP thread that's running right now. It seems to have been running for a while, even though the server seems healthy at the moment. I'd love to know what this all means and how it can help me. ~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;207172674;29440083;f Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:316167 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
