Re: [ACFUG Discuss] PermGen CF Server Memory Errors
As an update the server still seems to be humming along crash-free going on 3 weeks now and more importantly has cleared the christmas break hurdle. Thanks to everyone who helped with their advice and now I have a better understanding of what is going on in the JVM and permgen and hopefully others will find this thread and it will help them in the future. On Thu, Dec 15, 2011 at 8:24 AM, Cheyenne Throckmorton cheyenne.throckmor...@gmail.com wrote: Willino and I were just talking about it yesterday and no problems for a week. Thats a pretty positive sign. We have decided to not declare victory until after the christmas break, particularly since we won't be monitoring it as closely as during normal work times, which coincidentally is when we need it to work the most seamless. Thank you all for your help and I'll keep you updated particularly if we declare victory over the break. Actually, even if it dies once over the break we will be pretty happy with that too, but so far so good. On Fri, Dec 9, 2011 at 9:25 AM, Ajas Mohammed ajash...@gmail.com wrote: How is it looking now? Any more crashes after the changes? I hope not. :-) By the way lol@elevator example. That was funny. Ajas Mohammed / http://ajashadi.blogspot.com We cannot become what we need to be, remaining what we are. No matter what, find a way. Because thats what winners do. You can't improve what you don't measure. Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives. On Wed, Dec 7, 2011 at 3:14 PM, Cheyenne Throckmorton cheyenne.throckmor...@gmail.com wrote: Fortunately (weird I know) we just had the server crash again this afternoon. It at least gave me an excuse to throw some new arguments into the jvm.config file to restart the cf service We went with suggestions from everybody and one from a ugtv webcast with Carl Meyer that I've been watching to try and wrap my brain around JVM, memory, permgen stuff. Here are the commands we changed -Xmx1182m -XX:MaxPermSize=512m Here is a command we added since it the machine has 4 cores, we decided to hopefully utilize some multi-threading to improve performance -XX:ParallelGCThreads=4 The good thing about the crash was that I have pretty much ruled out any one application being the culprit, as with inspection of several distinct OOM permgen crashes we had different applications running and doing different things. Over the last month, the hibernate application I think is getting the finger pointed at it, only because I believe it uses the most amount of permgen space with all of its dynamic loading. Perhaps its kind of like always blaming me (at 6'11 340lbs for those who don't know me) for setting off the elevator overloaded alarm because I just so happen to use up the most space, but I could have been on the elevator just fine until 80 little kids jumped on and caused the problem of maxing out the load/space. In that scenario, kicking me off the elevator isn't the solution, and perhaps with these new commands we've just gotten a bigger elevator which may work a little faster even though long-term we may need to investigate putting the hibernate app on a diet. I will keep monitoring the situation and update everyone as I learn more, for those of you that are just reading to learn in case you run into this and for those or you expert server techs that love peering into the mind of a coder stuck trying to learn server maintenance problems. Cheyenne On Wed, Dec 7, 2011 at 9:43 AM, Ajas Mohammed ajash...@gmail.comwrote: On first look, I think you really need to go up on your max memory. On our servers, which are still on 32bit windows, we have min setting of 512MB and Max of 1182MB. # Arguments to VM java.args=-server -Xmx512m # Arguments to VM java.args=-server -Xmx1182m Having said that, like Charlie says always, its not a final solution to the problem but in your case I think its a good starting point.Unless you have some kind of other memory constraints, on this server which I dont know of, you should go up. Lot of people say, you should put both the min and max at same setting. In my case, that didnt work out, hence min at 512MB and Max 1182MB. About FusionReactor, I have to say there is nothing to learn. You will grasp in one day. I had written a getting started guide which was posted on FR community devnet site. http://www.fusion-reactor.com/community/developers/autodev_detailed.cfm?article=FRS-238I am sure they have more updated articles as mine was written at time of version 3.5.1 but still its a good start. Let us know if you need help with that. In your case, I would see System Metrics page, running requests page and also stack trace of the request to see what is going on. Others, correct me, if there is anything, I dont mind at all. :-) Ajas
Re: [ACFUG Discuss] PermGen CF Server Memory Errors
Willino and I were just talking about it yesterday and no problems for a week. Thats a pretty positive sign. We have decided to not declare victory until after the christmas break, particularly since we won't be monitoring it as closely as during normal work times, which coincidentally is when we need it to work the most seamless. Thank you all for your help and I'll keep you updated particularly if we declare victory over the break. Actually, even if it dies once over the break we will be pretty happy with that too, but so far so good. On Fri, Dec 9, 2011 at 9:25 AM, Ajas Mohammed ajash...@gmail.com wrote: How is it looking now? Any more crashes after the changes? I hope not. :-) By the way lol@elevator example. That was funny. Ajas Mohammed / http://ajashadi.blogspot.com We cannot become what we need to be, remaining what we are. No matter what, find a way. Because thats what winners do. You can't improve what you don't measure. Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives. On Wed, Dec 7, 2011 at 3:14 PM, Cheyenne Throckmorton cheyenne.throckmor...@gmail.com wrote: Fortunately (weird I know) we just had the server crash again this afternoon. It at least gave me an excuse to throw some new arguments into the jvm.config file to restart the cf service We went with suggestions from everybody and one from a ugtv webcast with Carl Meyer that I've been watching to try and wrap my brain around JVM, memory, permgen stuff. Here are the commands we changed -Xmx1182m -XX:MaxPermSize=512m Here is a command we added since it the machine has 4 cores, we decided to hopefully utilize some multi-threading to improve performance -XX:ParallelGCThreads=4 The good thing about the crash was that I have pretty much ruled out any one application being the culprit, as with inspection of several distinct OOM permgen crashes we had different applications running and doing different things. Over the last month, the hibernate application I think is getting the finger pointed at it, only because I believe it uses the most amount of permgen space with all of its dynamic loading. Perhaps its kind of like always blaming me (at 6'11 340lbs for those who don't know me) for setting off the elevator overloaded alarm because I just so happen to use up the most space, but I could have been on the elevator just fine until 80 little kids jumped on and caused the problem of maxing out the load/space. In that scenario, kicking me off the elevator isn't the solution, and perhaps with these new commands we've just gotten a bigger elevator which may work a little faster even though long-term we may need to investigate putting the hibernate app on a diet. I will keep monitoring the situation and update everyone as I learn more, for those of you that are just reading to learn in case you run into this and for those or you expert server techs that love peering into the mind of a coder stuck trying to learn server maintenance problems. Cheyenne On Wed, Dec 7, 2011 at 9:43 AM, Ajas Mohammed ajash...@gmail.com wrote: On first look, I think you really need to go up on your max memory. On our servers, which are still on 32bit windows, we have min setting of 512MB and Max of 1182MB. # Arguments to VM java.args=-server -Xmx512m # Arguments to VM java.args=-server -Xmx1182m Having said that, like Charlie says always, its not a final solution to the problem but in your case I think its a good starting point.Unless you have some kind of other memory constraints, on this server which I dont know of, you should go up. Lot of people say, you should put both the min and max at same setting. In my case, that didnt work out, hence min at 512MB and Max 1182MB. About FusionReactor, I have to say there is nothing to learn. You will grasp in one day. I had written a getting started guide which was posted on FR community devnet site. http://www.fusion-reactor.com/community/developers/autodev_detailed.cfm?article=FRS-238I am sure they have more updated articles as mine was written at time of version 3.5.1 but still its a good start. Let us know if you need help with that. In your case, I would see System Metrics page, running requests page and also stack trace of the request to see what is going on. Others, correct me, if there is anything, I dont mind at all. :-) Ajas Mohammed / http://ajashadi.blogspot.com We cannot become what we need to be, remaining what we are. No matter what, find a way. Because thats what winners do. You can't improve what you don't measure. Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives. On Wed, Dec 7, 2011 at 12:01 AM, Cheyenne Throckmorton
Re: [ACFUG Discuss] PermGen CF Server Memory Errors
On first look, I think you really need to go up on your max memory. On our
servers, which are still on 32bit windows, we have min setting of 512MB and
Max of 1182MB.
# Arguments to VM
java.args=-server -Xmx512m
# Arguments to VM
java.args=-server -Xmx1182m
Having said that, like Charlie says always, its not a final solution to the
problem but in your case I think its a good starting point.Unless you have
some kind of other memory constraints, on this server which I dont know of,
you should go up.
Lot of people say, you should put both the min and max at same setting. In
my case, that didnt work out, hence min at 512MB and Max 1182MB.
About FusionReactor, I have to say there is nothing to learn. You will
grasp in one day. I had written a getting started guide which was posted on
FR community devnet site.
http://www.fusion-reactor.com/community/developers/autodev_detailed.cfm?article=FRS-238I
am sure they have more updated articles as mine was written at time of
version 3.5.1 but still its a good start.
Let us know if you need help with that. In your case, I would see System
Metrics page, running requests page and also stack trace of the request to
see what is going on.
Others, correct me, if there is anything, I dont mind at all. :-)
Ajas Mohammed /
http://ajashadi.blogspot.com
We cannot become what we need to be, remaining what we are.
No matter what, find a way. Because thats what winners do.
You can't improve what you don't measure.
Quality is never an accident; it is always the result of high intention,
sincere effort, intelligent direction and skillful execution; it represents
the wise choice of many alternatives.
On Wed, Dec 7, 2011 at 12:01 AM, Cheyenne Throckmorton
cheyenne.throckmor...@gmail.com wrote:
You guys rock with all kinds of answers, I'll definitely be trying them
out tomorrow when I have a better brain and before whirly. Below is my
JVM.config file.
Also for others maybe following the thread I found this 1 of 4 article by
Charlie helpful, but then for Charlie I was trying to find the other parts
after I had determined that I was indeed getting the outofmemory errors in
the -out file
http://www.carehart.org/blog/client/index.cfm/2010/11/3/when_memory_problems_arent_what_they_seem_part_1
One thing I don't get about the -out files is why -out and -out1 are the
only current ones and then -out2 seems to be when the server first started
recording and then they only go to -out200 which was before I started
seeing this problem. I see in your article I can make them bigger than
200kb, which I may have to do, but it seems weird they stop making at 200
and then I essentially only have the latest 400kb in two files
Ajas good point on the 10-day free trials, we have a copy of one or the
other that we purchased awhile back, but admittedly in our organization
Eric Jones was the main server maintenance guy, until he moved on. He kept
track of those licenses and servers. I've been debating which would be
longer, to find the license and learn the software or fix the problem :)
Finally, revisiting Charlie's idea to look at the last funky thing before
the server went down around 16:24 I do see this 2min earlier, and it is in
the app that utilizes Hibernate, so I'll definitely need to see if I have a
CFC incorrectly set up as an interface or abstract class java style in a
way that CF doesn't like and that potentially could be the root of this
deal. Unfortunately, since my -out logs don't capture the last outage a
few days ago I can't compare, but maybe when I try to re-trigger the event
tomorrow I can get this same result.
12/06 16:22:28 Error [jrpp-4004] - Object instantiation exception.An
exception occurred while instantiating a Java object. The class must not be
an interface or an abstract class. Error: ''. The specific sequence of
files included or processed is:
F:\.\webroot\index.cfm, line: 883
12/06 16:22:44 Information [scheduler-1] - Session
8430f0f809b2ff4d892f6b78787f5d723575 ended on ip . Length: 2:00:01 Active
sessions: -1
12/06 16:22:52 Error [jrpp-4004] - Object instantiation exception.An
exception occurred while instantiating a Java object. The class must not be
an interface or an abstract class. Error: ''. The specific sequence of
files included or processed is: F:\.\webroot\index.cfm,
line: 883
12/06 16:22:56 Error [jrpp-4004] - Exception thrown by error-handling
template:
12/06 16:22:59 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen space
jvm.config file
#
# VM configuration
#
java.home=C:/Program Files/Java/jdk1.6.0_24/jre
# Arguments to VM
java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false
-XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch
-Dcoldfusion.rootDir={application.home}/../
-Djava.security.policy={application.home}/../lib/coldfusion.policy
-Djava.security.auth.policy={application.home}/../lib/neo_jaas.policy
Re: [ACFUG Discuss] PermGen CF Server Memory Errors
Fortunately (weird I know) we just had the server crash again this afternoon. It at least gave me an excuse to throw some new arguments into the jvm.config file to restart the cf service We went with suggestions from everybody and one from a ugtv webcast with Carl Meyer that I've been watching to try and wrap my brain around JVM, memory, permgen stuff. Here are the commands we changed -Xmx1182m -XX:MaxPermSize=512m Here is a command we added since it the machine has 4 cores, we decided to hopefully utilize some multi-threading to improve performance -XX:ParallelGCThreads=4 The good thing about the crash was that I have pretty much ruled out any one application being the culprit, as with inspection of several distinct OOM permgen crashes we had different applications running and doing different things. Over the last month, the hibernate application I think is getting the finger pointed at it, only because I believe it uses the most amount of permgen space with all of its dynamic loading. Perhaps its kind of like always blaming me (at 6'11 340lbs for those who don't know me) for setting off the elevator overloaded alarm because I just so happen to use up the most space, but I could have been on the elevator just fine until 80 little kids jumped on and caused the problem of maxing out the load/space. In that scenario, kicking me off the elevator isn't the solution, and perhaps with these new commands we've just gotten a bigger elevator which may work a little faster even though long-term we may need to investigate putting the hibernate app on a diet. I will keep monitoring the situation and update everyone as I learn more, for those of you that are just reading to learn in case you run into this and for those or you expert server techs that love peering into the mind of a coder stuck trying to learn server maintenance problems. Cheyenne On Wed, Dec 7, 2011 at 9:43 AM, Ajas Mohammed ajash...@gmail.com wrote: On first look, I think you really need to go up on your max memory. On our servers, which are still on 32bit windows, we have min setting of 512MB and Max of 1182MB. # Arguments to VM java.args=-server -Xmx512m # Arguments to VM java.args=-server -Xmx1182m Having said that, like Charlie says always, its not a final solution to the problem but in your case I think its a good starting point.Unless you have some kind of other memory constraints, on this server which I dont know of, you should go up. Lot of people say, you should put both the min and max at same setting. In my case, that didnt work out, hence min at 512MB and Max 1182MB. About FusionReactor, I have to say there is nothing to learn. You will grasp in one day. I had written a getting started guide which was posted on FR community devnet site. http://www.fusion-reactor.com/community/developers/autodev_detailed.cfm?article=FRS-238I am sure they have more updated articles as mine was written at time of version 3.5.1 but still its a good start. Let us know if you need help with that. In your case, I would see System Metrics page, running requests page and also stack trace of the request to see what is going on. Others, correct me, if there is anything, I dont mind at all. :-) Ajas Mohammed / http://ajashadi.blogspot.com We cannot become what we need to be, remaining what we are. No matter what, find a way. Because thats what winners do. You can't improve what you don't measure. Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives. On Wed, Dec 7, 2011 at 12:01 AM, Cheyenne Throckmorton cheyenne.throckmor...@gmail.com wrote: You guys rock with all kinds of answers, I'll definitely be trying them out tomorrow when I have a better brain and before whirly. Below is my JVM.config file. Also for others maybe following the thread I found this 1 of 4 article by Charlie helpful, but then for Charlie I was trying to find the other parts after I had determined that I was indeed getting the outofmemory errors in the -out file http://www.carehart.org/blog/client/index.cfm/2010/11/3/when_memory_problems_arent_what_they_seem_part_1 One thing I don't get about the -out files is why -out and -out1 are the only current ones and then -out2 seems to be when the server first started recording and then they only go to -out200 which was before I started seeing this problem. I see in your article I can make them bigger than 200kb, which I may have to do, but it seems weird they stop making at 200 and then I essentially only have the latest 400kb in two files Ajas good point on the 10-day free trials, we have a copy of one or the other that we purchased awhile back, but admittedly in our organization Eric Jones was the main server maintenance guy, until he moved on. He kept track of those licenses and servers. I've been debating which would be longer, to
Re: [ACFUG Discuss] PermGen CF Server Memory Errors
I would go ahead and make the perm gen 256 or 300MB
I would be surprised if they are throwing the hibernate objects into the
perm gen, but it sounds like that may be the case. Can you track the
perm gen usage before that app loads in and then after to see if it
increases when the hibernate app loads? Just throw in either merlin,
fusionreactor or seefusion to see.
John
ma...@fusionlink.com
On 12/6/11 10:09 PM, Cheyenne Throckmorton wrote:
One of our CF servers keeps needing to have the CF service rebooted on
it in order to work and continue serving our sites 1-2x / week.
Fortunately we do have a web service on one of the other box that
monitors this machine to let us know when it starts bugging out again.
The error I am getting on the server has to do with an out of memory
perm gen space issue, something that there is tons of stuff online
about but nothing that I've been able to succinctly tell is a good
idea to look toward resolving the issue. Here are the errors we get
before the service fails.
javax.servlet.ServletException: ROOT CAUSE:
java.lang.OutOfMemoryError: PermGen space
12/06 16:24:46 Error [jrpp-3954] - PermGen space The specific sequence
of files included or processed is:
F:\...\case-studies\ENDO-enduring-hot-seat-discussion\index.cfm''
12/06 16:24:46 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen space
javax.servlet.ServletException: ROOT CAUSE:
java.lang.OutOfMemoryError: PermGen space
Server: Windows Server 2008 R2 64bit
ColdFusion 9,0,1,274733 Standard
Java Version: 1.6.0_24
RAM 8GB
JVM Arguments
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m
-XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../
-Dcoldfusion.libPath={application.home}/../lib
Despite my attendance of multiple Charlie lectures and/or because of
such attendance I certainly do not consider myself an expert in the
memory underpinnings of CF. What I do believe is happening through
some research and late night testing is that essentially the server
stores information (in think in cfclasses) that is essentially
compiled java code. It does this to speed the delivery of data
generated by CF in a caching like mechanism.
There is a checkmark in the settings that says Save Class Files
which I suspect would solve the problem if indeed this area is getting
overflowed and not properly Garbage Collected (if GC even runs in
PermGen). However, unchecking that is not recommended for a
production machine, and it would obviously slow down all of the sites.
Similarly I've seen posts that say to just increase the MaxPermSize,
but I find many posts after those saying that is not solve and just
delaying the problem. I'd rather solve the problem.
Additionally, when observing the websites when it is acting up we get
a lot of totally blank pages along with 200 OK http response codes. I
have also gotten partial pages at times, but that all leads me to some
sort of partial template loading until the permgen runs out of memory.
Finally, I don't know that it is a 100% deal, but we have launched a
new application on this server that makes use of the new ORM Hibernate
functionality. This happened right around the same time we started
observing these issues. This was my first shot at ORM within CF and I
know I need to go back and re-look at how I have the lazy-loading set
up. I have a feeling that it can definitely be tuned better and that
it's possible this type of code may make heavy use of the permgen
space with dynamically loading data in the way that it works. This
heavy banging in the permgen along with an untuned JVM may be the
combo that is causing our problems, or maybe I'm way off. Figured I'd
throw it out there to group to see what thoughts and solutions might
be out there beyond my teams continued googling of words like
permgen, coldfusion, hibernate, orm and arehart :)
- Cheyenne Throckmorton
P.S. Looking forward to seeing folks at Whirlyball tomorrow night.
--
Cheyenne Throckmorton - Atlanta, GA
Blog : www.CheyenneJack.com http://www.CheyenneJack.com
Twitter : @cheyennejack
Founder : www.AtlantaUserGroups.com http://www.AtlantaUserGroups.com
www.TheTallStreetJournal.com http://www.TheTallStreetJournal.com
www.MohawksRock.com http://www.MohawksRock.com
-
To unsubscribe from this list, manage your profile @
http://www.acfug.org?fa=login.edituserform
For more info, see http://www.acfug.org/mailinglists
Archive @ http://www.mail-archive.com/discussion%40acfug.org/
List hosted by http://www.fusionlink.com
-
Re: [ACFUG Discuss] PermGen CF Server Memory Errors
Like John said, try increasing perm size. I see you are not limited by 32
bit system to you can allocated more than 2gb memory of jvm memory as well.
Not sure what you have right now.
Also, have you considered FusionReactor or SeeFusion monitoring tool. Its
amazing how much these tools help you out. You can get a trial for 10 days
to see what happens or what request is running etc.
As for JVM memory monitoring, first thing that comes in my mind is
http://visualvm.java.net/ visualvm. There are several others as well
Ajas Mohammed /
http://ajashadi.blogspot.com
We cannot become what we need to be, remaining what we are.
No matter what, find a way. Because thats what winners do.
You can't improve what you don't measure.
Quality is never an accident; it is always the result of high intention,
sincere effort, intelligent direction and skillful execution; it represents
the wise choice of many alternatives.
On Tue, Dec 6, 2011 at 10:09 PM, Cheyenne Throckmorton
cheyenne.throckmor...@gmail.com wrote:
One of our CF servers keeps needing to have the CF service rebooted on it
in order to work and continue serving our sites 1-2x / week. Fortunately
we do have a web service on one of the other box that monitors this machine
to let us know when it starts bugging out again.
The error I am getting on the server has to do with an out of memory perm
gen space issue, something that there is tons of stuff online about but
nothing that I've been able to succinctly tell is a good idea to look
toward resolving the issue. Here are the errors we get before the service
fails.
javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError:
PermGen space
12/06 16:24:46 Error [jrpp-3954] - PermGen space The specific sequence of
files included or processed is:
F:\...\case-studies\ENDO-enduring-hot-seat-discussion\index.cfm''
12/06 16:24:46 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen space
javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError:
PermGen space
Server: Windows Server 2008 R2 64bit
ColdFusion 9,0,1,274733 Standard
Java Version: 1.6.0_24
RAM 8GB JVM Arguments
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m
-XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../
-Dcoldfusion.libPath={application.home}/../lib
Despite my attendance of multiple Charlie lectures and/or because
of such attendance I certainly do not consider myself an expert in the
memory underpinnings of CF. What I do believe is happening through some
research and late night testing is that essentially the server stores
information (in think in cfclasses) that is essentially compiled java code.
It does this to speed the delivery of data generated by CF in a caching
like mechanism.
There is a checkmark in the settings that says Save Class Files which I
suspect would solve the problem if indeed this area is getting overflowed
and not properly Garbage Collected (if GC even runs in PermGen). However,
unchecking that is not recommended for a production machine, and it would
obviously slow down all of the sites. Similarly I've seen posts that say
to just increase the MaxPermSize, but I find many posts after those saying
that is not solve and just delaying the problem. I'd rather solve the
problem.
Additionally, when observing the websites when it is acting up we get a
lot of totally blank pages along with 200 OK http response codes. I have
also gotten partial pages at times, but that all leads me to some sort of
partial template loading until the permgen runs out of memory.
Finally, I don't know that it is a 100% deal, but we have launched a new
application on this server that makes use of the new ORM Hibernate
functionality. This happened right around the same time we started
observing these issues. This was my first shot at ORM within CF and I know
I need to go back and re-look at how I have the lazy-loading set up. I
have a feeling that it can definitely be tuned better and that it's
possible this type of code may make heavy use of the permgen space with
dynamically loading data in the way that it works. This heavy banging in
the permgen along with an untuned JVM may be the combo that is causing our
problems, or maybe I'm way off. Figured I'd throw it out there to group to
see what thoughts and solutions might be out there beyond my teams
continued googling of words like permgen, coldfusion, hibernate,
orm and arehart :)
- Cheyenne Throckmorton
P.S. Looking forward to seeing folks at Whirlyball tomorrow night.
--
Cheyenne Throckmorton - Atlanta, GA
Blog : www.CheyenneJack.com
Twitter : @cheyennejack
Founder : www.AtlantaUserGroups.com
www.TheTallStreetJournal.com
www.MohawksRock.com
Re: [ACFUG Discuss] PermGen CF Server Memory Errors
Actually if you can, please post your full jvm.config file. I'm
wondering if maybe the heap is filling up then throwing things into the
perm space.
John
ma...@fusionlink.com
On 12/6/11 10:09 PM, Cheyenne Throckmorton wrote:
One of our CF servers keeps needing to have the CF service rebooted on
it in order to work and continue serving our sites 1-2x / week.
Fortunately we do have a web service on one of the other box that
monitors this machine to let us know when it starts bugging out again.
The error I am getting on the server has to do with an out of memory
perm gen space issue, something that there is tons of stuff online
about but nothing that I've been able to succinctly tell is a good
idea to look toward resolving the issue. Here are the errors we get
before the service fails.
javax.servlet.ServletException: ROOT CAUSE:
java.lang.OutOfMemoryError: PermGen space
12/06 16:24:46 Error [jrpp-3954] - PermGen space The specific sequence
of files included or processed is:
F:\...\case-studies\ENDO-enduring-hot-seat-discussion\index.cfm''
12/06 16:24:46 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen space
javax.servlet.ServletException: ROOT CAUSE:
java.lang.OutOfMemoryError: PermGen space
Server: Windows Server 2008 R2 64bit
ColdFusion 9,0,1,274733 Standard
Java Version: 1.6.0_24
RAM 8GB
JVM Arguments
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m
-XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../
-Dcoldfusion.libPath={application.home}/../lib
Despite my attendance of multiple Charlie lectures and/or because of
such attendance I certainly do not consider myself an expert in the
memory underpinnings of CF. What I do believe is happening through
some research and late night testing is that essentially the server
stores information (in think in cfclasses) that is essentially
compiled java code. It does this to speed the delivery of data
generated by CF in a caching like mechanism.
There is a checkmark in the settings that says Save Class Files
which I suspect would solve the problem if indeed this area is getting
overflowed and not properly Garbage Collected (if GC even runs in
PermGen). However, unchecking that is not recommended for a
production machine, and it would obviously slow down all of the sites.
Similarly I've seen posts that say to just increase the MaxPermSize,
but I find many posts after those saying that is not solve and just
delaying the problem. I'd rather solve the problem.
Additionally, when observing the websites when it is acting up we get
a lot of totally blank pages along with 200 OK http response codes. I
have also gotten partial pages at times, but that all leads me to some
sort of partial template loading until the permgen runs out of memory.
Finally, I don't know that it is a 100% deal, but we have launched a
new application on this server that makes use of the new ORM Hibernate
functionality. This happened right around the same time we started
observing these issues. This was my first shot at ORM within CF and I
know I need to go back and re-look at how I have the lazy-loading set
up. I have a feeling that it can definitely be tuned better and that
it's possible this type of code may make heavy use of the permgen
space with dynamically loading data in the way that it works. This
heavy banging in the permgen along with an untuned JVM may be the
combo that is causing our problems, or maybe I'm way off. Figured I'd
throw it out there to group to see what thoughts and solutions might
be out there beyond my teams continued googling of words like
permgen, coldfusion, hibernate, orm and arehart :)
- Cheyenne Throckmorton
P.S. Looking forward to seeing folks at Whirlyball tomorrow night.
--
Cheyenne Throckmorton - Atlanta, GA
Blog : www.CheyenneJack.com http://www.CheyenneJack.com
Twitter : @cheyennejack
Founder : www.AtlantaUserGroups.com http://www.AtlantaUserGroups.com
www.TheTallStreetJournal.com http://www.TheTallStreetJournal.com
www.MohawksRock.com http://www.MohawksRock.com
-
To unsubscribe from this list, manage your profile @
http://www.acfug.org?fa=login.edituserform
For more info, see http://www.acfug.org/mailinglists
Archive @ http://www.mail-archive.com/discussion%40acfug.org/
List hosted by http://www.fusionlink.com
-
RE: [ACFUG Discuss] PermGen CF Server Memory Errors
On top of all that's been said, I'd add this:
Yes, it's likely that you simply need to increase the maxpermsize. Sometimes
just another 100-300 can be enough to solve the problem.
But you also want to make sure that this is really the *first* error that
puts CF into a bad state. It could be that something else precedes it and
precipitates it.
Look in in the [cf]\runtime\logs\, in its out log for this instance, at the
time of your crash, and look back in that log (before the crash) to see if
any other errors or unusual messages show up. If you see none for 1-20
minutes before that permgen message, then that alone would seem to be the
problem.
As to how it could happen, you're on the right track but not quite right in
what you've concluded. The saving of class files is not quite connected as
you think. That's about compiling, where the permgen is about loading. CF
loads classes into the template cache (and thereby the permgen, when they're
first requested--and are not in the cache already, or later when it's no
longer in the cache for any reason, including a restart. A class can be
removed from the template cache also during the life of the server when the
template cache becomes full and it needs to make room for a more newly
requested one, by removing the least recently used one).
And CF will obtain that class (to load when needed) either by compiling it
from source on the fly, or by pulling it from that cfclasses directory (if
you have enabled that save class files option in the Admin).
So how does the permgen get stressed? Most often by excessive loading of
templates into the template cache. Again, when it gets full, it removes the
oldest to make room for new ones. How could it get full? Because the number
of templates that need to be loaded (CFM files, CFCs, and indeed a class is
created for each METHOD in a CFC) exceeds its size. The default is 1024.
That's also not a hard cap. There is a soft cache on top of that.
If you look in the cfclasses directory, see how many files are there. You
might even sort it by date modified and see how many are recently edited. If
that number is at or near 1024, it's likely your app uses lots of files and
therefore creates lots of classes, which can then cause this stress. If you
increase the size of the template cache, beware that that cache is stored in
the heap, so you may need to raise that, too. (Yes, loading things into the
template cache then affects both the heap and the permgen, from my
observation.)
Does that make sense, does that help? You're right, you don't find much out
there about it. Hope this is helpful. I may blog it once we have some back
and forth. (I've hinted at this info in some entries, but not devoted one to
it.)
/charlie
From: ad...@acfug.org [mailto:ad...@acfug.org] On Behalf Of Cheyenne
Throckmorton
Sent: Tuesday, December 06, 2011 10:09 PM
To: discussion@acfug.org
Subject: [ACFUG Discuss] PermGen CF Server Memory Errors
One of our CF servers keeps needing to have the CF service rebooted on it in
order to work and continue serving our sites 1-2x / week. Fortunately we do
have a web service on one of the other box that monitors this machine to let
us know when it starts bugging out again.
The error I am getting on the server has to do with an out of memory perm
gen space issue, something that there is tons of stuff online about but
nothing that I've been able to succinctly tell is a good idea to look toward
resolving the issue. Here are the errors we get before the service fails.
javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError:
PermGen space
12/06 16:24:46 Error [jrpp-3954] - PermGen space The specific sequence of
files included or processed is:
F:\...\case-studies\ENDO-enduring-hot-seat-discussion\index.
cfm''
12/06 16:24:46 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen space
javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError:
PermGen space
Server: Windows Server 2008 R2 64bit
ColdFusion 9,0,1,274733 Standard
Java Version: 1.6.0_24
RAM 8GB
JVM Arguments
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m
-XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../
-Dcoldfusion.libPath={application.home}/../lib
Despite my attendance of multiple Charlie lectures and/or because of such
attendance I certainly do not consider myself an expert in the memory
underpinnings of CF. What I do believe is happening through some research
and late night testing is that essentially the server stores information (in
think in cfclasses) that is essentially compiled java code. It does this to
speed the delivery of data generated by CF in a caching like mechanism.
There is a checkmark in the settings that says Save Class Files which I
suspect would solve the problem if indeed this area is getting overflowed
and not properly Garbage Collected (if GC even runs in PermGen). However,
unchecking
Re: [ACFUG Discuss] PermGen CF Server Memory Errors
You guys rock with all kinds of answers, I'll definitely be trying them out
tomorrow when I have a better brain and before whirly. Below is my
JVM.config file.
Also for others maybe following the thread I found this 1 of 4 article by
Charlie helpful, but then for Charlie I was trying to find the other parts
after I had determined that I was indeed getting the outofmemory errors in
the -out file
http://www.carehart.org/blog/client/index.cfm/2010/11/3/when_memory_problems_arent_what_they_seem_part_1
One thing I don't get about the -out files is why -out and -out1 are the
only current ones and then -out2 seems to be when the server first started
recording and then they only go to -out200 which was before I started
seeing this problem. I see in your article I can make them bigger than
200kb, which I may have to do, but it seems weird they stop making at 200
and then I essentially only have the latest 400kb in two files
Ajas good point on the 10-day free trials, we have a copy of one or the
other that we purchased awhile back, but admittedly in our organization
Eric Jones was the main server maintenance guy, until he moved on. He kept
track of those licenses and servers. I've been debating which would be
longer, to find the license and learn the software or fix the problem :)
Finally, revisiting Charlie's idea to look at the last funky thing before
the server went down around 16:24 I do see this 2min earlier, and it is in
the app that utilizes Hibernate, so I'll definitely need to see if I have a
CFC incorrectly set up as an interface or abstract class java style in a
way that CF doesn't like and that potentially could be the root of this
deal. Unfortunately, since my -out logs don't capture the last outage a
few days ago I can't compare, but maybe when I try to re-trigger the event
tomorrow I can get this same result.
12/06 16:22:28 Error [jrpp-4004] - Object instantiation exception.An
exception occurred while instantiating a Java object. The class must not be
an interface or an abstract class. Error: ''. The specific sequence of
files included or processed is:
F:\.\webroot\index.cfm, line: 883
12/06 16:22:44 Information [scheduler-1] - Session
8430f0f809b2ff4d892f6b78787f5d723575 ended on ip . Length: 2:00:01 Active
sessions: -1
12/06 16:22:52 Error [jrpp-4004] - Object instantiation exception.An
exception occurred while instantiating a Java object. The class must not be
an interface or an abstract class. Error: ''. The specific sequence of
files included or processed is: F:\.\webroot\index.cfm,
line: 883
12/06 16:22:56 Error [jrpp-4004] - Exception thrown by error-handling
template:
12/06 16:22:59 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen space
jvm.config file
#
# VM configuration
#
java.home=C:/Program Files/Java/jdk1.6.0_24/jre
# Arguments to VM
java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false
-XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch
-Dcoldfusion.rootDir={application.home}/../
-Djava.security.policy={application.home}/../lib/coldfusion.policy
-Djava.security.auth.policy={application.home}/../lib/neo_jaas.policy
-Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars,{application.home}/../wwwroot/WEB-INF/flex/jars
-Dcoldfusion.libPath={application.home}/../lib
#
# commas will be converted to platform specific separator and the result
will be passed
# as -Djava.ext.dirs= to the VM
java.ext.dirs={jre.home}/lib/ext
#
# where to find shared libraries
java.library.path={application.home}/../lib,{application.home}/../jintegra/bin,{application.home}/../jintegra/bin/international,{application.home}/../lib/oosdk/classes/win
system.path.first=false
#
# set the current working directory - useful for Windows to control
# the default search path used when loading DLLs since it comes
# before system directory, windows directory and PATH
java.user.dir={application.home}/../../lib
# JVM classpath
java.class.path={application.home}/servers/lib,{application.home}/../lib/macromedia_drivers.jar,{application.home}/lib/cfmx_mbean.jar,{application.home}/../lib/oosdk/classes,{application.home}/../lib/oosdk/lib,{application.home}/lib
On Tue, Dec 6, 2011 at 10:58 PM, Charlie Arehart char...@carehart.orgwrote:
On top of all that’s been said, I’d add this:
Yes, it’s likely that you simply need to increase the maxpermsize.
Sometimes just another 100-300 can be enough to solve the problem.
But you also want to make sure that this is really the **first** error
that puts CF into a bad state. It could be that something else precedes it
and precipitates it.
Look in in the [cf]\runtime\logs\, in its out log for this instance, at
the time of your crash, and look back in that log (before the crash) to see
if any other errors or unusual messages show up. If you see none for 1-20
minutes before that permgen message, then that alone
