Re: Sudden cpu spike

Fri, 31 Jul 2009 14:24:32 -0700 


On Jul 31, 2009, at 2:22 PM, doug andrews wrote:


I'm not implementing terminate.
sleep is as follows:

public void sleep()
{
 makePageSecure = false;
 skipSecurityCheck = false;
 super.sleep();
}
These are a couple of variables to control if user needs to connect with https or not. 
These guys are using https.


Does not look like anything in there will throw.

How are you handling logout?  Calling session.terminate()?





On Jul 31, 2009, at 5:06 PM, Chuck Hill wrote:



On Jul 31, 2009, at 1:47 PM, doug andrews wrote:


This is still an issue for us.

We have a site with the following:
OS 10.5.7
2.8 Ghz Quad-Core Intel Xeon


How much memory?



WebObjects 5.4.3
Java 1.5.0_19
Apache2.2 (We are using the Apache adaptor from Apple right now, and not the Wonder version) 

There is one instance of our app.
There are about 7 sessions running.
Each session has a separate connection to the database.
Which database? Which version? Separate database connections can consume extra memory. Are you allocating enough heap space to the JVM?
Within about 5 hours or so, all 4 processors run at close to 100%, until eventually the app stops responding.
Which process(es) are using this CPU? top -u will quickly show this. I doubt that you will find it is Java (i.e. your WO app). You might find it is the database.
We have a few sites that have this problem, and the only thing i can see in common is running WO5.4.x on Leopard.
I did manage to get the jstack output when this machine's cpu usage was close to capacity. 
Below is the jstack output.

Does anything jump out as obviously wrong to anybody?
Not exactly. It looks idle. It was really slow to respond at some time in the past and created a lot of worker threads. Eventually those did finish processing. You have a bug in your session's sleep() and / or terminate() methods that is preventing them from being checked in. The bug is likely either a missing call to super() (unlikely, you'd notice this right away), or an exception that is thrown. Wrap the body of each in try...finally: 

public void sleep() {
        try {
                // your buggy code here :-)
        }
        finally {
                super.sleep();
        }
}
kill -QUIT gives better info, BTW. Apple's jstack seems to have gotten worse. 

This is just an idle worker thread:

Thread t...@64771: (state = BLOCKED)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0, line=382 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=50, line=450 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame) - com.webobjects.appserver._private.WOWorkerThread.run() @bci=26, line=210 (Interpreted frame) 
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
This is a request for a session that never got checked back in (see bug above): 
Thread t...@80131: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com .webobjects .appserver.WOSessionStore.checkOutSessionWithID(java.lang.String, com.webobjects.appserver.WORequest) @bci=48, line=191 (Interpreted frame) - com .webobjects .appserver.WOApplication.restoreSessionWithID(java.lang.String, com.webobjects.appserver.WOContext) @bci=9, line=1913 (Interpreted frame) - com .webobjects .appserver ._private .WOComponentRequestHandler ._dispatchWithPreparedApplication (com.webobjects.appserver.WOApplication, com.webobjects.appserver.WOContext, com.webobjects.foundation.NSDictionary) @bci=55, line=324 (Interpreted frame) - com .webobjects .appserver ._private .WOComponentRequestHandler ._handleRequest(com.webobjects.appserver.WORequest) @bci=113, line=369 (Interpreted frame) - com .webobjects .appserver ._private .WOComponentRequestHandler .handleRequest(com.webobjects.appserver.WORequest) @bci=46, line=445 (Interpreted frame) - com .webobjects .appserver .WOApplication.dispatchRequest(com.webobjects.appserver.WORequest) @bci=32, line=1687 (Compiled frame) - Application.dispatchRequest(com.webobjects.appserver.WORequest) @bci=2, line=629 (Interpreted frame) - com.webobjects.appserver._private.WOWorkerThread.runOnce() @bci=473, line=144 (Compiled frame) - com.webobjects.appserver._private.WOWorkerThread.run() @bci=129, line=226 (Interpreted frame) 
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
These are jstack / OS X bugs (and might possibly be hiding the problem in your app but I don't think so): 
Thread t...@95491: (state = IN_NATIVE)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0xc9e58955 with alignment: 4 at sun .jvm .hotspot .debugger.DebuggerUtilities.checkAlignment(DebuggerUtilities.java: 40) at sun .jvm .hotspot .debugger .macosx .MacOSXDebuggerLocal2.readCInteger(MacOSXDebuggerLocal2.java:387) at sun .jvm .hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java: 425) at sun .jvm .hotspot .debugger .macosx.MacOSXDebuggerLocal2.readAddress(MacOSXDebuggerLocal2.java: 257) at sun .jvm .hotspot .debugger.macosx.MacOSXAddress.getAddressAt(MacOSXAddress.java:54) 
        at sun.jvm.hotspot.runtime.x86.X86Frame.getLink(X86Frame.java:338)
        at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
        at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
        at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
        at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
        at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun .jvm .hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java: 231) 
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)


Thread t...@98051: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=120 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=136 (Compiled frame) 
Error occurred during stack walking:
java.lang.NullPointerException
        at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java: 355) 
        at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
        at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
        at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
        at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
        at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)


Thread t...@98311: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
        at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java: 355) 
        at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
        at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
        at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
        at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
        at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)




Chuck




On Sep 23, 2008, at 5:03 PM, Chuck Hill wrote:


Hi Doug,


On Sep 23, 2008, at 12:19 PM, doug andrews wrote:


Still in need of help-
I took your advice and made a build specifically for this site using java 1.5 and the latest xcode and wo. 
I am still having this problem.
We have one instance of our application that is causing each processor to spike to over 90 percent of its cpu capacity. When this happens, one random user will get the 'instance not available'. 

Have you gotten a thread dump from such a run away instance?
http://www.gvcsitemaker.com/gvc.webobjects/faq&mode=single&recordID=41413
Is that one specific instance, or just randomly one of them? If one specific, what is different about it? 




This only happens when someone upgrades to WO 5.4.2 or 5.4.3.
Tiger machines running WO 5.3.x have no problems.
Are the machines upgraded to WO 5.4 running a copy of the app compiled against the same version of WO? 




These are intel machines running OS 10.5.x.
Does anyone know of any changes made in WO 5.4.x (or OS 10.5.x) that could cause this? 

Not off hand.

Chuck




On Aug 25, 2008, at 3:22 PM, Mike Schrag wrote:


No, they were built using Xcode and WO 5.3.3.
This is a really bad idea ... I would never deploy on a version that's different than development. It's just asking for problems. While it MAY work, you're just playing roulette with your app. Either 1) replace 5.4 with 5.3 on that server, 2) rebuild your app with WO 5.3 embedded, or 3) Setup a proper 5.4.2 build and development environment and rebuild and test your app in 5.4.2. 

ms

_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com ) 
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/dandrews%40mediaspansoftware.com

This email sent to dandr...@mediaspansoftware.com


_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com ) 
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/chill%40global-village.net

This email sent to ch...@global-village.net



--
Chuck Hill             Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. 
http://www.global-village.net/products/practical_webobjects








_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com ) 
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/chill%40global-village.net

This email sent to ch...@global-village.net



--
Chuck Hill             Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. 
http://www.global-village.net/products/practical_webobjects











--
Chuck Hill             Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. 
http://www.global-village.net/products/practical_webobjects






_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Reply via email to