On Jul 31, 2009, at 2:17 PM, doug andrews wrote:
Memory: 4GB 800 Mhz DDR2 FB-DIMM
Database: MySQL 4.1.15
Heap Space: Xmx1024m and Xms1024m
Output from top -u:
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD
RSIZE VSIZE
17494 java 387.8% 8:24:25 81 1126 1091 10M 38M
736M 1322M
=8-0
20799 top 4.0% 0:00.79 1 18 29 604K 188K
1196K 18M
1826 Timbuktu H 2.5% 7:21.72 9 113 144 10M 8000K
16M 185M
1931 pmTool 2.1% 27:30.53 1 63 25 612K 1132K
1484K 27M
0 kernel_tas 1.9% 18:15.98 53 2 502 6116K 0
285M 112M
17761 Terminal 1.7% 0:08.43 3 105 188 4468K 9744K
12M 172M
1930 Activity M 1.6% 27:36.16 7 117 252 6616K 20M
15M 203M
58 hwmond 0.2% 17:11.70 3 65 46 2208K 416K
3852K 20M
183 WindowServ 0.2% 2:02.65 7 199 418 10M 18M
26M 164M
31 mDNSRespon 0.0% 0:10.91 2 36 27 572K 192K
1356K 20M
63 emond 0.0% 0:39.01 1 31 22 368K 1140K
1880K 27M
45 Python 0.0% 1:07.24 1 42 443 29M 1248K
33M 55M
44 java 0.0% 0:32.77 32 608 316 39M 16M
50M 322M
43 java 0.0% 0:55.18 32 661 321 37M 16M
49M 323M
17531 java 0.0% 0:13.42 32 603 312 46M 37M
63M 327M
17341 java 0.0% 0:20.98 31 691 309 59M 37M
77M 327M
17339 java 0.0% 0:13.08 30 585 305 42M 37M
58M 326M
17340 java 0.0% 0:14.07 30 590 302 42M 37M
59M 326M
17342 java 0.0% 0:14.69 32 604 310 45M 37M
62M 327M
40 ntpd 0.0% 0:06.40 1 15 19 196K 1
I will look at my session.terminate and sleep methods.
Can't do the kill QUIT right now because it's a live site.
a) at 388% CPU it is not very live :-P
b) kill -QUIT <pid> just does a thread dump, it does not stop the
process
See:
http://www.gvcsitemaker.com/gvc.webobjects/faq&mode=single&recordID=41413&nextMode=list
Note that if your wotaskd is not configured like this already, you
will need to restart it and the application for kill -QUIT to provide
you with any output.
Chuck
On Jul 31, 2009, at 5:06 PM, Chuck Hill wrote:
On Jul 31, 2009, at 1:47 PM, doug andrews wrote:
This is still an issue for us.
We have a site with the following:
OS 10.5.7
2.8 Ghz Quad-Core Intel Xeon
How much memory?
WebObjects 5.4.3
Java 1.5.0_19
Apache2.2 (We are using the Apache adaptor from Apple right now,
and not the Wonder version)
There is one instance of our app.
There are about 7 sessions running.
Each session has a separate connection to the database.
Which database? Which version? Separate database connections can
consume extra memory. Are you allocating enough heap space to the
JVM?
Within about 5 hours or so, all 4 processors run at close to 100%,
until eventually the app stops responding.
Which process(es) are using this CPU? top -u will quickly show
this. I doubt that you will find it is Java (i.e. your WO app).
You might find it is the database.
We have a few sites that have this problem, and the only thing i
can see in common is running WO5.4.x on Leopard.
I did manage to get the jstack output when this machine's cpu
usage was close to capacity.
Below is the jstack output.
Does anything jump out as obviously wrong to anybody?
Not exactly. It looks idle. It was really slow to respond at some
time in the past and created a lot of worker threads. Eventually
those did finish processing. You have a bug in your session's
sleep() and / or terminate() methods that is preventing them from
being checked in. The bug is likely either a missing call to
super() (unlikely, you'd notice this right away), or an exception
that is thrown. Wrap the body of each in try...finally:
public void sleep() {
try {
// your buggy code here :-)
}
finally {
super.sleep();
}
}
kill -QUIT gives better info, BTW. Apple's jstack seems to have
gotten worse.
This is just an idle worker thread:
Thread t...@64771: (state = BLOCKED)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0,
line=382 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=50,
line=450 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted
frame)
- com.webobjects.appserver._private.WOWorkerThread.run() @bci=26,
line=210 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
This is a request for a session that never got checked back in (see
bug above):
Thread t...@80131: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
-
com
.webobjects
.appserver.WOSessionStore.checkOutSessionWithID(java.lang.String,
com.webobjects.appserver.WORequest) @bci=48, line=191 (Interpreted
frame)
-
com
.webobjects
.appserver.WOApplication.restoreSessionWithID(java.lang.String,
com.webobjects.appserver.WOContext) @bci=9, line=1913 (Interpreted
frame)
-
com
.webobjects
.appserver
._private
.WOComponentRequestHandler
._dispatchWithPreparedApplication
(com.webobjects.appserver.WOApplication,
com.webobjects.appserver.WOContext,
com.webobjects.foundation.NSDictionary) @bci=55, line=324
(Interpreted frame)
-
com
.webobjects
.appserver
._private
.WOComponentRequestHandler
._handleRequest(com.webobjects.appserver.WORequest) @bci=113,
line=369 (Interpreted frame)
-
com
.webobjects
.appserver
._private
.WOComponentRequestHandler
.handleRequest(com.webobjects.appserver.WORequest) @bci=46,
line=445 (Interpreted frame)
-
com
.webobjects
.appserver
.WOApplication.dispatchRequest(com.webobjects.appserver.WORequest)
@bci=32, line=1687 (Compiled frame)
- Application.dispatchRequest(com.webobjects.appserver.WORequest)
@bci=2, line=629 (Interpreted frame)
- com.webobjects.appserver._private.WOWorkerThread.runOnce()
@bci=473, line=144 (Compiled frame)
- com.webobjects.appserver._private.WOWorkerThread.run() @bci=129,
line=226 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
These are jstack / OS X bugs (and might possibly be hiding the
problem in your app but I don't think so):
Thread t...@95491: (state = IN_NATIVE)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read
at address: 0xc9e58955 with alignment: 4
at
sun
.jvm
.hotspot
.debugger.DebuggerUtilities.checkAlignment(DebuggerUtilities.java:
40)
at
sun
.jvm
.hotspot
.debugger
.macosx
.MacOSXDebuggerLocal2.readCInteger(MacOSXDebuggerLocal2.java:387)
at
sun
.jvm
.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:
425)
at
sun
.jvm
.hotspot
.debugger
.macosx.MacOSXDebuggerLocal2.readAddress(MacOSXDebuggerLocal2.java:
257)
at
sun
.jvm
.hotspot
.debugger.macosx.MacOSXAddress.getAddressAt(MacOSXAddress.java:54)
at sun.jvm.hotspot.runtime.x86.X86Frame.getLink(X86Frame.java:338)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at
sun
.jvm
.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:
231)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Thread t...@98051: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=120
(Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=136 (Compiled
frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:
355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Thread t...@98311: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:
355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Chuck
On Sep 23, 2008, at 5:03 PM, Chuck Hill wrote:
Hi Doug,
On Sep 23, 2008, at 12:19 PM, doug andrews wrote:
Still in need of help-
I took your advice and made a build specifically for this site
using java 1.5 and the latest xcode and wo.
I am still having this problem.
We have one instance of our application that is causing each
processor to spike to over 90 percent of its cpu capacity.
When this happens, one random user will get the 'instance not
available'.
Have you gotten a thread dump from such a run away instance?
http://www.gvcsitemaker.com/gvc.webobjects/faq&mode=single&recordID=41413
Is that one specific instance, or just randomly one of them? If
one specific, what is different about it?
This only happens when someone upgrades to WO 5.4.2 or 5.4.3.
Tiger machines running WO 5.3.x have no problems.
Are the machines upgraded to WO 5.4 running a copy of the app
compiled against the same version of WO?
These are intel machines running OS 10.5.x.
Does anyone know of any changes made in WO 5.4.x (or OS 10.5.x)
that could cause this?
Not off hand.
Chuck
On Aug 25, 2008, at 3:22 PM, Mike Schrag wrote:
No, they were built using Xcode and WO 5.3.3.
This is a really bad idea ... I would never deploy on a version
that's different than development. It's just asking for
problems. While it MAY work, you're just playing roulette with
your app. Either 1) replace 5.4 with 5.3 on that server, 2)
rebuild your app with WO 5.3 embedded, or 3) Setup a proper
5.4.2 build and development environment and rebuild and test
your app in 5.4.2.
ms
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com
)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/dandrews%40mediaspansoftware.com
This email sent to dandr...@mediaspansoftware.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com
)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/chill%40global-village.net
This email sent to ch...@global-village.net
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve
specific problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com
)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/chill%40global-village.net
This email sent to ch...@global-village.net
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
