On Mar 25, 2008, at 4:05 PM, Gary Teter wrote:
jstack for the win! Not sure how I forgot about that tool, but it
was able to get me good info.
Chalk one up for jstack!
I switched to ActiveMQ, which is a billion times better than
OpenJMS, but we're still having temporarily deaf instances. jstack
reported that Every. Single. Thread. was "state = BLOCKED", except
one:
Thread [EMAIL PROTECTED]: (state = IN_NATIVE)
- sun.nio.ch.KQueueArrayWrapper.kevent0(int, long, int, long) @bci=0
(Interpreted frame)
- sun.nio.ch.KQueueArrayWrapper.poll(long) @bci=12, line=118
(Interpreted frame)
- sun.nio.ch.KQueueSelectorImpl.doSelect(long) @bci=46, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled
frame)
- net.spy.memcached.MemcachedClient.run() @bci=11, line=1054
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at
sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at
sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:
218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
So it looks like I'm off to download a different memcached client.
On Mar 24, 2008, at 12:32 PM, Mike Schrag wrote:
That's what everyone says, but JGroups made this particular
problem worse. :(
Hmm .. I've been running the JGroups version for almost two years
in production and with some pretty substantial load tests (500-1000
insane-clicky-concurrent-making-changing users). I've never run it
on top of JMS, and admittedly, you lose persistent message delivery
with the non-JMS one, but for us this wasn't a problem.
But still, I don't think this is a deadlock because you come back,
and you don't come back from deadlocks. JGroups does love to
create threads, but they don't have many live concurrently,
generally (just lots created over time). Without being able to get
a thread dump, it will be pretty hard to diagnose this. If you're
running on 1.5 (I think you must be because I think JGroups is 1.5
only?) you might try running a jstack on your app when this
happens, though I would think it would have less success than a
kill -QUIT would ... Worth a shot, though.
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
This email sent to [EMAIL PROTECTED]
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to [EMAIL PROTECTED]
