Re: jvm cowardly refuses to print a thread dump
Martin Gainty wrote: so the solution is put all updates/inserts to the arraylist into a synchronized method? You must synchronize all read and write methods, because nothing may run in parallel with a write method, and so read methods must be prevented from executing if a write method runs already. The only exception is if you can ensure that the ArrayList is read-only at some point in your program (e. g., after a setup phase). Arguing that the size() function cannot loop does not help, because the size function could be called internally in an infinite loop (this would be visible in the call stack). Regards, Oliver Schoett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
On Fri, Dec 5, 2008 at 9:50 AM, Oliver Schoett [EMAIL PROTECTED] wrote: Martin Gainty wrote: so the solution is put all updates/inserts to the arraylist into a synchronized method? You must synchronize all read and write methods, because nothing may run in parallel with a write method, and so read methods must be prevented from executing if a write method runs already. The only exception is if you can ensure that the ArrayList is read-only at some point in your program (e. g., after a setup phase). Arguing that the size() function cannot loop does not help, because the size function could be called internally in an infinite loop (this would be visible in the call stack). Oliver, I'm very sorry but i really don't see your point. The only possible need to synchronize access to ArrayList.size is when you a) access the list from different threads and b) need the result to be exact. Neither was the fact in my original post, in fact, as we already resolved it, the vm was getting out of old gen space and afterwards just behaving weird. In 99% of the cases you don't need to synchronize calls to ArrayList, because usually you will use it in a local scope or single thread. And you definitely don't need to synchronize .size(), the worst thing that can happen is that the result is inaccurate, because another thread just removed or added something (except for reasons stated in first part of the mail). regards Leon - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
Leon Rosenberg wrote: On Fri, Dec 5, 2008 at 9:50 AM, Oliver Schoett [EMAIL PROTECTED] wrote: Martin Gainty wrote: so the solution is put all updates/inserts to the arraylist into a synchronized method? You must synchronize all read and write methods, because nothing may run in parallel with a write method, and so read methods must be prevented from executing if a write method runs already. The only exception is if you can ensure that the ArrayList is read-only at some point in your program (e. g., after a setup phase). Arguing that the size() function cannot loop does not help, because the size function could be called internally in an infinite loop (this would be visible in the call stack). Oliver, I'm very sorry but i really don't see your point. The only possible need to synchronize access to ArrayList.size is when you a) access the list from different threads and b) need the result to be exact. Neither was the fact in my original post, in fact, as we already resolved it, the vm was getting out of old gen space and afterwards just behaving weird. Three points: (1) In the absence of call stack information, we do not know whether the size() calls showing up in the thread dumps come from the application or are internal calls from some other ArrayList function that may be in an infinite loop. (2) That size() cannot loop may be a property of the current implementation; it is not guaranteed by the API. (3) In general, when you query the size, you then want to do something according to the value you got. Unless there is a synchronizaton block around the size call and the subsequent action that guarantees that there are no intervening changes to the ArrayList, the size call is useless, as the ArrayList may change arbitrarily between the size call and the action. Thus in general, even harmless query functions will appear inside synchronized sections together with the actions that use the result value. (The only exception I see is where the result value is just used for information and does not control any action). Regards, Oliver Schoett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
On Fri, Dec 5, 2008 at 12:37 PM, Oliver Schoett [EMAIL PROTECTED] wrote: Leon Rosenberg wrote: On Fri, Dec 5, 2008 at 9:50 AM, Oliver Schoett [EMAIL PROTECTED] wrote: Martin Gainty wrote: so the solution is put all updates/inserts to the arraylist into a synchronized method? You must synchronize all read and write methods, because nothing may run in parallel with a write method, and so read methods must be prevented from executing if a write method runs already. The only exception is if you can ensure that the ArrayList is read-only at some point in your program (e. g., after a setup phase). Arguing that the size() function cannot loop does not help, because the size function could be called internally in an infinite loop (this would be visible in the call stack). Oliver, I'm very sorry but i really don't see your point. The only possible need to synchronize access to ArrayList.size is when you a) access the list from different threads and b) need the result to be exact. Neither was the fact in my original post, in fact, as we already resolved it, the vm was getting out of old gen space and afterwards just behaving weird. Three points: (1) In the absence of call stack information, we do not know whether the size() calls showing up in the thread dumps come from the application or are internal calls from some other ArrayList function that may be in an infinite loop. The stack trace was sent to people who wanted it. (2) That size() cannot loop may be a property of the current implementation; it is not guaranteed by the API. Actually it is. * The ttsize/tt, ttisEmpty/tt, ttget/tt, ttset/tt, * ttiterator/tt, and ttlistIterator/tt operations run in constant * time. The ttadd/tt operation runs in iamortized constant time/i, * that is, adding n elements requires O(n) time. All of the other operations * run in linear time (roughly speaking). The constant factor is low compared * to that for the ttLinkedList/tt implementation.p (3) In general, when you query the size, you then want to do something according to the value you got. Unless there is a synchronizaton block around the size call and the subsequent action that guarantees that there are no intervening changes to the ArrayList, the size call is useless, as the ArrayList may change arbitrarily between the size call and the action. Thus in general, even harmless query functions will appear inside synchronized sections together with the actions that use the result value. (The only exception I see is where the result value is just used for information and does not control any action). You should consider using synchronized lists :-) (for example Vector). regards Leon - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
Leon Rosenberg wrote: You should consider using synchronized lists :-) (for example Vector). Does not help :-) If you do not put sync. brackets around Check some property (e. g., size()); Do something according to the property value you lose, because the property may have changed already when the second line is executed. You need the sync. brackets to preserve the state between checking some property and acting on the result. Regards, Oliver Schoett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
On Fri, Dec 5, 2008 at 3:45 PM, Oliver Schoett [EMAIL PROTECTED] wrote: Leon Rosenberg wrote: You should consider using synchronized lists :-) (for example Vector). Does not help :-) If you do not put sync. brackets around Check some property (e. g., size()); Do something according to the property value you lose, because the property may have changed already when the second line is executed. Hello Oliver, yes, in this very special case, which has nothing to do with the original post of this thread you are absolutely right. regards Leon - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: jvm cowardly refuses to print a thread dump
if its a bean make sure you
declare your attributes as having either private or protected access
in the case of attribute called size contained in class FuBar
class FuBar
{
public FuBar() { ; } //default no-arg constructor
private int Size;
//declare accessor getSize
public getSize()
{
return Size;
}
//declare mutator setSize
public setSize(int Size)
{
this.Size = Size;
}
} //end class definition
HTH
Martin
__
Disclaimer and confidentiality note
Everything in this e-mail and any attachments relates to the official business
of Sender. This transmission is of a confidential nature and Sender does not
endorse distribution to any party other than intended recipient. Sender does
not necessarily endorse content contained within this transmission.
From: [EMAIL PROTECTED]
To: users@tomcat.apache.org
Date: Tue, 2 Dec 2008 08:47:05 -0600
Subject: RE: jvm cowardly refuses to print a thread dump
From: news [mailto:[EMAIL PROTECTED] On Behalf Of Oliver Schoett
Subject: Re: jvm cowardly refuses to print a thread dump
Very likely you have unsynchronized accesses to the ArrayList
That's irrelevant for the size() method, where the entire code consists of
this:
public int size() {
return size;
}
Not much chance for a loop there...
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you received
this in error, please contact the sender and delete the e-mail and its
attachments from all computers.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
_
You live life online. So we put Windows on the web.
http://clk.atdmt.com/MRT/go/127032869/direct/01/
Re: jvm cowardly refuses to print a thread dump
Martin Gainty wrote:
if its a bean make sure you
declare your attributes as having either private or protected access
in the case of attribute called size contained in class FuBar
class FuBar
{
public FuBar() { ; } //default no-arg constructor
private int Size;
//declare accessor getSize
public getSize()
{
return Size;
}
//declare mutator setSize
public setSize(int Size)
{
this.Size = Size;
}
} //end class definition
HTH
Martin
You probably also want to declare Size as volatile if it's going to be
accessed by multiple threads and isn't enclosed in a synchronized
method or block. You may also want to declare the argument to the setter
as final, your variable naming convention conflicts with the Sun one,
and the default constructor is redundant. I'll remove my pedant hat now.
Chris
__
Disclaimer and confidentiality note
Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission.
From: [EMAIL PROTECTED]
To: users@tomcat.apache.org
Date: Tue, 2 Dec 2008 08:47:05 -0600
Subject: RE: jvm cowardly refuses to print a thread dump
From: news [mailto:[EMAIL PROTECTED] On Behalf Of Oliver Schoett
Subject: Re: jvm cowardly refuses to print a thread dump
Very likely you have unsynchronized accesses to the ArrayList
That's irrelevant for the size() method, where the entire code consists of this:
public int size() {
return size;
}
Not much chance for a loop there...
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you received
this in error, please contact the sender and delete the e-mail and its
attachments from all computers.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
_
You live life online. So we put Windows on the web.
http://clk.atdmt.com/MRT/go/127032869/direct/01/
--
Chris Wareham
Senior Software Engineer
Visit London Ltd
6th floor,
2 More London Riverside, London SE1 2RR
Tel: +44 (0)20 7234 5848
Fax: +44 (0)20 7234 5753
www.visitlondon.com
'Visit London Limited' is registered in England under No.761149;
Registered Office: Visit London, 2 More London Riverside, London SE1 2RR.
Visit London is the official visitor organisation for London. Visit London is
partly funded by Partnership, the Mayor's London Development Agency and London
Councils.
The information contained in this e-mail is confidential and intended for the
named recipient(s) only. If you have received it in error, please notify the
sender immediately and then delete the message. If you are not the intended
recipient, you must not use, disclose, copy or distribute this email. The views
expressed in this e-mail are those of the individual and not of Visit London.
We reserve the right to read and monitor any email or attachment entering or
leaving our systems without prior notice.
Please don't print this e-mail unless you really need to.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
Leon Rosenberg wrote: But back to the ArrayList, how high is the probability of having ArrayList.size() twice in the similar call-tree in one ThreadDump? Very likely you have unsynchronized accesses to the ArrayList, which from time to time send one of the threads involved into an infinite tight loop inside an ArrayList access function. A way to confirm this is to take two thread dumps a minute apart; if you find the *same* threads in the same ArrayList access functions, they loop infinitely. Regards, Oliver Schoett - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: jvm cowardly refuses to print a thread dump
so the solution is put all updates/inserts to the arraylist into a synchronized method?Martin __ Disclaimer and confidentiality note Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission. To: users@tomcat.apache.org From: [EMAIL PROTECTED] Subject: Re: jvm cowardly refuses to print a thread dump Date: Tue, 2 Dec 2008 14:35:33 +0100 Leon Rosenberg wrote: But back to the ArrayList, how high is the probability of having ArrayList.size() twice in the similar call-tree in one ThreadDump?Very likely you have unsynchronized accesses to the ArrayList, which from time to time send one of the threads involved into an infinite tight loop inside an ArrayList access function. A way to confirm this is to take two thread dumps a minute apart; if you find the *same* threads in the same ArrayList access functions, they loop infinitely. Regards, Oliver Schoett - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] _ Send e-mail anywhere. No map, no compass. http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008
RE: jvm cowardly refuses to print a thread dump
From: news [mailto:[EMAIL PROTECTED] On Behalf Of Oliver Schoett
Subject: Re: jvm cowardly refuses to print a thread dump
Very likely you have unsynchronized accesses to the ArrayList
That's irrelevant for the size() method, where the entire code consists of this:
public int size() {
return size;
}
Not much chance for a loop there...
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you received
this in error, please contact the sender and delete the e-mail and its
attachments from all computers.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: jvm cowardly refuses to print a thread dump
From: Leon Rosenberg [mailto:[EMAIL PROTECTED] Subject: jvm cowardly refuses to print a thread dump we were unable to create a thread dump (kill -QUIT pid was just ignored by the jvm). That's ugly. Sounds like either the OS failed to deliver the signal, or the JVM is locked up internally (probably the latter). I presume you've looked at the logs and found nothing of interest. Can you launch JConsole on the same machine to poke around in the Tomcat JVM? (My suspicion is no, if a signal doesn't penetrate.) It wouldn't impact performance much if you brought up a JConsole window when you start Tomcat and just let it sit until the hang. Any indication of high heap usage before the hang? Does running with -verbose:gc show anything? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: jvm cowardly refuses to print a thread dump
From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] That's ugly. Sounds like either the OS failed to deliver the signal, or the JVM is locked up internally (probably the latter). That's rather what I was reckoning. I've had signal delivery fail before if all the threads were stuck in kernel code via system calls. This was SunOS 3.x, which shows my age - NFS was kernel-mode at that point, which made for interesting times if file accesses got stuck. Processes not responding to a kill -9 was a new one on me! Might the code have ended up with all threads stuck in OS calls? It feels a little unlikely, as I assume the JVM keeps a couple of threads for itself... - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
its very weird indeed. We had acted proactively on a tomcat which
started to show same symptoms (cpu load going higher (35%) but users
going down (means it replies slower than the other machines and the lb
gives less traffic to it) and after checking the logs, sent the QUIT
signal which worked this time (hence i can't be sure, that its
completely the same problem).
But the thread dump was weird, from
51 RUNNABLE threads
38 in at java.net.SocketInputStream.socketRead0(Native Method)
2 in write to file in logging
...
and two in
at java.util.ArrayList.size(ArrayList.java:177)
Now this is something i personally consider weird, since it's hard
enough to find a thread in any places in working (not waiting) code in
a thread dump, but twice in the same code line?
The calling code line (our code) is
public boolean isRoot() {
return pathElements.size() == 0;
}
We assume that the other tomcats didn't produce any thread dumps
because our trigger happy system admins just sent the kill -9 too soon
after the kill -3, and the jvm was overloaded. Well it's a wild guess,
but i have no better one yet (unless jvm really locked up internally,
but why??? we haven't change the jvm or tomcat or os or anything for a
year now).
But back to the ArrayList, how high is the probability of having
ArrayList.size() twice in the similar call-tree in one ThreadDump?
regards
Leon
On Mon, Dec 1, 2008 at 5:41 PM, Peter Crowther
[EMAIL PROTECTED] wrote:
From: Caldarale, Charles R [mailto:[EMAIL PROTECTED]
That's ugly. Sounds like either the OS failed to deliver the
signal, or the JVM is locked up internally (probably the
latter).
That's rather what I was reckoning. I've had signal delivery fail before if
all the threads were stuck in kernel code via system calls. This was SunOS
3.x, which shows my age - NFS was kernel-mode at that point, which made for
interesting times if file accesses got stuck. Processes not responding to a
kill -9 was a new one on me!
Might the code have ended up with all threads stuck in OS calls? It feels a
little unlikely, as I assume the JVM keeps a couple of threads for itself...
- Peter
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: jvm cowardly refuses to print a thread dump
Leon Rosenberg schrieb: We assume that the other tomcats didn't produce any thread dumps because our trigger happy system admins just sent the kill -9 too soon after the kill -3, and the jvm was overloaded. Well it's a wild guess, but i have no better one yet (unless jvm really locked up internally, but why??? we haven't change the jvm or tomcat or os or anything for a year now). Just in case processes will not respond on kill -QUIT again: you could try strace and gstack. If the process still does issue system calls, you can get those with strace (and maybe get an idea what it is doing), if it is hanging somewhere in native code, you could look at it with gstack. If it is hanging in Java code, neither strace nor gstack will help. But back to the ArrayList, how high is the probability of having ArrayList.size() twice in the similar call-tree in one ThreadDump? Tell us the whole stack, not just the top line. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: jvm cowardly refuses to print a thread dump
From: Leon Rosenberg [mailto:[EMAIL PROTECTED] Subject: Re: jvm cowardly refuses to print a thread dump and two in at java.util.ArrayList.size(ArrayList.java:177) Just for grins, can you post the full stack trace for those two? (Or e-mail me the whole thing.) There's nothing unusual at that line of code - it's not native or even synchronized. One possibility is that the JIT is trying to optimize use of that method by recompiling it, but the Compiler thread is stuck someplace, and the app threads are suspended until the JIT finishes. Have you changed JVM levels recently? If you haven't, maybe you should... We assume that the other tomcats didn't produce any thread dumps because our trigger happy system admins just sent the kill -9 too soon after the kill -3, and the jvm was overloaded. I'm not buying it. Does this happen on more than one system? Have you had a cosmic ray storm in your area recently? (Once upon a time, we had a system that crashed every day at 16:50 for about two weeks, but never at any other time. After the two weeks, the crashes stopped; we suspected power fluctuations, but we never did figure out exactly what was going on.) But back to the ArrayList, how high is the probability of having ArrayList.size() twice in the similar call-tree in one ThreadDump? Unless your code is really, really paranoid and does a lot of unnecessary size checking, I'd guess pretty close to zero. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: jvm cowardly refuses to print a thread dump
From: Peter Crowther [mailto:[EMAIL PROTECTED] Subject: RE: jvm cowardly refuses to print a thread dump Might the code have ended up with all threads stuck in OS calls? It feels a little unlikely, as I assume the JVM keeps a couple of threads for itself... Unlikely that everybody's off in native code. The JVM uses quite a few threads internally, only some of which show up in a JVM-produced thread dump. There is a Signal Dispatcher thread, so if that one got stuck someplace, I would expect signals to not get processed in a timely fashion. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
