RE: CLOSE_WAIT and what to do about it
From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: CLOSE_WAIT and what to do about it If these sockets disappear during a GC, then it must mean that they are still being referenced by some abandoned objects sitting on the Heap, which have not yet been reclaimed by the GC. Which probably means that the objects in question have gone out of scope, before the socket they used was properly close()'d. Your analysis looks reasonable to me. There are some analysis tools that will examine a live heap (or dump thereof) and find the reachable and unreachable objects; jhat is a free one that comes with JDK 6: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/tooldescr.html#gblfj - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CLOSE_WAIT and what to do about it
Caldarale, Charles R wrote: From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: CLOSE_WAIT and what to do about it Relatedly, does there exist any way to force a given JVM process to do a full GC interactively, but from a Linux command-line ? Found a command line tool that will do what you want: http://code.google.com/p/jmxsh/ I've used it to trigger a GC in Tomcat via the following steps. 1) Start Tomcat with the following options: -Dcom.sun.management.jmxremote.port=port -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false (You can, of course, set the authentication and SSL options as needed.) 2) Start jmxsh from the directory its jar is in with this: java -jar jmxsh*.jar 3) Enter the following commands (but not the bracketed bits): jmx_connect -h localhost -p port [blank line to enter browse mode] 5 [selects java.lang] 1 [selects the Memory mbean] 5 [performs a GC] The doc for jmxsh indicates the above steps should be scriptable, but I haven't tried that. It is likely that you could use jmx_connect with a different kind of service and avoid opening up an RMI port; if I figure that out, I'll let you know. Hi. Thanks a million for providing the above info. That jmxsh program is really useful. I don't really know what I'm doing here, but I can at least more or less figure out what happens. To recall, my original issue is that I have some Java applications (among which a Tomcat webapp and a couple of stand-alone Java daemon-like programs) which apparently leave an ever-increasing number of sockets lingering in a CLOSE_WAIT state. And I was wondering if it was possible, as one test, to force the JVM running these applications to perform a GC, right now, from the outside. Well, it is. Following is a trace of a session with jmxsh, with one of these applications. Initial socket situation : r...@arthur:/home/star/xml# netstat -pan | grep CLOSE tcp6 0 0 :::127.0.0.1:48267 :::127.0.0.1:11002 CLOSE_WAIT 7618/java tcp6 12 0 :::127.0.0.1:36936 :::127.0.0.1:11002 CLOSE_WAIT 7816/java tcp6 12 0 :::127.0.0.1:50322 :::127.0.0.1:11002 CLOSE_WAIT 7816/java r...@arthur:/home/star/xml# ps -ef | grep 7618 root 7618 1 1 14:32 pts/300:00:15 ./java -server -Dcom.sun.management.jmxremote.port=11201 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xms64M -Xmx64M -Dpgm=STARWeb -jar /home//web4/java/xyz.jar -c /home/star/web4/config -p 11101 The above is the process which I am going to stress, in the sense of communicating with it, which has the result of having it itself open a TCP connection with another server listening on port 11002, then closing this socket (in principle), and this multiple times. (As you see, the program was started with the jmxremote options allowing later communication with jmxsh.) Now some interactions with the application pid=7618 ... Situation later on : r...@arthur:/home/star/xml# netstat -pan | grep CLOSE tcp6 0 0 :::127.0.0.1:55798 :::127.0.0.1:11002 CLOSE_WAIT 7618/java tcp6 0 0 :::127.0.0.1:57029 :::127.0.0.1:11002 CLOSE_WAIT 7618/java tcp6 0 0 :::127.0.0.1:48267 :::127.0.0.1:11002 CLOSE_WAIT 7618/java tcp6 0 0 :::127.0.0.1:56781 :::127.0.0.1:11002 CLOSE_WAIT 7618/java tcp6 12 0 :::127.0.0.1:36936 :::127.0.0.1:11002 CLOSE_WAIT 7816/java tcp6 12 0 :::127.0.0.1:58341 :::127.0.0.1:11002 CLOSE_WAIT 7816/java tcp6 0 0 :::127.0.0.1:32972 :::127.0.0.1:11002 CLOSE_WAIT 7618/java tcp6 12 0 :::127.0.0.1:50322 :::127.0.0.1:11002 CLOSE_WAIT 7816/java So this application indeed left a number of sockets in the CLOSE_WAIT state. Now triggering a GC with jmxsh : a...@arthur:~$ java -jar bin/jmxsh-R4.jar jmxsh v1.0, Tue Jan 22 17:23:12 GMT+01:00 2008 Type 'help' for help. Give the option '-?' to any command for usage help. Starting up in shell mode. % jmx_connect -h localhost -p 11201 Connected to service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi. % Entering browse mode. Available Domains: 1. java.util.logging 2. JMImplementation 3. java.lang SERVER: service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi Select a domain: 3 Available MBeans: 1. java.lang:type=Compilation 2. java.lang:type=MemoryManager,name=CodeCacheManager 3. java.lang:type=GarbageCollector,name=Copy 4. java.lang:type=MemoryPool,name=Eden Space 5. java.lang:type=Runtime 6. java.lang:type=ClassLoading 7. java.lang:type=MemoryPool,name=Survivor Space 8.
Re: CLOSE_WAIT and what to do about it
Caldarale, Charles R wrote:
From: André Warnier [mailto:a...@ice-sa.com]
Subject: Re: CLOSE_WAIT and what to do about it
If these sockets disappear during a GC, then it must mean that they are
still being referenced by some abandoned objects sitting on the Heap,
which have not yet been reclaimed by the GC.
Which probably means that the objects in question have gone out of
scope, before the socket they used was properly close()'d.
Your analysis looks reasonable to me. There are some analysis tools that will
examine a live heap (or dump thereof) and find the reachable and unreachable
objects; jhat is a free one that comes with JDK 6:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/tooldescr.html#gblfj
Allright, I have done that too.
I generated a Heap dump using
jmap -heap:format=b pid
That gave me file heap.bin of some 4.5 MB.
I then used the jhat program to open it.
jhat launches itself by default as a webserver on port 7000, which you
can access using a normal browser.
That's where my problem starts however, because being a mere Java
fiddler I don't really know what I am looking at, and what to look for.
I did a lot of guesswork anyway, and using my knowledge of the
application more than the links, I came upon the name of a class that
looks like it is reponsible for opening/closing the sockets that remain
in CLOSE_WAIT.
I found the following function in the class :
public void close()
throws SomeException
{
putEndRequest();
flush();
socket = null;
}
flush() being another function which reads the socket until there's
nothing left to read, and throws away the result.
socket is a property of the object created by this class, obtained
somewhere else from a java.net.Socket object.
Looking at that code above, it is obvious that socket is open, until
it is set to null, without previously doing a socket.close().
I don't know Java enough to know if this alone could cause that socket
to be lingering until the GC, but I kind of suspect so.
How does a Java expert look at that ?
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Can anyone explain why a jar file can't be removed during a updeployment process?
http://wiki.apache.org/tomcat/FAQ/Windows#Q5
RE: CLOSE_WAIT and what to do about it
From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: CLOSE_WAIT and what to do about it Looking at that code above, it is obvious that socket is open, until it is set to null, without previously doing a socket.close(). I don't know Java enough to know if this alone could cause that socket to be lingering until the GC, but I kind of suspect so. For not being that familiar with Java, you've done an admirable job of tracking this down. What you've found certainly looks like the cause of the problem; the class you encountered appears to be a wrapper for a plain java.net.Socket, and whoever wrote it simply missed putting in a socket.close() call. Perhaps this was originally developed on an older JVM with more frequent non-generational garbage collection, so the problem wasn't noticed then. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CLOSE_WAIT and what to do about it
Caldarale, Charles R wrote: From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: CLOSE_WAIT and what to do about it Looking at that code above, it is obvious that socket is open, until it is set to null, without previously doing a socket.close(). I don't know Java enough to know if this alone could cause that socket to be lingering until the GC, but I kind of suspect so. For not being that familiar with Java, you've done an admirable job of tracking this down. What you've found certainly looks like the cause of the problem; the class you encountered appears to be a wrapper for a plain java.net.Socket, and whoever wrote it simply missed putting in a socket.close() call. Perhaps this was originally developed on an older JVM with more frequent non-generational garbage collection, so the problem wasn't noticed then. I was standing on the shoulders of giants. Thanks for the help. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
How to tell how much memory each web application is using?
I'm very new to Tomcat and the Java world in general. I have Tomcat setup and am running several java web apps on it (Adobe ColdFusion, Railo, and BlueDragon). I'd like to see how much memory each of these applications is using. Is that possible? I've installed Lambda Probe, and it gives me a lot of interesting information, including how much memory Tomcat is using inside the heap, etc. But I don't see anywhere where it breaks down how much each application is using. Is this possible? Thanks. -- View this message in context: http://www.nabble.com/How-to-tell-how-much-memory-each-web-application-is-using--tp23017029p23017029.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: How to tell how much memory each web application is using?
From: zirconx [mailto:r...@stillnet.org] Subject: How to tell how much memory each web application is using? I'd like to see how much memory each of these applications is using. Is that possible? Not easily. There's just the one heap for the JVM, and it's shared by all webapps and Tomcat internal operations. In theory, you could use jhat to go through a heap dump and find out all the objects that are reachable from another object, but it would be quite tedious. The easier approach is to deploy only one application, run whatever test you want, then measure heap usage. Repeat for each webapp. Also measure the usage with no apps (or just the manager) to get a baseline. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
