Re: [Resin-interest] Deadlocks Part 2
I would look at the first thread dump and find the thread holding the locks that all the other threads were waiting on. If there were more than one, I will start with the lock on which the most threads are waiting on. Using the thread id, I would trace the life span of the thread holding the lock in the rest of the thread dumps to see how long it was holding the lock. If the same thread held the lock through out, there may be some external factor slowing it down. In this case it is doing file I/O so I would for things like file system running out of space or the file size getting too big. Since your problem may be related to log archiving, can you disable that feature to see if it makes any different. Very often I found that a complete different bottleneck may surface when you remove one that is the most obvious. Hopefully that is not your case. BillOn 10/14/06, sksamuel <[EMAIL PROTECTED]> wrote: >A thread dump is a snap shot of the JVM at a particular moment in time.> I find it useful to take consective thread dumps about 5 seconds apart.> That gives a moving picture of the JVM. You may want to vary the number > of dumps or the time apart.>> The thing to look for is how long are the threads waiting for a lock.> Lots of time I found things to be transient. (ie the first dump showing> many> threads waiting for lock(s) but in the dump 5 seconds later they are > done).>> You can use the thread id to following what is happening to a thread over> multiple thread dumps.Thanks for the reply Bill. I've done what you suggested and now have thread dumps spanning a minute. There are many threads still sat in the waiting formonitor stage, any advice on where I go next to try and figure out what thedeadlock is ?___ resin-interest mailing listresin-interest@caucho.comhttp://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Deadlocks part 3
On Oct 14, 2006, at 4:32 PM, sksamuel wrote: > The thread dumps are showing threads waiting on a lock on a > com.caucho.vfs.AbstractRolloverLog object. The thread that owns > this lock is > waiting for java.io.FileInputStream.readBytes to return, which is a > blocking > operation as it waits for input. Could something be stopping that > returning > and thus meaning the Resin thread is blocked waiting for it. Ok. That makes sense. The archiving is not currently efficient, but it would need to be fixed in the 3.1 tree since the needed changes are fairly large. -- Scott > > resin 3.0.21 jdk 1.5.0.08 > > "resin-1081" daemon prio=1 tid=0xac0b7160 nid=0x3879 runnable > [0xa7df8000..0xa7df8fb0] > at java.io.FileInputStream.readBytes(Native Method) > at java.io.FileInputStream.read(FileInputStream.java:194) > at com.caucho.vfs.FileReadStream.read(FileReadStream.java:143) > at com.caucho.vfs.Path.writeToStream(Path.java:1062) > at > com.caucho.vfs.AbstractRolloverLog.movePathToArchive > (AbstractRolloverLog.java:513) > at > com.caucho.vfs.AbstractRolloverLog.rolloverLog > (AbstractRolloverLog.java:385) > - locked <0x6a0f75e8> (a com.caucho.vfs.AbstractRolloverLog) > at > com.caucho.vfs.AbstractRolloverLog.rollover > (AbstractRolloverLog.java:335) > at com.caucho.vfs.RotateStream.handleAlarm(RotateStream.java:292) > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Deadlocks Part 2
>A thread dump is a snap shot of the JVM at a particular moment in time. > I find it useful to take consective thread dumps about 5 seconds apart. > That gives a moving picture of the JVM. You may want to vary the number > of dumps or the time apart. > > The thing to look for is how long are the threads waiting for a lock. > Lots of time I found things to be transient. (ie the first dump showing > many > threads waiting for lock(s) but in the dump 5 seconds later they are > done). > > You can use the thread id to following what is happening to a thread over > multiple thread dumps. Thanks for the reply Bill. I've done what you suggested and now have thread dumps spanning a minute. There are many threads still sat in the waiting for monitor stage, any advice on where I go next to try and figure out what the deadlock is ? ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Deadlocks Part 2
On Oct 13, 2006, at 7:26 AM, sksamuel wrote: > Further to my deadlocks issue, > > Is it normal to have lots of threads waiting for a lock on SSL > sockets? There should be about 5-10 threads waiting for the socket.It's normal. Those are just threads waiting for the next connection. -- Scott > I > know I don't have that many people trying to connect through SSL - > certainly > the number of threads waiting is disproportionate to the number of > visitors. > Example: > > > "resin-tcp-connection-127.0.0.1:6802-15" daemon prio=1 tid=0x095bbb80 > nid=0x7dbf waiting for monitor entry [0xa0c09000..0xa0c0a0b0] > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:382) > - waiting to lock <0x681975a0> (a java.net.SocksSocketImpl) > at java.net.ServerSocket.implAccept(ServerSocket.java:450) > at java.net.ServerSocket.accept(ServerSocket.java:421) > at com.caucho.vfs.QServerSocketWrapper.accept > (QServerSocketWrapper.java:97) > at com.caucho.server.port.Port.accept(Port.java:798) > at com.caucho.server.port.TcpConnection.run(TcpConnection.java:492) > at com.caucho.util.ThreadPool.runTasks(ThreadPool.java:516) > at com.caucho.util.ThreadPool.run(ThreadPool.java:442) > at java.lang.Thread.run(Thread.java:595) > > > > I also have a few threads waiting for a lock on ServletMapping - > again is > that normal? Example: > > > "resin-tcp-connection-*:80-85" daemon prio=1 tid=0xb1edba48 nid=0x13a5 > waiting for monitor entry [0xac862000..0xac862eb0] > at > com.caucho.server.dispatch.ServletConfigImpl.createServlet > (ServletConfigImpl.java:577) > - waiting to lock <0x6a01ad00> (a > com.caucho.server.dispatch.ServletMapping) > at > com.caucho.server.dispatch.ServletFilterChain.doFilter > (ServletFilterChain.java:97) > at org.global7.epublisher.iface.filters.RewriteFilter.doFilter > (Unknown > Source) > at > com.caucho.server.dispatch.FilterFilterChain.doFilter > (FilterFilterChain.java:70) > at > com.caucho.server.webapp.WebAppFilterChain.doFilter > (WebAppFilterChain.java:173) > at > com.caucho.server.dispatch.ServletInvocation.service > (ServletInvocation.java:229) > at com.caucho.server.http.HttpRequest.handleRequest > (HttpRequest.java:274) > at com.caucho.server.port.TcpConnection.run(TcpConnection.java:511) > - locked <0x6d819818> (a java.lang.Object) > at com.caucho.util.ThreadPool.runTasks(ThreadPool.java:516) > at com.caucho.util.ThreadPool.run(ThreadPool.java:442) > at java.lang.Thread.run(Thread.java:595) > > > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Deadlocks Part 2
A thread dump is a snap shot of the JVM at a particular moment in time. I find it useful to take consective thread dumps about 5 seconds apart. That gives a moving picture of the JVM. You may want to vary the number of dumps or the time apart. The thing to look for is how long are the threads waiting for a lock. Lots of time I found things to be transient. (ie the first dump showing many threads waiting for lock(s) but in the dump 5 seconds later they are done). You can use the thread id to following what is happening to a thread over multiple thread dumps. BillOn 10/13/06, sksamuel <[EMAIL PROTECTED]> wrote: Further to my deadlocks issue,Is it normal to have lots of threads waiting for a lock on SSL sockets? Iknow I don't have that many people trying to connect through SSL - certainlythe number of threads waiting is disproportionate to the number of visitors. Example:"resin-tcp-connection-127.0.0.1:6802-15" daemon prio=1 tid=0x095bbb80nid=0x7dbf waiting for monitor entry [0xa0c09000..0xa0c0a0b0] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java :382) - waiting to lock <0x681975a0> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at com.caucho.vfs.QServerSocketWrapper.accept (QServerSocketWrapper.java:97) at com.caucho.server.port.Port.accept(Port.java:798) at com.caucho.server.port.TcpConnection.run(TcpConnection.java:492) at com.caucho.util.ThreadPool.runTasks(ThreadPool.java:516) at com.caucho.util.ThreadPool.run(ThreadPool.java:442) at java.lang.Thread.run(Thread.java:595)I also have a few threads waiting for a lock on ServletMapping - again isthat normal? Example: "resin-tcp-connection-*:80-85" daemon prio=1 tid=0xb1edba48 nid=0x13a5waiting for monitor entry [0xac862000..0xac862eb0] atcom.caucho.server.dispatch.ServletConfigImpl.createServlet(ServletConfigImpl.java :577) - waiting to lock <0x6a01ad00> (acom.caucho.server.dispatch.ServletMapping) atcom.caucho.server.dispatch.ServletFilterChain.doFilter(ServletFilterChain.java:97) at org.global7.epublisher.iface.filters.RewriteFilter.doFilter (UnknownSource) atcom.caucho.server.dispatch.FilterFilterChain.doFilter(FilterFilterChain.java:70) atcom.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:173) atcom.caucho.server.dispatch.ServletInvocation.service (ServletInvocation.java:229) at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:274) at com.caucho.server.port.TcpConnection.run(TcpConnection.java:511) - locked <0x6d819818> (a java.lang.Object ) at com.caucho.util.ThreadPool.runTasks(ThreadPool.java:516) at com.caucho.util.ThreadPool.run(ThreadPool.java:442) at java.lang.Thread.run(Thread.java:595)___ resin-interest mailing listresin-interest@caucho.comhttp://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] deadlocks
On Oct 12, 2006, at 7:43 AM, sksamuel wrote: > I'm getting this deadlock on my resin servers. > > does anyone know if I disable logging completely in resin will that > stop > this happening or will all I do is just stop the outputting to the > file ? > > My servers are crashing daily (at least) so its getting to be a major > earbashing issue for me now. This is fixed in 3.0.19 and later. In earlier versions, the log rotation code was logging a message for the rotation, which could cause a deadlock. -- Scott > > > > Found one Java-level deadlock: > = > "resin-2140": > waiting to lock monitor 0x0975151c (object 0x834aff58, a > com.caucho.vfs.WriteStream), > which is held by "resin-tcp-connection-*:80-570" > "resin-tcp-connection-*:80-570": > waiting to lock monitor 0x094d1a8c (object 0x833d7c48, a > java.io.PrintStream), > which is held by "resin-2140" > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] deadlocks
On Oct 12, 2006, at 7:43 AM, sksamuel wrote: > I'm getting this deadlock on my resin servers. > > does anyone know if I disable logging completely in resin will that > stop > this happening or will all I do is just stop the outputting to the > file ? > > My servers are crashing daily (at least) so its getting to be a major > earbashing issue for me now. > > > > Found one Java-level deadlock: > = > "resin-2140": > waiting to lock monitor 0x0975151c (object 0x834aff58, a > com.caucho.vfs.WriteStream), > which is held by "resin-tcp-connection-*:80-570" > "resin-tcp-connection-*:80-570": > waiting to lock monitor 0x094d1a8c (object 0x833d7c48, a > java.io.PrintStream), > which is held by "resin-2140" Do you have thread dumps for both threads? It looks like it might be a log rotation issue, but it's a little hard to tell from that summary information. -- Scott > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] deadlocks
What version of Resin and Java are you using? We had extremely bad lockups with 3.0.12 Pro and Sun's 1.4.2_08. We upgraded to Resin Pro 3.0.19 and the 1.5.0_08 JRE and we have gone from locking up 2-3 times a day to only once every few weeks. We still have not been able to find the cause of the lockup and cannot even get a thread dump or attach a debugger when it does lockup. -Original Message- I'm getting this deadlock on my resin servers. does anyone know if I disable logging completely in resin will that stop this happening or will all I do is just stop the outputting to the file ? My servers are crashing daily (at least) so its getting to be a major earbashing issue for me now. Found one Java-level deadlock: = "resin-2140": waiting to lock monitor 0x0975151c (object 0x834aff58, a com.caucho.vfs.WriteStream), which is held by "resin-tcp-connection-*:80-570" "resin-tcp-connection-*:80-570": waiting to lock monitor 0x094d1a8c (object 0x833d7c48, a java.io.PrintStream), which is held by "resin-2140" ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Deadlocks
On Oct 6, 2006, at 11:10 AM, sksamuel wrote: > My resin installation is going into deadlock at least once a day. > > I have a thread dump that shows the TCP threads are waiting on for > a lock at > com.caucho.log.StreamHandler.publish(StreamHandler.java:112). > > Is that normal? Is that just Resin waiting to lock the log file for > output > and I can rule this out? Or is there something more serious going on? Which version? Also, try to find the thread that owns that lock and send it so we can see what it's doing. Deadlocks are important. -- Scott > > > > "resin-tcp-connection-*:80-39" daemon prio=1 tid=0x09e12430 nid=0x340e > waiting for monitor entry [0xa624a000..0xa624af30] > at com.caucho.log.StreamHandler.publish(StreamHandler.java:112) > - waiting to lock <0x67f80098> (a com.caucho.vfs.WriteStream) > at com.caucho.log.SubHandler.publish(SubHandler.java:73) > at java.util.logging.Logger.log(Logger.java:452) > at com.caucho.log.EnvironmentLogger.log(EnvironmentLogger.java:323) > at java.util.logging.Logger.doLog(Logger.java:474) > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
