Re: [jetty-discuss] Re: [JBoss-dev] Possible Jetty race condition?Or threading mystery? Help!!
Already in CVS. cheers PS. don't blow away Marc's jasper hack in JBoss Jules Gosnell wrote: Guys, I reproduced the problem on JBoss-3.0.1 easily, added Greg's fixes to the Jetty component and could not then reproduce the Exception. Greg, If you stick this into Mort Bay Jetty cvs (or shall I?), I will do a merge into the necessary JBoss branches. Jules Greg Wilkins wrote: David, thanks for the detailed analysis of this.I had seen this once before but had never been able to reproduce it. I think I know what is going on It is not strickly a race rather the thread is being returned to the pool with interrupted status true - this is effecting it being used for other purposes. I think the fix is two fold - although I can't test this as I still cant reproduce it. Firstly I need to stop interrupting myself. Close does an interupt to so that the HttpServer.stop method does not hang for those JVMs where closing a socket is not enough to wakeup a blocked thread. So I think close should be: if (_handlingThread!=null Thread.currentThread!=_handlingThread) _handlingThread.interrupt(); Secondly, threads returned to the ThreadPool need to have their interrupted status cleared by calling Thread.interrupted.For now I have done this in the ThreadPool.JobRunner.run method: while(_running) { // clear interrupts Thread.interrupted(); Can you make these changes and see if that fixes the problem? thanks David Jencks wrote: I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Here's some annotated trace info (some from modifying Thread to tell us what is going on). This is all about thread 4ac866. It appears to being used by 2 HttpConnections more or less simultaneously (??) --we start by HttpConnection recording it's thread. 2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting _handlingThread to Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --later, it is done with this thread so it interrupts it. Note we are using HttpConnection 462080 2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection] interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --Here's Thread telling us all about the interrupt call 2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called-- on thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] 2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace from interrupt 2002-08-12 10:17:08,836 ERROR [STDERR] at java.lang.Thread.interrupt(Thread.java:665) 2002-08-12 10:17:08,836 ERROR [STDERR] at org.mortbay.http.HttpConnection.close(HttpConnection.java:248) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.HttpConnection.handle(HttpConnection.java:808) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322) 2002-08-12 10:17:08,839 ERROR [STDERR] at org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713) 2002-08-12 10:17:08,840 ERROR [STDERR] at java.lang.Thread.run(Thread.java:479) --The interrupt completed successfully, we are still in HttpConnection 462080 2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner:
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
These two changes have made my example work without the interrupts. What's the procedure for changing Jetty? Should I check these in? Jules? Wait for a new Jetty? thanks david jencks On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote: David, thanks for the detailed analysis of this.I had seen this once before but had never been able to reproduce it. I think I know what is going on It is not strickly a race rather the thread is being returned to the pool with interrupted status true - this is effecting it being used for other purposes. I think the fix is two fold - although I can't test this as I still cant reproduce it. Firstly I need to stop interrupting myself. Close does an interupt to so that the HttpServer.stop method does not hang for those JVMs where closing a socket is not enough to wakeup a blocked thread. So I think close should be: if (_handlingThread!=null Thread.currentThread!=_handlingThread) _handlingThread.interrupt(); Secondly, threads returned to the ThreadPool need to have their interrupted status cleared by calling Thread.interrupted.For now I have done this in the ThreadPool.JobRunner.run method: while(_running) { // clear interrupts Thread.interrupted(); Can you make these changes and see if that fixes the problem? thanks David Jencks wrote: I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Here's some annotated trace info (some from modifying Thread to tell us what is going on). This is all about thread 4ac866. It appears to being used by 2 HttpConnections more or less simultaneously (??) --we start by HttpConnection recording it's thread. 2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting _handlingThread to Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --later, it is done with this thread so it interrupts it. Note we are using HttpConnection 462080 2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection] interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --Here's Thread telling us all about the interrupt call 2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called-- on thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] 2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace from interrupt 2002-08-12 10:17:08,836 ERROR [STDERR] at java.lang.Thread.interrupt(Thread.java:665) 2002-08-12 10:17:08,836 ERROR [STDERR] at org.mortbay.http.HttpConnection.close(HttpConnection.java:248) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.HttpConnection.handle(HttpConnection.java:808) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322) 2002-08-12 10:17:08,839 ERROR [STDERR] at org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713) 2002-08-12 10:17:08,840 ERROR [STDERR] at java.lang.Thread.run(Thread.java:479) --The interrupt completed successfully, we are still in HttpConnection 462080 2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread
RE: [jetty-discuss] Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Greg, Already in CVS. cheers PS. don't blow away Marc's jasper hack in JBoss You wrote it :) Hey the other day I was at this Sabre division and one of the things they were asking about is precompiled JSP, I didn't know you had commited the fix for live deployment, it is really useful. Is there any way jasper integrates this. marcf --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [jetty-discuss] Re: [JBoss-dev] Possible Jetty race condition?Or threading mystery? Help!!
I meant the hack I did for Marc. I have raised a bug against the jasper people for it - but I have seen no action from them yet.In fact they have 290+ open bugs against tomcat/jasper and not much action on any of them! The hack is in the jboss jetty tree, but not in the main jetty tree. Don't know why really - just hedging bets I guess. cheers marc fleury wrote: Greg, Already in CVS. cheers PS. don't blow away Marc's jasper hack in JBoss You wrote it :) Hey the other day I was at this Sabre division and one of the things they were asking about is precompiled JSP, I didn't know you had commited the fix for live deployment, it is really useful. Is there any way jasper integrates this. marcf Yahoo! Groups Sponsor -~-- 4 DVDs Free +sp Join Now http://us.click.yahoo.com/pt6YBB/NXiEAA/RN.GAA/CefplB/TM -~- For the latest information about Jetty, please see http://jetty.mortbay.org To alter your subscription to this list goto http://groups.yahoo.com/group/jetty-discuss Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ -- Greg Wilkins[EMAIL PROTECTED] GB Phone: +44-(0)7092063462 Mort Bay Consulting Australia and UK.Mbl Phone: +61-(0)4 17786631 http://www.mortbay.com AU Phone: +61-(0)2 98107029 --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
Which Branch[es] do you think that I should update? I'm getting confused about how many there are. I figure : HEAD Branch_3_2 JBoss_3_0_1 any more ? Let me know and I will merge an up-to-date Jetty into all of them today. Jules David Jencks wrote: These two changes have made my example work without the interrupts. What's the procedure for changing Jetty? Should I check these in? Jules? Wait for a new Jetty? thanks david jencks On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote: David, thanks for the detailed analysis of this.I had seen this once before but had never been able to reproduce it. I think I know what is going on It is not strickly a race rather the thread is being returned to the pool with interrupted status true - this is effecting it being used for other purposes. I think the fix is two fold - although I can't test this as I still cant reproduce it. Firstly I need to stop interrupting myself. Close does an interupt to so that the HttpServer.stop method does not hang for those JVMs where closing a socket is not enough to wakeup a blocked thread. So I think close should be: if (_handlingThread!=null Thread.currentThread!=_handlingThread) _handlingThread.interrupt(); Secondly, threads returned to the ThreadPool need to have their interrupted status cleared by calling Thread.interrupted.For now I have done this in the ThreadPool.JobRunner.run method: while(_running) { // clear interrupts Thread.interrupted(); Can you make these changes and see if that fixes the problem? thanks David Jencks wrote: I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Here's some annotated trace info (some from modifying Thread to tell us what is going on). This is all about thread 4ac866. It appears to being used by 2 HttpConnections more or less simultaneously (??) --we start by HttpConnection recording it's thread. 2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting _handlingThread to Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --later, it is done with this thread so it interrupts it. Note we are using HttpConnection 462080 2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection] interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --Here's Thread telling us all about the interrupt call 2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called-- on thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] 2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace from interrupt 2002-08-12 10:17:08,836 ERROR [STDERR] at java.lang.Thread.interrupt(Thread.java:665) 2002-08-12 10:17:08,836 ERROR [STDERR] at org.mortbay.http.HttpConnection.close(HttpConnection.java:248) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.HttpConnection.handle(HttpConnection.java:808) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322) 2002-08-12 10:17:08,839 ERROR [STDERR] at org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713) 2002-08-12 10:17:08,840 ERROR [STDERR] at java.lang.Thread.run(Thread.java:479) --The interrupt completed successfully, we are still in HttpConnection 462080 2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
I've pretty much given up on working on anything but head. I think your list of branches is complete. david On 2002.08.14 10:36:11 -0400 Jules Gosnell wrote: Which Branch[es] do you think that I should update? I'm getting confused about how many there are. I figure : HEAD Branch_3_2 JBoss_3_0_1 any more ? Let me know and I will merge an up-to-date Jetty into all of them today. Jules David Jencks wrote: These two changes have made my example work without the interrupts. What's the procedure for changing Jetty? Should I check these in? Jules? Wait for a new Jetty? thanks david jencks On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote: David, thanks for the detailed analysis of this.I had seen this once before but had never been able to reproduce it. I think I know what is going on It is not strickly a race rather the thread is being returned to the pool with interrupted status true - this is effecting it being used for other purposes. I think the fix is two fold - although I can't test this as I still cant reproduce it. Firstly I need to stop interrupting myself. Close does an interupt to so that the HttpServer.stop method does not hang for those JVMs where closing a socket is not enough to wakeup a blocked thread. So I think close should be: if (_handlingThread!=null Thread.currentThread!=_handlingThread) _handlingThread.interrupt(); Secondly, threads returned to the ThreadPool need to have their interrupted status cleared by calling Thread.interrupted.For now I have done this in the ThreadPool.JobRunner.run method: while(_running) { // clear interrupts Thread.interrupted(); Can you make these changes and see if that fixes the problem? thanks David Jencks wrote: I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Here's some annotated trace info (some from modifying Thread to tell us what is going on). This is all about thread 4ac866. It appears to being used by 2 HttpConnections more or less simultaneously (??) --we start by HttpConnection recording it's thread. 2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting _handlingThread to Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --later, it is done with this thread so it interrupts it. Note we are using HttpConnection 462080 2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection] interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --Here's Thread telling us all about the interrupt call 2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called-- on thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] 2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace from interrupt 2002-08-12 10:17:08,836 ERROR [STDERR] at java.lang.Thread.interrupt(Thread.java:665) 2002-08-12 10:17:08,836 ERROR [STDERR] at org.mortbay.http.HttpConnection.close(HttpConnection.java:248) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.HttpConnection.handle(HttpConnection.java:808) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322) 2002-08-12 10:17:08,839 ERROR [STDERR] at org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713) 2002-08-12 10:17:08,840 ERROR [STDERR] at
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
I'm committing the latest Jetty onto HEAD as we speak.It includes the patch. Jules David Jencks wrote: I've pretty much given up on working on anything but head. I think your list of branches is complete. david On 2002.08.14 10:36:11 -0400 Jules Gosnell wrote: Which Branch[es] do you think that I should update? I'm getting confused about how many there are. I figure : HEAD Branch_3_2 JBoss_3_0_1 any more ? Let me know and I will merge an up-to-date Jetty into all of them today. Jules David Jencks wrote: These two changes have made my example work without the interrupts. What's the procedure for changing Jetty? Should I check these in? Jules? Wait for a new Jetty? thanks david jencks On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote: David, thanks for the detailed analysis of this.I had seen this once before but had never been able to reproduce it. I think I know what is going on It is not strickly a race rather the thread is being returned to the pool with interrupted status true - this is effecting it being used for other purposes. I think the fix is two fold - although I can't test this as I still cant reproduce it. Firstly I need to stop interrupting myself. Close does an interupt to so that the HttpServer.stop method does not hang for those JVMs where closing a socket is not enough to wakeup a blocked thread. So I think close should be: if (_handlingThread!=null Thread.currentThread!=_handlingThread) _handlingThread.interrupt(); Secondly, threads returned to the ThreadPool need to have their interrupted status cleared by calling Thread.interrupted.For now I have done this in the ThreadPool.JobRunner.run method: while(_running) { // clear interrupts Thread.interrupted(); Can you make these changes and see if that fixes the problem? thanks David Jencks wrote: I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Here's some annotated trace info (some from modifying Thread to tell us what is going on). This is all about thread 4ac866. It appears to being used by 2 HttpConnections more or less simultaneously (??) --we start by HttpConnection recording it's thread. 2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting _handlingThread to Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --later, it is done with this thread so it interrupts it. Note we are using HttpConnection 462080 2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection] interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --Here's Thread telling us all about the interrupt call 2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called-- on thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] 2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace from interrupt 2002-08-12 10:17:08,836 ERROR [STDERR] at java.lang.Thread.interrupt(Thread.java:665) 2002-08-12 10:17:08,836 ERROR [STDERR] at org.mortbay.http.HttpConnection.close(HttpConnection.java:248) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102) 2002-08-12 10:17:08,837 ERROR [STDERR] at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.HttpConnection.handle(HttpConnection.java:808) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186) 2002-08-12 10:17:08,838 ERROR [STDERR] at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322) 2002-08-12 10:17:08,839 ERROR [STDERR] at org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713) 2002-08-12 10:17:08,840 ERROR [STDERR] at
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
JBoss_3_0_1 is a tag, the branch tag is Branch_3_0 and I think this fix needs to be included there as there will be a 3.0.2 release. Scott Stark Chief Technology Officer JBoss Group, LLC - Original Message - From: Jules Gosnell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, August 14, 2002 7:36 AM Subject: Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!! Which Branch[es] do you think that I should update? I'm getting confused about how many there are. I figure : HEAD Branch_3_2 JBoss_3_0_1 any more ? Let me know and I will merge an up-to-date Jetty into all of them today. Jules --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
OK. I have a few other fixes to make to HEAD, then I will roll them all into Branch_3_0. What is your schedule for 3.0.2 ? What about Branch_3_2 ? Jules Scott M Stark wrote: JBoss_3_0_1 is a tag, the branch tag is Branch_3_0 and I think this fix needs to be included there as there will be a 3.0.2 release. Scott Stark Chief Technology Officer JBoss Group, LLC - Original Message - From: Jules Gosnell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, August 14, 2002 7:36 AM Subject: Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!! Which Branch[es] do you think that I should update? I'm getting confused about how many there are. I figure : HEAD Branch_3_2 JBoss_3_0_1 any more ? Let me know and I will merge an up-to-date Jetty into all of them today. Jules --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
3.0.2 will be released Aug 24. A 3.2 beta will follow that in a day or so. Scott Stark Chief Technology Officer JBoss Group, LLC - Original Message - From: Jules Gosnell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, August 14, 2002 3:05 PM Subject: Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!! OK. I have a few other fixes to make to HEAD, then I will roll them all into Branch_3_0. What is your schedule for 3.0.2 ? What about Branch_3_2 ? Jules --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
David, thanks for the detailed analysis of this.I had seen this once before but had never been able to reproduce it. I think I know what is going on It is not strickly a race rather the thread is being returned to the pool with interrupted status true - this is effecting it being used for other purposes. I think the fix is two fold - although I can't test this as I still cant reproduce it. Firstly I need to stop interrupting myself. Close does an interupt to so that the HttpServer.stop method does not hang for those JVMs where closing a socket is not enough to wakeup a blocked thread. So I think close should be: if (_handlingThread!=null Thread.currentThread!=_handlingThread) _handlingThread.interrupt(); Secondly, threads returned to the ThreadPool need to have their interrupted status cleared by calling Thread.interrupted.For now I have done this in the ThreadPool.JobRunner.run method: while(_running) { // clear interrupts Thread.interrupted(); Can you make these changes and see if that fixes the problem? thanks David Jencks wrote: I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Here's some annotated trace info (some from modifying Thread to tell us what is going on). This is all about thread 4ac866. It appears to being used by 2 HttpConnections more or less simultaneously (??) --we start by HttpConnection recording it's thread. 2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting _handlingThread to Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --later, it is done with this thread so it interrupts it. Note we are using HttpConnection 462080 2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection] interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 --Here's Thread telling us all about the interrupt call 2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called-- on thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] 2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace from interrupt 2002-08-12 10:17:08,836 ERROR [STDERR]at java.lang.Thread.interrupt(Thread.java:665) 2002-08-12 10:17:08,836 ERROR [STDERR]at org.mortbay.http.HttpConnection.close(HttpConnection.java:248) 2002-08-12 10:17:08,837 ERROR [STDERR]at org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102) 2002-08-12 10:17:08,837 ERROR [STDERR]at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066) 2002-08-12 10:17:08,838 ERROR [STDERR]at org.mortbay.http.HttpConnection.handle(HttpConnection.java:808) 2002-08-12 10:17:08,838 ERROR [STDERR]at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186) 2002-08-12 10:17:08,838 ERROR [STDERR]at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322) 2002-08-12 10:17:08,839 ERROR [STDERR]at org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713) 2002-08-12 10:17:08,840 ERROR [STDERR]at java.lang.Thread.run(Thread.java:479) --The interrupt completed successfully, we are still in HttpConnection 462080 2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished interrupting handling thread: Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] from thread Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]: jobrunner: SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080] in HttpConnection: org.mortbay.http.HttpConnection@462080 stuff about another thread and HttpConnection removed --I don't know for sure if this activity is for this
RE: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Sorry if this has been addressed, out email has been out for a while. I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Do you have a simple test case for 3.0.1? If so then I would like to volunteer to look at it, I should have some time later today or tomorrow (once I get progress working with jboss :-)) Kev Kevin Conner Orchard Information Systems Limited Newcastle Technopole, Kings Manor Newcastle Upon Tyne, NE1 6PA. United Kingdom Registered in England, Number 1900078 Tel: +44 (0) 191-2032536 Fax: +44 (0) 191 2302515 --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Do you have a simple test case for 3.0.1? If so then I would like to volunteer to look at it, I should have some time later today That's the spirit, it seems this bug is well cornered between David, Jules and Greg but thanks for proposing regards marcf --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Do you have a simple test case for 3.0.1? If so then I would like to volunteer to look at it, I should have some time later today That's the spirit, it seems this bug is well cornered between David, Jules and Greg but thanks for proposing Sorry, I must have missed that. Our email is still filtering through from our earlier outage. Glad to hear it's under control. Still, it would have been fun :-) Kev Kevin Conner Orchard Information Systems Limited Newcastle Technopole, Kings Manor Newcastle Upon Tyne, NE1 6PA. United Kingdom Registered in England, Number 1900078 Tel: +44 (0) 191-2032536 Fax: +44 (0) 191 2302515 --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development