Re: [jetty-discuss] Re: [JBoss-dev] Possible Jetty race condition?Or threading mystery? Help!!
Already in CVS.
cheers
PS. don't blow away Marc's jasper hack in JBoss
Jules Gosnell wrote:
Guys,
I reproduced the problem on JBoss-3.0.1 easily, added Greg's fixes to
the Jetty component and could not then reproduce the Exception.
Greg, If you stick this into Mort Bay Jetty cvs (or shall I?), I will do
a merge into the necessary JBoss branches.
Jules
Greg Wilkins wrote:
David,
thanks for the detailed analysis of this.I had seen this once
before but
had never been able to reproduce it.
I think I know what is going on It is not strickly a race rather
the thread is being returned to the pool with interrupted status true -
this is effecting it being used for other purposes.
I think the fix is two fold - although I can't test this as I still
cant reproduce it.
Firstly I need to stop interrupting myself. Close does an interupt
to so that the HttpServer.stop method does not hang for those JVMs where
closing a socket is not enough to wakeup a blocked thread. So I think
close
should be:
if (_handlingThread!=null Thread.currentThread!=_handlingThread)
_handlingThread.interrupt();
Secondly, threads returned to the ThreadPool need to have their
interrupted
status cleared by calling Thread.interrupted.For now I have done this
in the ThreadPool.JobRunner.run method:
while(_running)
{
// clear interrupts
Thread.interrupted();
Can you make these changes and see if that fixes the problem?
thanks
David Jencks wrote:
I've been investigating the Interrupted while requesting permit
exception
that now occurs in recent jboss 3 versions under heavy load and
wonder if
it is due to a race condition in the jetty thread pool. In any case it
seems clear that the Interrupt() is called from Jetty's
HttpConnection.close() method.
Here's some annotated trace info (some from modifying Thread to tell us
what is going on). This is all about thread 4ac866. It appears to
being
used by 2 HttpConnections more or less simultaneously (??)
--we start by HttpConnection recording it's thread.
2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--later, it is done with this thread so it interrupts it. Note we are
using HttpConnection 462080
2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--Here's Thread telling us all about the interrupt call
2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt
called--
on thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace
from interrupt
2002-08-12 10:17:08,836 ERROR [STDERR] at
java.lang.Thread.interrupt(Thread.java:665)
2002-08-12 10:17:08,836 ERROR [STDERR] at
org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
2002-08-12 10:17:08,839 ERROR [STDERR] at
org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
2002-08-12 10:17:08,840 ERROR [STDERR] at
java.lang.Thread.run(Thread.java:479)
--The interrupt completed successfully, we are still in HttpConnection
462080
2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
These two changes have made my example work without the interrupts.
What's the procedure for changing Jetty? Should I check these in? Jules?
Wait for a new Jetty?
thanks
david jencks
On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote:
David,
thanks for the detailed analysis of this.I had seen this once before
but
had never been able to reproduce it.
I think I know what is going on It is not strickly a race rather
the thread is being returned to the pool with interrupted status true -
this is effecting it being used for other purposes.
I think the fix is two fold - although I can't test this as I still
cant reproduce it.
Firstly I need to stop interrupting myself. Close does an interupt
to so that the HttpServer.stop method does not hang for those JVMs where
closing a socket is not enough to wakeup a blocked thread. So I think
close
should be:
if (_handlingThread!=null Thread.currentThread!=_handlingThread)
_handlingThread.interrupt();
Secondly, threads returned to the ThreadPool need to have their
interrupted
status cleared by calling Thread.interrupted.For now I have done this
in the ThreadPool.JobRunner.run method:
while(_running)
{
// clear interrupts
Thread.interrupted();
Can you make these changes and see if that fixes the problem?
thanks
David Jencks wrote:
I've been investigating the Interrupted while requesting permit
exception
that now occurs in recent jboss 3 versions under heavy load and wonder
if
it is due to a race condition in the jetty thread pool. In any case it
seems clear that the Interrupt() is called from Jetty's
HttpConnection.close() method.
Here's some annotated trace info (some from modifying Thread to tell us
what is going on). This is all about thread 4ac866. It appears to
being
used by 2 HttpConnections more or less simultaneously (??)
--we start by HttpConnection recording it's thread.
2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--later, it is done with this thread so it interrupts it. Note we are
using HttpConnection 462080
2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--Here's Thread telling us all about the interrupt call
2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt
called--
on thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace
from interrupt
2002-08-12 10:17:08,836 ERROR [STDERR] at
java.lang.Thread.interrupt(Thread.java:665)
2002-08-12 10:17:08,836 ERROR [STDERR] at
org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
2002-08-12 10:17:08,839 ERROR [STDERR] at
org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
2002-08-12 10:17:08,840 ERROR [STDERR] at
java.lang.Thread.run(Thread.java:479)
--The interrupt completed successfully, we are still in HttpConnection
462080
2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection]
finished
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
RE: [jetty-discuss] Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Greg, Already in CVS. cheers PS. don't blow away Marc's jasper hack in JBoss You wrote it :) Hey the other day I was at this Sabre division and one of the things they were asking about is precompiled JSP, I didn't know you had commited the fix for live deployment, it is really useful. Is there any way jasper integrates this. marcf --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [jetty-discuss] Re: [JBoss-dev] Possible Jetty race condition?Or threading mystery? Help!!
I meant the hack I did for Marc. I have raised a bug against the jasper people for it - but I have seen no action from them yet.In fact they have 290+ open bugs against tomcat/jasper and not much action on any of them! The hack is in the jboss jetty tree, but not in the main jetty tree. Don't know why really - just hedging bets I guess. cheers marc fleury wrote: Greg, Already in CVS. cheers PS. don't blow away Marc's jasper hack in JBoss You wrote it :) Hey the other day I was at this Sabre division and one of the things they were asking about is precompiled JSP, I didn't know you had commited the fix for live deployment, it is really useful. Is there any way jasper integrates this. marcf Yahoo! Groups Sponsor -~-- 4 DVDs Free +sp Join Now http://us.click.yahoo.com/pt6YBB/NXiEAA/RN.GAA/CefplB/TM -~- For the latest information about Jetty, please see http://jetty.mortbay.org To alter your subscription to this list goto http://groups.yahoo.com/group/jetty-discuss Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ -- Greg Wilkins[EMAIL PROTECTED] GB Phone: +44-(0)7092063462 Mort Bay Consulting Australia and UK.Mbl Phone: +61-(0)4 17786631 http://www.mortbay.com AU Phone: +61-(0)2 98107029 --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
Which Branch[es] do you think that I should update? I'm getting confused
about how many there are.
I figure :
HEAD
Branch_3_2
JBoss_3_0_1
any more ?
Let me know and I will merge an up-to-date Jetty into all of them today.
Jules
David Jencks wrote:
These two changes have made my example work without the interrupts.
What's the procedure for changing Jetty? Should I check these in? Jules?
Wait for a new Jetty?
thanks
david jencks
On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote:
David,
thanks for the detailed analysis of this.I had seen this once before
but
had never been able to reproduce it.
I think I know what is going on It is not strickly a race rather
the thread is being returned to the pool with interrupted status true -
this is effecting it being used for other purposes.
I think the fix is two fold - although I can't test this as I still
cant reproduce it.
Firstly I need to stop interrupting myself. Close does an interupt
to so that the HttpServer.stop method does not hang for those JVMs where
closing a socket is not enough to wakeup a blocked thread. So I think
close
should be:
if (_handlingThread!=null Thread.currentThread!=_handlingThread)
_handlingThread.interrupt();
Secondly, threads returned to the ThreadPool need to have their
interrupted
status cleared by calling Thread.interrupted.For now I have done this
in the ThreadPool.JobRunner.run method:
while(_running)
{
// clear interrupts
Thread.interrupted();
Can you make these changes and see if that fixes the problem?
thanks
David Jencks wrote:
I've been investigating the Interrupted while requesting permit
exception
that now occurs in recent jboss 3 versions under heavy load and wonder
if
it is due to a race condition in the jetty thread pool. In any case it
seems clear that the Interrupt() is called from Jetty's
HttpConnection.close() method.
Here's some annotated trace info (some from modifying Thread to tell us
what is going on). This is all about thread 4ac866. It appears to
being
used by 2 HttpConnections more or less simultaneously (??)
--we start by HttpConnection recording it's thread.
2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--later, it is done with this thread so it interrupts it. Note we are
using HttpConnection 462080
2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--Here's Thread telling us all about the interrupt call
2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt
called--
on thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace
from interrupt
2002-08-12 10:17:08,836 ERROR [STDERR] at
java.lang.Thread.interrupt(Thread.java:665)
2002-08-12 10:17:08,836 ERROR [STDERR] at
org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
2002-08-12 10:17:08,839 ERROR [STDERR] at
org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
2002-08-12 10:17:08,840 ERROR [STDERR] at
java.lang.Thread.run(Thread.java:479)
--The interrupt completed successfully, we are still in HttpConnection
462080
2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection]
finished
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
I've pretty much given up on working on anything but head.
I think your list of branches is complete.
david
On 2002.08.14 10:36:11 -0400 Jules Gosnell wrote:
Which Branch[es] do you think that I should update? I'm getting confused
about how many there are.
I figure :
HEAD
Branch_3_2
JBoss_3_0_1
any more ?
Let me know and I will merge an up-to-date Jetty into all of them today.
Jules
David Jencks wrote:
These two changes have made my example work without the interrupts.
What's the procedure for changing Jetty? Should I check these in?
Jules?
Wait for a new Jetty?
thanks
david jencks
On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote:
David,
thanks for the detailed analysis of this.I had seen this once
before
but
had never been able to reproduce it.
I think I know what is going on It is not strickly a race rather
the thread is being returned to the pool with interrupted status true -
this is effecting it being used for other purposes.
I think the fix is two fold - although I can't test this as I still
cant reproduce it.
Firstly I need to stop interrupting myself. Close does an interupt
to so that the HttpServer.stop method does not hang for those JVMs
where
closing a socket is not enough to wakeup a blocked thread. So I think
close
should be:
if (_handlingThread!=null
Thread.currentThread!=_handlingThread)
_handlingThread.interrupt();
Secondly, threads returned to the ThreadPool need to have their
interrupted
status cleared by calling Thread.interrupted.For now I have done
this
in the ThreadPool.JobRunner.run method:
while(_running)
{
// clear interrupts
Thread.interrupted();
Can you make these changes and see if that fixes the problem?
thanks
David Jencks wrote:
I've been investigating the Interrupted while requesting permit
exception
that now occurs in recent jboss 3 versions under heavy load and
wonder
if
it is due to a race condition in the jetty thread pool. In any case
it
seems clear that the Interrupt() is called from Jetty's
HttpConnection.close() method.
Here's some annotated trace info (some from modifying Thread to tell
us
what is going on). This is all about thread 4ac866. It appears to
being
used by 2 HttpConnections more or less simultaneously (??)
--we start by HttpConnection recording it's thread.
2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection]
setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--later, it is done with this thread so it interrupts it. Note we are
using HttpConnection 462080
2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--Here's Thread telling us all about the interrupt call
2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt
called--
on thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack
trace
from interrupt
2002-08-12 10:17:08,836 ERROR [STDERR] at
java.lang.Thread.interrupt(Thread.java:665)
2002-08-12 10:17:08,836 ERROR [STDERR] at
org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
2002-08-12 10:17:08,839 ERROR [STDERR] at
org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
2002-08-12 10:17:08,840 ERROR [STDERR] at
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
I'm committing the latest Jetty onto HEAD as we speak.It includes the patch.
Jules
David Jencks wrote:
I've pretty much given up on working on anything but head.
I think your list of branches is complete.
david
On 2002.08.14 10:36:11 -0400 Jules Gosnell wrote:
Which Branch[es] do you think that I should update? I'm getting confused
about how many there are.
I figure :
HEAD
Branch_3_2
JBoss_3_0_1
any more ?
Let me know and I will merge an up-to-date Jetty into all of them today.
Jules
David Jencks wrote:
These two changes have made my example work without the interrupts.
What's the procedure for changing Jetty? Should I check these in?
Jules?
Wait for a new Jetty?
thanks
david jencks
On 2002.08.13 04:57:18 -0400 Greg Wilkins wrote:
David,
thanks for the detailed analysis of this.I had seen this once
before
but
had never been able to reproduce it.
I think I know what is going on It is not strickly a race rather
the thread is being returned to the pool with interrupted status true -
this is effecting it being used for other purposes.
I think the fix is two fold - although I can't test this as I still
cant reproduce it.
Firstly I need to stop interrupting myself. Close does an interupt
to so that the HttpServer.stop method does not hang for those JVMs
where
closing a socket is not enough to wakeup a blocked thread. So I think
close
should be:
if (_handlingThread!=null
Thread.currentThread!=_handlingThread)
_handlingThread.interrupt();
Secondly, threads returned to the ThreadPool need to have their
interrupted
status cleared by calling Thread.interrupted.For now I have done
this
in the ThreadPool.JobRunner.run method:
while(_running)
{
// clear interrupts
Thread.interrupted();
Can you make these changes and see if that fixes the problem?
thanks
David Jencks wrote:
I've been investigating the Interrupted while requesting permit
exception
that now occurs in recent jboss 3 versions under heavy load and
wonder
if
it is due to a race condition in the jetty thread pool. In any case
it
seems clear that the Interrupt() is called from Jetty's
HttpConnection.close() method.
Here's some annotated trace info (some from modifying Thread to tell
us
what is going on). This is all about thread 4ac866. It appears to
being
used by 2 HttpConnections more or less simultaneously (??)
--we start by HttpConnection recording it's thread.
2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection]
setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--later, it is done with this thread so it interrupts it. Note we are
using HttpConnection 462080
2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--Here's Thread telling us all about the interrupt call
2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt
called--
on thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack
trace
from interrupt
2002-08-12 10:17:08,836 ERROR [STDERR] at
java.lang.Thread.interrupt(Thread.java:665)
2002-08-12 10:17:08,836 ERROR [STDERR] at
org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
2002-08-12 10:17:08,839 ERROR [STDERR] at
org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
2002-08-12 10:17:08,840 ERROR [STDERR] at
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
JBoss_3_0_1 is a tag, the branch tag is Branch_3_0 and I think this fix needs to be included there as there will be a 3.0.2 release. Scott Stark Chief Technology Officer JBoss Group, LLC - Original Message - From: Jules Gosnell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, August 14, 2002 7:36 AM Subject: Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!! Which Branch[es] do you think that I should update? I'm getting confused about how many there are. I figure : HEAD Branch_3_2 JBoss_3_0_1 any more ? Let me know and I will merge an up-to-date Jetty into all of them today. Jules --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
OK. I have a few other fixes to make to HEAD, then I will roll them all into Branch_3_0. What is your schedule for 3.0.2 ? What about Branch_3_2 ? Jules Scott M Stark wrote: JBoss_3_0_1 is a tag, the branch tag is Branch_3_0 and I think this fix needs to be included there as there will be a 3.0.2 release. Scott Stark Chief Technology Officer JBoss Group, LLC - Original Message - From: Jules Gosnell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, August 14, 2002 7:36 AM Subject: Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!! Which Branch[es] do you think that I should update? I'm getting confused about how many there are. I figure : HEAD Branch_3_2 JBoss_3_0_1 any more ? Let me know and I will merge an up-to-date Jetty into all of them today. Jules --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
3.0.2 will be released Aug 24. A 3.2 beta will follow that in a day or so. Scott Stark Chief Technology Officer JBoss Group, LLC - Original Message - From: Jules Gosnell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, August 14, 2002 3:05 PM Subject: Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!! OK. I have a few other fixes to make to HEAD, then I will roll them all into Branch_3_0. What is your schedule for 3.0.2 ? What about Branch_3_2 ? Jules --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Possible Jetty race condition? Or threading mystery?Help!!
David,
thanks for the detailed analysis of this.I had seen this once before but
had never been able to reproduce it.
I think I know what is going on It is not strickly a race rather
the thread is being returned to the pool with interrupted status true -
this is effecting it being used for other purposes.
I think the fix is two fold - although I can't test this as I still
cant reproduce it.
Firstly I need to stop interrupting myself. Close does an interupt
to so that the HttpServer.stop method does not hang for those JVMs where
closing a socket is not enough to wakeup a blocked thread. So I think close
should be:
if (_handlingThread!=null Thread.currentThread!=_handlingThread)
_handlingThread.interrupt();
Secondly, threads returned to the ThreadPool need to have their interrupted
status cleared by calling Thread.interrupted.For now I have done this
in the ThreadPool.JobRunner.run method:
while(_running)
{
// clear interrupts
Thread.interrupted();
Can you make these changes and see if that fixes the problem?
thanks
David Jencks wrote:
I've been investigating the Interrupted while requesting permit exception
that now occurs in recent jboss 3 versions under heavy load and wonder if
it is due to a race condition in the jetty thread pool. In any case it
seems clear that the Interrupt() is called from Jetty's
HttpConnection.close() method.
Here's some annotated trace info (some from modifying Thread to tell us
what is going on). This is all about thread 4ac866. It appears to being
used by 2 HttpConnections more or less simultaneously (??)
--we start by HttpConnection recording it's thread.
2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--later, it is done with this thread so it interrupts it. Note we are
using HttpConnection 462080
2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--Here's Thread telling us all about the interrupt call
2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called--
on thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace
from interrupt
2002-08-12 10:17:08,836 ERROR [STDERR]at
java.lang.Thread.interrupt(Thread.java:665)
2002-08-12 10:17:08,836 ERROR [STDERR]at
org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
2002-08-12 10:17:08,837 ERROR [STDERR]at
org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
2002-08-12 10:17:08,837 ERROR [STDERR]at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
2002-08-12 10:17:08,838 ERROR [STDERR]at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
2002-08-12 10:17:08,838 ERROR [STDERR]at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
2002-08-12 10:17:08,838 ERROR [STDERR]at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
2002-08-12 10:17:08,839 ERROR [STDERR]at
org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
2002-08-12 10:17:08,840 ERROR [STDERR]at
java.lang.Thread.run(Thread.java:479)
--The interrupt completed successfully, we are still in HttpConnection
462080
2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
stuff about another thread and HttpConnection removed
--I don't know for sure if this activity is for this
RE: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Sorry if this has been addressed, out email has been out for a while. I've been investigating the Interrupted while requesting permit exception that now occurs in recent jboss 3 versions under heavy load and wonder if it is due to a race condition in the jetty thread pool. In any case it seems clear that the Interrupt() is called from Jetty's HttpConnection.close() method. Do you have a simple test case for 3.0.1? If so then I would like to volunteer to look at it, I should have some time later today or tomorrow (once I get progress working with jboss :-)) Kev Kevin Conner Orchard Information Systems Limited Newcastle Technopole, Kings Manor Newcastle Upon Tyne, NE1 6PA. United Kingdom Registered in England, Number 1900078 Tel: +44 (0) 191-2032536 Fax: +44 (0) 191 2302515 --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Do you have a simple test case for 3.0.1? If so then I would like to volunteer to look at it, I should have some time later today That's the spirit, it seems this bug is well cornered between David, Jules and Greg but thanks for proposing regards marcf --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
Do you have a simple test case for 3.0.1? If so then I would like to volunteer to look at it, I should have some time later today That's the spirit, it seems this bug is well cornered between David, Jules and Greg but thanks for proposing Sorry, I must have missed that. Our email is still filtering through from our earlier outage. Glad to hear it's under control. Still, it would have been fun :-) Kev Kevin Conner Orchard Information Systems Limited Newcastle Technopole, Kings Manor Newcastle Upon Tyne, NE1 6PA. United Kingdom Registered in England, Number 1900078 Tel: +44 (0) 191-2032536 Fax: +44 (0) 191 2302515 --- This sf.net email is sponsored by: Dice - The leading online job board for high-tech professionals. Search and apply for tech jobs today! http://seeker.dice.com/seeker.epl?rel_code=31 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
[JBoss-dev] Possible Jetty race condition? Or threading mystery? Help!!
I've been investigating the Interrupted while requesting permit exception
that now occurs in recent jboss 3 versions under heavy load and wonder if
it is due to a race condition in the jetty thread pool. In any case it
seems clear that the Interrupt() is called from Jetty's
HttpConnection.close() method.
Here's some annotated trace info (some from modifying Thread to tell us
what is going on). This is all about thread 4ac866. It appears to being
used by 2 HttpConnections more or less simultaneously (??)
--we start by HttpConnection recording it's thread.
2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--later, it is done with this thread so it interrupts it. Note we are
using HttpConnection 462080
2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
--Here's Thread telling us all about the interrupt call
2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called--
on thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace
from interrupt
2002-08-12 10:17:08,836 ERROR [STDERR] at
java.lang.Thread.interrupt(Thread.java:665)
2002-08-12 10:17:08,836 ERROR [STDERR] at
org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
2002-08-12 10:17:08,837 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
2002-08-12 10:17:08,838 ERROR [STDERR] at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
2002-08-12 10:17:08,839 ERROR [STDERR] at
org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
2002-08-12 10:17:08,840 ERROR [STDERR] at
java.lang.Thread.run(Thread.java:479)
--The interrupt completed successfully, we are still in HttpConnection
462080
2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished
interrupting handling thread:
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
from thread
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@462080
stuff about another thread and HttpConnection removed
--I don't know for sure if this activity is for this thread.
2002-08-12 10:17:08,944 DEBUG [org.jboss.system.Registry] lookup
-1570654348=jboss.j2ee:service=EJB,jndiName=CategoryHome
2002-08-12 10:17:08,945 DEBUG
[org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory]
Using properties: {user=postgres, password=}
--Now our thread is being used by HttpConnection@4dfd19, repeatedly
2002-08-12 10:17:08,971 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@4dfd19
2002-08-12 10:17:08,976 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
in HttpConnection: org.mortbay.http.HttpConnection@4dfd19
2002-08-12 10:17:08,982 INFO [org.mortbay.http.HttpConnection] setting
_handlingThread to
Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
jobrunner:
SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
in HttpConnection:
