Guys,
I reproduced the problem on JBoss-3.0.1 easily, added Greg's fixes to
the Jetty component and could not then reproduce the Exception.
Greg, If you stick this into Mort Bay Jetty cvs (or shall I?), I will do
a merge into the necessary JBoss branches.
Jules
Greg Wilkins wrote:
> David,
>
> thanks for the detailed analysis of this. I had seen this once before but
> had never been able to reproduce it.
>
> I think I know what is going on.... It is not strickly a race rather
> the thread is being returned to the pool with interrupted status true -
> this is effecting it being used for other purposes.
>
> I think the fix is two fold - although I can't test this as I still
> cant reproduce it.
>
> Firstly I need to stop interrupting myself. Close does an interupt
> to so that the HttpServer.stop method does not hang for those JVMs where
> closing a socket is not enough to wakeup a blocked thread. So I think close
> should be:
>
> if (_handlingThread!=null && Thread.currentThread!=_handlingThread)
> _handlingThread.interrupt();
>
>
> Secondly, threads returned to the ThreadPool need to have their interrupted
> status cleared by calling Thread.interrupted. For now I have done this
> in the ThreadPool.JobRunner.run method:
>
> while(_running)
> {
> // clear interrupts
> Thread.interrupted();
>
>
> Can you make these changes and see if that fixes the problem?
>
> thanks
>
>
>
>
>
>
>
>
>
> David Jencks wrote:
>
>>I've been investigating the "Interrupted while requesting permit" exception
>>that now occurs in recent jboss 3 versions under heavy load and wonder if
>>it is due to a race condition in the jetty thread pool. In any case it
>>seems clear that the Interrupt() is called from Jetty's
>>HttpConnection.close() method.
>>
>>Here's some annotated trace info (some from modifying Thread to tell us
>>what is going on). This is all about thread 4ac866. It appears to being
>>used by 2 HttpConnections more or less simultaneously (??)
>>
>>
>>--we start by HttpConnection recording it's thread.
>>2002-08-12 10:17:08,829 INFO [org.mortbay.http.HttpConnection] setting
>>_handlingThread to
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@462080
>>
>>--later, it is done with this thread so it interrupts it. Note we are
>>using HttpConnection 462080
>>
>>2002-08-12 10:17:08,833 INFO [org.mortbay.http.HttpConnection]
>>interrupting handling thread:
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
>>from thread
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@462080
>>
>>--Here's Thread telling us all about the interrupt call
>>2002-08-12 10:17:08,834 ERROR [STDERR] Interrupt called------------------
>>on thread
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
>>from thread
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
>>2002-08-12 10:17:08,835 ERROR [STDERR] java.lang.Exception: Stack trace
>>from interrupt
>>2002-08-12 10:17:08,836 ERROR [STDERR] at
>>java.lang.Thread.interrupt(Thread.java:665)
>>2002-08-12 10:17:08,836 ERROR [STDERR] at
>>org.mortbay.http.HttpConnection.close(HttpConnection.java:248)
>>2002-08-12 10:17:08,837 ERROR [STDERR] at
>>org.mortbay.http.HttpConnection.destroy(HttpConnection.java:1102)
>>2002-08-12 10:17:08,837 ERROR [STDERR] at
>>org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:1066)
>>2002-08-12 10:17:08,838 ERROR [STDERR] at
>>org.mortbay.http.HttpConnection.handle(HttpConnection.java:808)
>>2002-08-12 10:17:08,838 ERROR [STDERR] at
>>org.mortbay.http.SocketListener.handleConnection(SocketListener.java:186)
>>2002-08-12 10:17:08,838 ERROR [STDERR] at
>>org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:322)
>>2002-08-12 10:17:08,839 ERROR [STDERR] at
>>org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:713)
>>2002-08-12 10:17:08,840 ERROR [STDERR] at
>>java.lang.Thread.run(Thread.java:479)
>>
>>--The interrupt completed successfully, we are still in HttpConnection
>>462080
>>
>>2002-08-12 10:17:08,841 INFO [org.mortbay.http.HttpConnection] finished
>>interrupting handling thread:
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
>>from thread
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50079,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@462080
>>
>><stuff about another thread and HttpConnection removed>
>>
>>--I don't know for sure if this activity is for this thread.
>>2002-08-12 10:17:08,944 DEBUG [org.jboss.system.Registry] lookup
>>-1570654348=jboss.j2ee:service=EJB,jndiName=CategoryHome
>>2002-08-12 10:17:08,945 DEBUG
>[org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory]
>>Using properties: {user=postgres, password=}
>>
>>--Now our thread is being used by HttpConnection@4dfd19, repeatedly
>>2002-08-12 10:17:08,971 INFO [org.mortbay.http.HttpConnection] setting
>>_handlingThread to
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@4dfd19
>>2002-08-12 10:17:08,976 INFO [org.mortbay.http.HttpConnection] setting
>>_handlingThread to
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@4dfd19
>>2002-08-12 10:17:08,982 INFO [org.mortbay.http.HttpConnection] setting
>>_handlingThread to
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@4dfd19
>>2002-08-12 10:17:08,987 INFO [org.mortbay.http.HttpConnection] setting
>>_handlingThread to
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@4dfd19
>>2002-08-12 10:17:08,993 INFO [org.mortbay.http.HttpConnection] setting
>>_handlingThread to
>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
>>in HttpConnection: org.mortbay.http.HttpConnection@4dfd19
>>2002-08-12 10:17:08,997 DEBUG [org.jboss.system.Registry] lookup
>>-1570654348=jboss.j2ee:service=EJB,jndiName=CategoryHome
>>
>>--Now the thread is finally interrupted, apparently after a request from a
>>different HttpConnection has made its way to the managed connection pool.
>>2002-08-12 10:17:09,003 INFO
>[org.jboss.resource.connectionmanager.JBossManagedConnectionPool]
>>Interrupted while requesting permit on thread:
>>Threadorg.mortbay.util.ThreadPool$PoolThread@4ac866[SocketListener-49,5,jboss]:
>>jobrunner:
>SocketListener-49|0|Socket[addr=localhost/127.0.0.1,port=50083,localport=8080]
>>java.lang.InterruptedException
>> at
>EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore.attempt(QueuedSemaphore.java:42)
>> at
>org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:73)
>> at
>org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:312)
>> at
>org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:530)
>> at
>org.jboss.resource.connectionmanager.LocalTxConnectionManager.getManagedConnection(LocalTxConnectionManager.java:224)
>> at
>org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:595)
>> at
>org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:875)
>> at
>org.jboss.resource.adapter.jdbc.local.LocalDataSource.getConnection(LocalDataSource.java:102)
>> at
>org.jboss.ejb.plugins.jaws.jdbc.JDBCCommand.getConnection(JDBCCommand.java:700)
>> at
>org.jboss.ejb.plugins.jaws.jdbc.JDBCCommand.jdbcExecute(JDBCCommand.java:168)
>> at
>org.jboss.ejb.plugins.jaws.jdbc.JDBCFinderCommand.execute(JDBCFinderCommand.java:114)
>>
>>and so on.
>>
>>Frankly, I don't understand what is going on here. It looks to me as if
>>the interrupt() is delayed until the thread does a wait later on... but
>>this normally works. Any suggestions would be appreciated.
>>
>>
>>I also don't understand the point of interrupt() in HttpConnection.close:
>>
>> public void close()
>> throws IOException
>> {
>> try{
>> _outputStream.close();
>> _inputStream.close();
>> }
>> finally
>> {
>> if (_handlingThread!=null)
>> {
>> log.info("interrupting handling thread: " + _handlingThread
>>+ " from thread " + Thread.currentThread() + " in HttpConnection: " +
>>this);
>> _handlingThread.interrupt();
>> log.info("finished interrupting handling thread: " +
>>_handlingThread + " from thread " + Thread.currentThread() + " in
>>HttpConnection: " + this);
>> }
>> }
>> }
>>
>>
>>(logging added by me). I always find that _handlingThread ==
>>Thread.currentThread(), and haven't been able to understand how they can be
>>different.
>>
>>Thanks
>>david jencks
>>
>>
>>-------------------------------------------------------
>>This sf.net email is sponsored by: Dice - The leading online job board
>>for high-tech professionals. Search and apply for tech jobs today!
>>http://seeker.dice.com/seeker.epl?rel_code=31
>>_______________________________________________
>>Jboss-development mailing list
>>[EMAIL PROTECTED]
>>https://lists.sourceforge.net/lists/listinfo/jboss-development
>
>
>
-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development