I am using Derby as a database as part of a Glassfish installation. I am
processing JMS messages with each one causing access to the Derby database via
Eclipselink. After about 80K to 100K messages processed, I have a hung system.
I run jstack on the Derby network server and there is no deadlock. Also I run
jstack on the Glassfish and there is no deadlock. In the thread stack for
Glassfish I see a thread with the following stack trace:
"p: thread-pool-1; w: 3" daemon prio=6 tid=0x5de34800 nid=0xb94 runnable
[0x60abe000..0x60abfd94]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.derby.client.net.Reply.fill(Unknown Source)
at org.apache.derby.client.net.Reply.ensureALayerDataInBuffer(Unknown
Source)
at org.apache.derby.client.net.Reply.readDssHeader(Unknown Source)
at org.apache.derby.client.net.Reply.startSameIdChainParse(Unknown
Source)
at
org.apache.derby.client.net.NetConnectionReply.readExchangeServerAttributes(Unknown
Source)
at
org.apache.derby.client.net.NetConnection.readServerAttributesAndKeyExchange(Unknown
Source)
at
org.apache.derby.client.net.NetConnection.readAllConnectCommandsChained(Unknown
Source)
at org.apache.derby.client.net.NetConnection.readDeferredReset(Unknown
Source)
at
org.apache.derby.client.net.NetAgent.readDeferredResetConnection(Unknown Source)
at org.apache.derby.client.net.NetAgent.beginReadChain(Unknown Source)
at org.apache.derby.client.am.Agent.flow(Unknown Source)
at org.apache.derby.client.am.Statement.flowExecute(Unknown Source)
at org.apache.derby.client.am.Statement.executeQueryX(Unknown Source)
at org.apache.derby.client.am.Statement.executeQuery(Unknown Source)
- locked <0x1d90dba0> (a org.apache.derby.client.net.NetConnection40)
at
org.apache.derby.client.am.Connection.getTransactionIsolation(Unknown Source)
at
org.apache.derby.client.am.LogicalConnection.getTransactionIsolation(Unknown
Source)
at
com.sun.gjc.spi.ManagedConnectionFactory.resetIsolation(ManagedConnectionFactory.java:463)
at
com.sun.gjc.spi.ManagedConnection.resetConnectionProperties(ManagedConnection.java:354)
at
com.sun.gjc.spi.ManagedConnection.getConnection(ManagedConnection.java:328)
at
com.sun.enterprise.resource.ConnectorAllocator.fillInResourceObjects(ConnectorAllocator.java:155)
at
com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:514)
at
com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
at
com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
This thread is making no progress at all. This is the one hanging my system.
The derby log shows:
2010-11-15 23:57:46.593 GMT : Apache Derby Network Server - 10.5.3.0 - (802917)
started and ready to accept connections on port 1527
----------------------------------------------------------------
2010-11-16 00:00:20.078 GMT:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.5.3.0
- (802917): instance a816c00e-012c-51f9-12b2-0000376d74d3
on database directory C:\Documents and Settings\Brett
Bergquist\.netbeans-derby\csemdb
Database Class Loader started - derby.database.classpath='CSEM.csemderby'
(END)
The rest of Glassfish is up and running as well as other parts of my
application. This one thread, however, is causing my JMS processing to be hung.
This is the second time in two days of testing that this has occurred. The
previous lockup has exactly the same thread stack traceback in a few other
threads, all from different database queries, all stuck at the same point.
Restarting the derby network server, caused the condition to clear up as far as
the application server is concerned, so I am pretty confident it is related to
derby.
Any help on where to look to find this problem will be greatly appreciated.
Brett
