On 4/12/15 10:31 AM, Shawn Heisey wrote: > On 4/12/2015 9:16 AM, Phil Steitz wrote: >> Can you post your full pool config and info on versions of DBCP and >> POOL? >> >> Do you call isClosed() before closing a connection (not recommended >> - that is really the issue in the ticket you commented)? >> >> To make sure there is not an actual deadlock somewhere, you should >> examine a full thread dump. The thread you posted in the ticket is >> waiting on the pool, not holding any DBCP or POOL related locks; but >> it does hold this application lock: >> at com.REDACTED.idxbuild.solr.Chain.doReinsert(Chain.java:1332) >> - locked <0x00000000db0011c8> (a java.lang.Object) >> Make sure no other threads in process of doing things that will >> return connections to the pool aren't waiting on this lock. >> >> Given that you are using DBCP 2.1, if you have access to a JMX >> console it would also be good to look at what is happening in the >> pool when your application hangs. In particular, numActive, numIdle >> and listAllObjects would be good to look at. Looks like you are >> using PoolingDataSource directly, so you need to look at the >> underlying GenericObject's JMX properties. > My code does not contain the string "isClosed" at all. > > I thought I had saved the full thread dump, but the stdout logfile was > wiped when I restarted the program, and I didn't think to save a copy. > I will be sure to save it next time there's a problem.
Full thread dump will be helpful. > > Here is the entire java file for my "Database" class, edited to redact > the company top-level domain: > > http://apaste.info/6pX Several methods, getSingleObject, getResultSet, closeResultSet, doUpdate can abandon connections on exception paths. You should insert finally blocks in these methods to ensure that connections are closed. Have a look at the PoolingDataSource example referenced [1] on the DBCP web page for a model to use with try-catch-finally. Not closing connections on exception paths will cause leaks if exceptions are thrown. If you are seeing BuildExceptions generated by this code, each of them is leaking a connection. > > The object that I am locking on (0x00000000db0011c8) is a global lock > for my whole program which I synchronize on for a handful of > normally-short-lived db-related operations just to be sure that they > aren't happening simultaneously, and to help make sure that the outcome > of certain operations is visible across threads. I'm reasonably sure > that there are no operations in those synchronization blocks that are > expected to be slow, though I haven't verified that to 100% certainty. They don't have to be slow to deadlock, if you hit the right timing and the following kind of scenario: Thread 1 grab the lock and hold it while waiting to get a connection (what the thread you showed in the ticket is doing) Threads 2, ..., n try to get the lock before returning connections they hold Key is to get a full thread dump and see which threads are waiting on / holding which locks. Phil [1] http://s.apache.org/0Nl > One of the operations handled in that kind of synchronization lock is > the optimization of certain DB tables, which only happens if the size of > those tables is below a certain threshold -- part of making sure that > the operation completes quickly and doesn't deadlock the program. > > Here are all the jar files in my program's lib directory. My program > uses javamail, jrobin, dbcp, and solrj: > > commons-dbcp2-2.1.jar > commons-io-2.4.jar > commons-pool2-2.3.jar > httpclient-4.4.1.jar > httpcore-4.4.1.jar > httpmime-4.4.1.jar > javax.mail.jar > jcl-over-slf4j-1.7.12.jar > jrobin-1.5.14.jar > jul-to-slf4j-1.7.12.jar > log4j-1.2.17.jar > mysql-connector-java-5.1.35-bin.jar > noggit-0.7.jar > slf4j-api-1.7.12.jar > slf4j-log4j12-1.7.12.jar > solr-solrj-5.0.0.jar > stax2-api-3.1.1.jar > woodstox-core-asl-4.2.0.jar > zookeeper-3.4.6.jar > > I do not have remote JMX enabled for this application, and the server > where it runs does not have a gui. I will add some code that spits out > the three requested pool stats every time the watchdog thread detects a > problem. I cannot predict when there will be a problem ... this code > can run for weeks with nothing going wrong, or the deadlock might happen > only a few minutes after startup. > > Thanks, > Shawn > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > For additional commands, e-mail: user-h...@commons.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
