Re: Tomcat JDBC Pool Cleaner Deadlock Problem
Dear Felix, Thanks for your feedback , the health check period is so aggressive now. SRE team's and apm monitor tools call internally and externally endpoint and health indicators. I asked the sre guys to decrease the frequency of those calls. They made some decrease on calls but nothing changed. Due to an old spring mvc project it doesn't have a kubernetes probe , about 100 calls in minutes. Gökhan On Thu, Aug 27, 2020 at 9:48 PM Felix Schumacher < felix.schumac...@internetallee.de> wrote: > > Am 27.08.20 um 11:47 schrieb Gokhan Akgul: > > Hi , > > > > I have been facing the deadlock issue for the last 2 months about > > JDBCPoolCleaner Thread . > > > > Following config set in context.xml > > > > > auth="Container" > > type="javax.sql.DataSource" > > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" > > driverClassName="com.mysql.jdbc.Driver" > > > url="jdbc:mysql://adress:3306/db?useUnicode=truecharacterEncoding=latin5characterResultSet=latin5zeroDateTimeBehavior=convertToNullautoReconnect=trueinteractiveClient=true" > > username="user" > > password="pass" > > initialSize="10" > > maxActive="30" > > maxIdle="15" > > minIdle="10" > > maxWait="3" > > timeBetweenEvictionRunsMillis="5000" > > minEvictableIdleTimeMillis="6" > > removeAbandonedTimeout="600" > > removeAbandoned="true" > > logAbandoned="false" > > testWhileIdle="true" > > testOnBorrow="true" > > testOnReturn="false" > > validationQuery="/* ping */ SELECT 1" > > validationInterval="3" > > jmxEnabled="true" > > > jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTimer;SlowQueryReport" > > /> > > > > > > > > Thread dump > > > > Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED > > - waiting to lock <0x57dcb0b7> (a > com.mysql.jdbc.JDBC4PreparedStatement) > > owned by http-nio-8080-exec-8 id=25 > > So, Tomcat tries to close a connection, that it thinks is abandoned > (i.e. in your setup hasn't been used for more than 600 seconds and the > pool of idle connections is getting empty). > > The connection is still in use ... > > > at > com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078) > > at > com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584) > > at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) > > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) > > at > org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388) > > at > org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468) > > at java.util.TimerThread.mainLoop(Timer.java:555) > > at java.util.TimerThread.run(Timer.java:505) > > > > Locked synchronizers: count = 1 > > - > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 > > > > > > > > http-nio-8080-exec-8 id=25 state=BLOCKED > > - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) > > owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 > > ... by hibernate. The question now is, how often is your healtheck > getting called (every 10 min == 600s)? > > If they are close together, you might want to set the abandoned timeout > higher than the healthcheck interval, or you could try to close the > connection (or whatever the equivalent is in hibernate (a session?)) in > your code. > > Felix > > > at > com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435) > > at > com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.j
Re: Tomcat JDBC Pool Cleaner Deadlock Problem
Dear Phil , Thanks for your feedback. I forgot to mention the mysql driver version. The Mysql driver version is 5.1.32. My plan is to upgrade the mysql driver to 5.1.46 version and monitor for a while. On Sat, Aug 29, 2020 at 6:50 PM Phil Steitz wrote: > > On 8/27/20 2:47 AM, Gokhan Akgul wrote: > > Hi , > > > > I have been facing the deadlock issue for the last 2 months about > > JDBCPoolCleaner Thread . > > > > Following config set in context.xml > > > > > auth="Container" > > type="javax.sql.DataSource" > > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" > > driverClassName="com.mysql.jdbc.Driver" > > > > url="jdbc:mysql://adress:3306/db?useUnicode=truecharacterEncoding=latin5characterResultSet=latin5zeroDateTimeBehavior=convertToNullautoReconnect=trueinteractiveClient=true" > > username="user" > > password="pass" > > initialSize="10" > > maxActive="30" > > maxIdle="15" > > minIdle="10" > > maxWait="3" > > timeBetweenEvictionRunsMillis="5000" > > minEvictableIdleTimeMillis="6" > > removeAbandonedTimeout="600" > > removeAbandoned="true" > > logAbandoned="false" > > testWhileIdle="true" > > testOnBorrow="true" > > testOnReturn="false" > > validationQuery="/* ping */ SELECT 1" > > validationInterval="3" > > jmxEnabled="true" > > > > jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTimer;SlowQueryReport" > > /> > > > > > > > > Thread dump > > > > Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED > > - waiting to lock <0x57dcb0b7> (a > com.mysql.jdbc.JDBC4PreparedStatement) > > owned by http-nio-8080-exec-8 id=25 > > at > com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078) > > at > com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584) > > at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) > > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) > > at > org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388) > > at > org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999) > > at > org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468) > > at java.util.TimerThread.mainLoop(Timer.java:555) > > at java.util.TimerThread.run(Timer.java:505) > > > > Locked synchronizers: count = 1 > >- > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 > > > > > > > > http-nio-8080-exec-8 id=25 state=BLOCKED > > - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) > > owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 > > at > com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435) > > at > com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.java:3269) > > at com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675) > > at > com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java:39) > > at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown > Source) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.java:657) > > at > com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3064) > > at > com.mysql.jdbc.ConnectionImpl.getMetaData(Connectio
Tomcat JDBC Pool Cleaner Deadlock Problem
let.java:733) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1626) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) - locked org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@526aa725 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@21c6b5b9 Tomcat version : 8.8.57 Java 7 Application is a simple spring mvc application. Runs on Kubernetes Do you have any idea how to solve it? -- Gokhan Akgul Best regards
