Is your postgresql server responsive?
Yes, there is no error logged in db server and server is very much
responsive.
Have you ever been able to get a valid connection during the lifetime of
the application?
Yes
Was it working for sometime, and then you suddenly start getting this
issue...?
Yes. it happenes suddendly and sometime not always.
Assuming you have isolated this thread correctly as one which is stuck(by
taking multiple thread dump...) for how long it has been stuck?
Many thread dump took, analyzed and says thread stuck while getting the
connection from the db or trying to execute queries.
Is it possible firewall dropping connection when Apps server trying to get
the db connection.
Is it ever coming out of that stuck state?
yes. However, number of threads are not coming down.
Look into the state of tcp connection using netstat.
1 FIN_WAIT1
1 Foreign
1 established)
2 LAST_ACK
2 SYN_SENT
4 CLOSING
10 LISTEN
17 CLOSE_WAIT
116 TIME_WAIT
287 ESTABLISHED
On Sat, Nov 22, 2014 at 3:08 PM, Ameer Mawia <ameer.ma...@gmail.com> wrote:
> Jaiswal,
>
> This issue is of connection between your host and postgresql server, you
> are trying to connect and has nothing to do with Tomcat per say.
>
> To debug the issue, you will have to answer few more question:
> Is your postgresql server responsive?
> Have you ever been able to get a valid connection during the lifetime of
> the application? Was it working for sometime, and then you suddenly start
> getting this issue...?
> Assuming you have isolated this thread correctly as one which is stuck(by
> taking multiple thread dump...) for how long it has been stuck? Is it ever
> coming out of that stuck state?
> Look into the state of tcp connection using netstat.
>
> Regards,
> Ameer Mawia
>
>
> On Sat, Nov 22, 2014 at 10:44 AM, Dhaval Jaiswal <dhaval.jais...@via.com>
> wrote:
>
> > yes. the thread is stucked and started increasing load of the server.
> >
> > My configuration is as below. Query is fetching one row only.
> >
> >
> > <Resource name="jdbc" auth="Container"
> type="javax.sql.DataSource"
> > driverClassName="org.postgresql.Driver" url="jdbc:postgresql"
> > maxActive="100" maxIdle="20" maxWait="30000"
> > validationQuery="select 1" testOnBorrow="true"
> > removeAbandoned="true" removeAbandonedTimeout="120"
> > logAbandoned="true" />
> >
> >
> >
> >
> >
> >
> > On Sat, Nov 22, 2014 at 1:29 AM, Ameer Mawia <ameer.ma...@gmail.com>
> > wrote:
> >
> > > Jaiswal, what your exact question is?
> > >
> > > Obviously in your application, you need a DB connection and if this is
> > > thread which is stuck, stack trace tells that tomcat dbcp, is trying to
> > get
> > > you a connection, but stuck while reading the result of validate query
> > > which it had run before returning you a valid connection.
> > >
> > > Probably you can take a look at validation query, for now? Ideally it
> > > should be a very simple query, returning just one row and column.
> > >
> > > Regards,
> > > Ameer Mawia
> > >
> > > On Sat, Nov 22, 2014 at 12:30 AM, Dhaval Jaiswal <
> dhaval.jais...@via.com
> > >
> > > wrote:
> > >
> > > > Following thread stuck and increased the load.
> > > >
> > > >
> > > > java.net.SocketInputStream.read(SocketInputStream.java:129)
> > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
> > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
> > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
> > > > org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
> > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
> > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
> > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332)
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991)
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84)
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
> > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > > Source)
> > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > > Source)
> > > >
> > > >
> > >
> >
> com.eos.accounts.policies.PolicyManager.getAllPoliciesForUser(PolicyManager.java:62)
> > > >
> > > >
> > >
> >
> com.eos.accounts.policies.Policies.loadAllPoliciesForUser(Policies.java:25)
> > > > com.eos.accounts.data.User.createFromDataSet(User.java:3165)
> > > > com.eos.accounts.data.User.getUserById(User.java:2878)
> > > >
> > > >
> > >
> >
> com.eos.b2c.beans.GenericProductBean.setDeliveryDetails(GenericProductBean.java:320)
> > > >
> com.eos.b2c.ui.ResellerNavigation.doGet(ResellerNavigation.java:2362)
> > > >
> com.eos.b2c.ui.ResellerNavigation.doPost(ResellerNavigation.java:579)
> > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > > >
> > > >
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > > >
> > > >
> > > >
> > > > thread dump
> > > >
> > > >
> > > > "http-8080-2" daemon prio=10 tid=0x000000005335e000 nid=0x6d82
> runnable
> > > > [0x00002b8b9a5c9000..0x00002b8b9a5caa90]
> > > > java.lang.Thread.State: RUNNABLE
> > > > at java.net.SocketInputStream.socketRead0(Native Method)
> > > > at
> java.net.SocketInputStream.read(SocketInputStream.java:129)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
> > > > at
> org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> > > > - locked <0x00002b881d689760> (a
> > > > org.postgresql.core.v3.QueryExecutorImpl)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
> > > > at
> > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
> > > > at
> > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > Source)
> > > > at
> > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > Source)
> > > > at
> > > >
> > > >
> > >
> >
> com.eos.gds.util.GenericSearchQuery.executeQuery(GenericSearchQuery.java:191)
> > > >
> > > >
> > > >
> > > > On Sat, Nov 22, 2014 at 12:26 AM, Dhaval Jaiswal <
> > dhaval.jais...@via.com
> > > >
> > > > wrote:
> > > >
> > > > > Below thread stucked and increased the load.
> > > > >
> > > > >
> > > > > java.net.SocketInputStream.read(SocketInputStream.java:129)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
> > > > > org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
> > > > >
> > > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
> > > > >
> com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > > > Source)
> > > > >
> com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > > > Source)
> > > > >
> > > > >
> > > >
> > >
> >
> com.eos.accounts.policies.PolicyManager.getAllPoliciesForUser(PolicyManager.java:62)
> > > > >
> > > > >
> > > >
> > >
> >
> com.eos.accounts.policies.Policies.loadAllPoliciesForUser(Policies.java:25)
> > > > > com.eos.accounts.data.User.createFromDataSet(User.java:3165)
> > > > > com.eos.accounts.data.User.getUserById(User.java:2878)
> > > > >
> > > > >
> > > >
> > >
> >
> com.eos.b2c.beans.GenericProductBean.setDeliveryDetails(GenericProductBean.java:320)
> > > > >
> > com.eos.b2c.ui.ResellerNavigation.doGet(ResellerNavigation.java:2362)
> > > > >
> > com.eos.b2c.ui.ResellerNavigation.doPost(ResellerNavigation.java:579)
> > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > > > >
> > > > >
> > > > >
> > > > > thread dump output
> > > > >
> > > > >
> > > > > "http-8080-2" daemon prio=10 tid=0x000000005335e000 nid=0x6d82
> > runnable
> > > > > [0x00002b8b9a5c9000..0x00002b8b9a5caa90]
> > > > > java.lang.Thread.State: RUNNABLE
> > > > > at java.net.SocketInputStream.socketRead0(Native Method)
> > > > > at
> > java.net.SocketInputStream.read(SocketInputStream.java:129)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
> > > > > at
> > org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> > > > > - locked <0x00002b881d689760> (a
> > > > > org.postgresql.core.v3.QueryExecutorImpl)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
> > > > > at
> > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > > Source)
> > > > > at
> > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown
> > > > Source)
> > > > > at
> > > > >
> > > >
> > >
> >
> com.eos.gds.util.GenericSearchQuery.executeQuery(GenericSearchQuery.java:191)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
