Hi Abimaran,
As per Danesh comment, I have increase the max active connection till 120.
Still i am seeing this error printed continuously. Do we need to increase
the value further
[2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start new
registry transaction.
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-234-thread-10]
Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none
available[size:120; busy:120; idle:0; lastwait:60000].
at
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
at
org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
at
org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
at
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177)
at
org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616)
at
org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)
at
org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)
at
org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)
at java.security.AccessController.doPrivileged(Native Method)
at
org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing.
org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to start
new registry transaction.
at
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177)
at
org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616)
at
org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60)
at
org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596)
at
org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593)
at java.security.AccessController.doPrivileged(Native Method)
at
org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException:
[pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in
60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000].
at
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
at
org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
at
org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
at
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
... 16 more
On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan <[email protected]>
wrote:
> Hi Chalitha,
>
> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException:
> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60
> seconds, none available[size:50; busy:50; idle:0; lastwait:60000].
>
> From the above logs, the problem lies with the database connection which
> has pool size of 50. So it should be carbon db, because for other dbsm you
> are having 100 as the pool size. You need to increase it as well.
>
> On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage <[email protected]>
> wrote:
>
>> Hi Danesh,
>>
>> Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to 100
>> which are pointed to oracle database. But for the carbon db we didn't
>> change it since it's local h2.
>> Do we need to change it also?
>>
>> On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <[email protected]> wrote:
>>
>>> Hi Chalitha,
>>>
>>> Can you increase the maxActive in Registry DB configuration. Seems like
>>> there are no available connections in tomcat jdbc pool. Please set this to
>>> 80 at least.
>>>
>>> Thanks
>>> Danesh
>>>
>>> On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage <[email protected]
>>> > wrote:
>>>
>>>> Hi All,
>>>>
>>>> We have setup an API Manager cluster (beta 15th pack) fronted by nginx
>>>> with following details.
>>>>
>>>> Publisher
>>>> Two store nodes
>>>> Gateway Cluster(one manager with two workers)
>>>> IS as key manager(two node cluster)
>>>> DB-Oracle 12c
>>>> Userstore-Read/write ldap
>>>>
>>>> We have ran a long running test for some days and observed out of
>>>> memory on one of the IS nodes. Once we have cleaned all created heap
>>>> dumps,logs and try to restart the cluster again and found the following
>>>> error continuously on publisher, gateway and IS nodes.
>>>>
>>>> I have attached the current database running process details with [1].
>>>> Further we have used the [2] for fine tunning the servers.
>>>>
>>>> Appreciate your feedback on resolving this.
>>>>
>>>>
>>>>
>>>> ERROR - TransactionManager Failed to start new registry transaction.
>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-216-thread-7]
>>>> Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none
>>>> available[size:50; busy:50; idle:0; lastwait:60000].
>>>> at
>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
>>>> at
>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
>>>> at
>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)
>>>> at
>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)
>>>> at
>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)
>>>> at
>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)
>>>> at
>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)
>>>> at
>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing.
>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>> start new registry transaction.
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)
>>>> at
>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)
>>>> at
>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)
>>>> at
>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)
>>>> at
>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)
>>>> at
>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException:
>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60
>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000].
>>>> at
>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
>>>> at
>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
>>>> at
>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
>>>> ... 18 more
>>>>
>>>>
>>>> [1]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [2] https://docs.wso2.com/display/AM1100/Tuning+Performance
>>>>
>>>>
>>>> Thank you,
>>>> --
>>>> *Chalitha Maheshwari*
>>>> Software Engineer-QA,
>>>> WSO2 Inc.
>>>>
>>>> *E-mail:* [email protected]
>>>> *Mobile: *+94710 411 112
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Dev mailing list
>>>> [email protected]
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Danesh Kuruppu
>>> Software Engineer
>>> WSO2 Inc,
>>> Mobile: +94 (77) 1690552
>>>
>>
>>
>>
>> --
>> *Chalitha Maheshwari*
>> Software Engineer-QA,
>> WSO2 Inc.
>>
>> *E-mail:* [email protected]
>> *Mobile: *+94710 411 112
>>
>> _______________________________________________
>> Dev mailing list
>> [email protected]
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> Thanks
> Abimaran Kugathasan
> Senior Software Engineer
>
> Email : [email protected]
> Mobile : +94 773922820
>
> <http://stackoverflow.com/users/515034>
> <http://lk.linkedin.com/in/abimaran>
> <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank>
> <https://twitter.com/abimaran>
>
>
--
*Chalitha Maheshwari*
Software Engineer-QA,
WSO2 Inc.
*E-mail:* [email protected]
*Mobile: *+94710 411 112
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
