Hi All,
We have setup an API Manager cluster (beta 15th pack) fronted by nginx with
following details.
Publisher
Two store nodes
Gateway Cluster(one manager with two workers)
IS as key manager(two node cluster)
DB-Oracle 12c
Userstore-Read/write ldap
We have ran a long running test for some days and observed out of memory on
one of the IS nodes. Once we have cleaned all created heap dumps,logs and
try to restart the cluster again and found the following error continuously
on publisher, gateway and IS nodes.
I have attached the current database running process details with [1].
Further we have used the [2] for fine tunning the servers.
Appreciate your feedback on resolving this.
ERROR - TransactionManager Failed to start new registry transaction.
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-216-thread-7]
Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none
available[size:50; busy:50; idle:0; lastwait:60000].
at
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
at
org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
at
org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
at
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)
at
org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)
at
org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)
at
org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)
at
org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)
at java.security.AccessController.doPrivileged(Native Method)
at
org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)
at
org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)
at
org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing.
org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to start
new registry transaction.
at
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403)
at
org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398)
at
org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60)
at
org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381)
at
org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378)
at java.security.AccessController.doPrivileged(Native Method)
at
org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378)
at
org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201)
at
org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212)
at
org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException:
[pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60
seconds, none available[size:50; busy:50; idle:0; lastwait:60000].
at
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
at
org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
at
org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
at
org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
... 18 more
[1]
[2] https://docs.wso2.com/display/AM1100/Tuning+Performance
Thank you,
--
*Chalitha Maheshwari*
Software Engineer-QA,
WSO2 Inc.
*E-mail:* [email protected]
*Mobile: *+94710 411 112
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
