Hi Sajini, Can you please point which part of above configuration fixed the issue?
Thanks, Bhathiya On Mon, Oct 6, 2014 at 9:30 AM, Sajini De Silva <[email protected]> wrote: > Hi, > > This issue was fixed with following configurations. > > *master-datasources.xml in MB node:* > > <!-- external Cassandra data source. --> > <datasource> > <name>WSO2_CASSANDRA_DB</name> > <description>The datasource used for cassandra</description> > <jndiConfig> > <name>CassandraRepo</name> > </jndiConfig> > <definition type="CASSANDRA"> > <configuration> > <async>false</async> > <clusterName>TestCluster</clusterName> > <compression>SNAPPY</compression> > <concurrency>100</concurrency> > <username>admin</username> > <password encrypted="true">admin</password> > <port>9042</port> > <maxConnections>100</maxConnections> > > <hosts> > <host>127.0.0.1</host> > </hosts> > <loadBalancePolicy> > <exclusionThreshold>2.5</exclusionThreshold> > <latencyAware>true</latencyAware> > <minMeasure>100</minMeasure> > <policyName>RoundRobinPolicy</policyName> > <retryPeriod>10</retryPeriod> > <scale>2</scale> > </loadBalancePolicy> > > <poolOptions> > > <coreConnectionsForLocal>1000</coreConnectionsForLocal> > > <coreConnectionsForRemote>1000</coreConnectionsForRemote> > > <maxConnectionsForLocal>1000</maxConnectionsForLocal> > > <maxConnectionsForRemote>1000</maxConnectionsForRemote> > > <maxSimultaneousRequestsForLocal>1000</maxSimultaneousRequestsForLocal> > > <maxSimultaneousRequestsForRemote>1000</maxSimultaneousRequestsForRemote> > > <minSimultaneousRequestsForLocal>1000</minSimultaneousRequestsForLocal> > > <minSimultaneousRequestsForRemote>1000</minSimultaneousRequestsForRemote> > </poolOptions> > > <reconnectPolicy> > <baseDelayMs>3000</baseDelayMs> > <policyName>ConstantReconnectionPolicy</policyName> > </reconnectPolicy> > <socketOptions> > <connectTimeoutMillis>10000</connectTimeoutMillis> > <keepAlive>true</keepAlive> > <readTimeoutMillis>15000</readTimeoutMillis> > <tcpNoDelay>true</tcpNoDelay> > </socketOptions> > > </configuration> > > </definition> > </datasource> > > > > *cassandra-env.sh in cassandra node:* > > system_memory_in_mb="10240" > system_cpu_cores="6" > > Tested with an SSD hard drive. > > Thank you, > Sajini. > > On Fri, Oct 3, 2014 at 12:23 PM, Sajini De Silva <[email protected]> wrote: > >> Hi Prabath, >> >> This is the error we got while deleting messages. >> >> [2014-10-03 12:16:55,059] ERROR >> {org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl} - Error while >> deleting messages >> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >> at >> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >> at >> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >> at >> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >> at com.datastax.driver.core.Session.execute(Session.java:111) >> at >> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >> at >> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299) >> at >> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327) >> at >> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203) >> at >> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30) >> at >> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19) >> at >> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: >> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during >> read)) >> at >> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106) >> at >> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177) >> ... 3 more >> Oct 3, 2014 12:16:55 PM com.lmax.disruptor.FatalExceptionHandler >> handleEventException >> SEVERE: Exception processing: 12095 >> org.wso2.andes.kernel.AndesAckData@5341cc77 >> org.wso2.andes.kernel.AndesException: >> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >> at >> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303) >> at >> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327) >> at >> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203) >> at >> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30) >> at >> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19) >> at >> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: >> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during >> read)) >> at >> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >> at >> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >> at >> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >> at com.datastax.driver.core.Session.execute(Session.java:111) >> at >> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >> at >> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299) >> ... 8 more >> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: >> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during >> read)) >> at >> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106) >> at >> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177) >> ... 3 more >> Exception in thread "pool-25-thread-1" java.lang.RuntimeException: >> org.wso2.andes.kernel.AndesException: >> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >> at >> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45) >> at >> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: org.wso2.andes.kernel.AndesException: >> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >> at >> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303) >> at >> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327) >> at >> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203) >> at >> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30) >> at >> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19) >> at >> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >> ... 3 more >> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: >> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during >> read)) >> at >> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >> at >> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >> at >> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >> at com.datastax.driver.core.Session.execute(Session.java:111) >> at >> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >> at >> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299) >> ... 8 more >> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: >> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during >> read)) >> at >> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106) >> at >> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177) >> ... 3 more >> >> >> We tried this scenario increasing the number of threads in thread pool to >> 1000, but same error occurred. >> >> Thank you, >> Sajini. >> >> >> On Wed, Oct 1, 2014 at 5:22 PM, Sajini De Silva <[email protected]> wrote: >> >>> Hi, >>> >>> I'm getting following exception in MB when doing load testing. >>> >>> SEVERE: Exception processing: 81 >>> org.wso2.andes.kernel.distrupter.CassandraDataEvent@2d524d15 >>> com.datastax.driver.core.exceptions.NoHostAvailableException: All >>> host(s) tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>> at >>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >>> at >>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >>> at >>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >>> at com.datastax.driver.core.Session.execute(Session.java:111) >>> at >>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >>> at >>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.storeMessagePart(CQLBasedMessageStoreImpl.java:121) >>> at >>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:81) >>> at >>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:38) >>> at >>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:662) >>> >>> Does anyone knows the cause for this error? >>> >>> -- >>> Sajini De SIlva >>> Software Engineer; WSO2 Inc.; http://wso2.com , >>> Email: [email protected] >>> Blog: http://sajinid.blogspot.com/ >>> Git hub profile: https://github.com/sajinidesilva >>> >>> Phone: +94 712797729 >>> >>> >> >> >> -- >> Sajini De SIlva >> Software Engineer; WSO2 Inc.; http://wso2.com , >> Email: [email protected] >> Blog: http://sajinid.blogspot.com/ >> Git hub profile: https://github.com/sajinidesilva >> >> Phone: +94 712797729 >> >> > > > -- > Sajini De SIlva > Software Engineer; WSO2 Inc.; http://wso2.com , > Email: [email protected] > Blog: http://sajinid.blogspot.com/ > Git hub profile: https://github.com/sajinidesilva > > Phone: +94 712797729 > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Bhathiya Jayasekara* *Software Engineer,* *WSO2 inc., http://wso2.com <http://wso2.com>* *Phone: +94715478185 <%2B94715478185>* *LinkedIn: http://www.linkedin.com/in/bhathiyaj <http://www.linkedin.com/in/bhathiyaj>* *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>* *Blog: http://movingaheadblog.blogspot.com <http://movingaheadblog.blogspot.com/>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
