Hi,
This issue was fixed with following configurations.
*master-datasources.xml in MB node:*
<!-- external Cassandra data source. -->
<datasource>
<name>WSO2_CASSANDRA_DB</name>
<description>The datasource used for cassandra</description>
<jndiConfig>
<name>CassandraRepo</name>
</jndiConfig>
<definition type="CASSANDRA">
<configuration>
<async>false</async>
<clusterName>TestCluster</clusterName>
<compression>SNAPPY</compression>
<concurrency>100</concurrency>
<username>admin</username>
<password encrypted="true">admin</password>
<port>9042</port>
<maxConnections>100</maxConnections>
<hosts>
<host>127.0.0.1</host>
</hosts>
<loadBalancePolicy>
<exclusionThreshold>2.5</exclusionThreshold>
<latencyAware>true</latencyAware>
<minMeasure>100</minMeasure>
<policyName>RoundRobinPolicy</policyName>
<retryPeriod>10</retryPeriod>
<scale>2</scale>
</loadBalancePolicy>
<poolOptions>
<coreConnectionsForLocal>1000</coreConnectionsForLocal>
<coreConnectionsForRemote>1000</coreConnectionsForRemote>
<maxConnectionsForLocal>1000</maxConnectionsForLocal>
<maxConnectionsForRemote>1000</maxConnectionsForRemote>
<maxSimultaneousRequestsForLocal>1000</maxSimultaneousRequestsForLocal>
<maxSimultaneousRequestsForRemote>1000</maxSimultaneousRequestsForRemote>
<minSimultaneousRequestsForLocal>1000</minSimultaneousRequestsForLocal>
<minSimultaneousRequestsForRemote>1000</minSimultaneousRequestsForRemote>
</poolOptions>
<reconnectPolicy>
<baseDelayMs>3000</baseDelayMs>
<policyName>ConstantReconnectionPolicy</policyName>
</reconnectPolicy>
<socketOptions>
<connectTimeoutMillis>10000</connectTimeoutMillis>
<keepAlive>true</keepAlive>
<readTimeoutMillis>15000</readTimeoutMillis>
<tcpNoDelay>true</tcpNoDelay>
</socketOptions>
</configuration>
</definition>
</datasource>
*cassandra-env.sh in cassandra node:*
system_memory_in_mb="10240"
system_cpu_cores="6"
Tested with an SSD hard drive.
Thank you,
Sajini.
On Fri, Oct 3, 2014 at 12:23 PM, Sajini De Silva <[email protected]> wrote:
> Hi Prabath,
>
> This is the error we got while deleting messages.
>
> [2014-10-03 12:16:55,059] ERROR
> {org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl} - Error while
> deleting messages
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1 (Timeout during read))
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
> at
> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
> at
> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
> at com.datastax.driver.core.Session.execute(Session.java:111)
> at
> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
> at
> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
> at
> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
> at
> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
> at
> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
> at
> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
> read))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
> ... 3 more
> Oct 3, 2014 12:16:55 PM com.lmax.disruptor.FatalExceptionHandler
> handleEventException
> SEVERE: Exception processing: 12095
> org.wso2.andes.kernel.AndesAckData@5341cc77
> org.wso2.andes.kernel.AndesException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1 (Timeout during read))
> at
> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303)
> at
> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
> at
> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
> at
> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
> at
> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
> read))
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
> at
> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
> at
> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
> at com.datastax.driver.core.Session.execute(Session.java:111)
> at
> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
> at
> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
> ... 8 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
> read))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
> ... 3 more
> Exception in thread "pool-25-thread-1" java.lang.RuntimeException:
> org.wso2.andes.kernel.AndesException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1 (Timeout during read))
> at
> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.wso2.andes.kernel.AndesException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1 (Timeout during read))
> at
> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303)
> at
> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
> at
> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
> at
> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
> at
> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
> ... 3 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
> read))
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
> at
> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
> at
> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
> at com.datastax.driver.core.Session.execute(Session.java:111)
> at
> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
> at
> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
> ... 8 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
> read))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
> ... 3 more
>
>
> We tried this scenario increasing the number of threads in thread pool to
> 1000, but same error occurred.
>
> Thank you,
> Sajini.
>
>
> On Wed, Oct 1, 2014 at 5:22 PM, Sajini De Silva <[email protected]> wrote:
>
>> Hi,
>>
>> I'm getting following exception in MB when doing load testing.
>>
>> SEVERE: Exception processing: 81
>> org.wso2.andes.kernel.distrupter.CassandraDataEvent@2d524d15
>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>> at
>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>> at
>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>> at
>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>> at com.datastax.driver.core.Session.execute(Session.java:111)
>> at
>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>> at
>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.storeMessagePart(CQLBasedMessageStoreImpl.java:121)
>> at
>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:81)
>> at
>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:38)
>> at
>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Does anyone knows the cause for this error?
>>
>> --
>> Sajini De SIlva
>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>> Email: [email protected]
>> Blog: http://sajinid.blogspot.com/
>> Git hub profile: https://github.com/sajinidesilva
>>
>> Phone: +94 712797729
>>
>>
>
>
> --
> Sajini De SIlva
> Software Engineer; WSO2 Inc.; http://wso2.com ,
> Email: [email protected]
> Blog: http://sajinid.blogspot.com/
> Git hub profile: https://github.com/sajinidesilva
>
> Phone: +94 712797729
>
>
--
Sajini De SIlva
Software Engineer; WSO2 Inc.; http://wso2.com ,
Email: [email protected]
Blog: http://sajinid.blogspot.com/
Git hub profile: https://github.com/sajinidesilva
Phone: +94 712797729
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
