[
https://issues.apache.org/jira/browse/CASSANDRA-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354787#comment-15354787
]
Sam Tunnicliffe commented on CASSANDRA-12103:
---------------------------------------------
bq. then,why it's normal in DC2?we are using 128G memory,20 core CPU in DC1,any
advice for tuning this?
It's hard to say based on the information here (and JIRA is not the right forum
for that discussion). Perhaps the fact that there are 3 times more nodes in DC2
than DC1 is relevant? (Assuming that your other keyspaces aren't setup with
RF=n like {{system_auth}}). As far as tuning goes, some useful resources are:
https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html
http://blakeeggleston.com/cassandra-tuning-the-jvm-for-read-heavy-workloads.html
http://stackoverflow.com/questions/30207779/optimal-jvm-settings-for-cassandra
As for the OpsCenter agent issue, it's the heap that's exhausted there, so you
probably want to look at tuning that JVM too. Please note though, that the
Apache Cassandra JIRA is not the place to report issues with third party
software like OpsCenter. For that, you'll need to open a support ticket with
the vendor.
> Cassandra is hang and cqlsh was not able to login with OperationTimeout error
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-12103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12103
> Project: Cassandra
> Issue Type: Bug
> Components: Core, Local Write-Read Paths
> Environment: centos 6.5 cassandra 2.1.9
> Reporter: peng xiao
> Priority: Critical
> Attachments: system.log.2016-06-28_1257.gz
>
>
> Hi,
> We have two DCs(DC1 and DC2) with DC1 3 nodes and DC2 9 nodes.
> And we experienced a Timeout error today,all applications connected to DC1
> were hang and no response,even cqlsh was not able to log into any node in DC1.
> I restarted the 3 nodes in DC1,the problem was not resolved.
> Then we switched to DC2,then applications back to normal.
> Could you please help to take a look?
> Thanks
> many errors like below:
> ERROR [SharedPool-Worker-43] 2016-06-28 11:58:49,705 Message.java:538 -
> Unexpected exception during request; channel = [id: 0x87e315d6,
> /172.16.10.198:13604 => /172.16.11.13:9042]
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at org.apache.cassandra.auth.Auth.isExistingUser(Auth.java:86)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.service.ClientState.login(ClientState.java:206)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:82)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
> [apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
> [apache-cassandra-2.1.9.jar:2.1.9]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> [apache-cassandra-2.1.9.jar:2.1.9]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.9.jar:2.1.9]
> at java.lang.Thread.run(Thread.java:744) [na:1.8.0]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
