Rick Branson created CASSANDRA-9258:
---------------------------------------
Summary: Range movement causes CPU & performance impact
Key: CASSANDRA-9258
URL: https://issues.apache.org/jira/browse/CASSANDRA-9258
Project: Cassandra
Issue Type: Bug
Environment: Cassandra 2.1.4
Reporter: Rick Branson
Observing big CPU & latency regressions when doing range movements on clusters
with many tens of thousands of vnodes. See CPU usage increase by ~80% when a
single node is being replaced.
Top methods are:
1) Ljava/math/BigInteger;.compareTo in
Lorg/apache/cassandra/dht/ComparableObjectToken;.compareTo
2) Lcom/google/common/collect/AbstractMapBasedMultimap;.wrapCollection in
Lcom/google/common/collect/AbstractMapBasedMultimap$AsMap$AsMapIterator;.next
3) Lorg/apache/cassandra/db/DecoratedKey;.compareTo in
Lorg/apache/cassandra/dht/Range;.contains
Here's a sample stack from a thread dump:
"Thrift:50673" daemon prio=10 tid=0x00007f2f20164800 nid=0x3a04af runnable
[0x00007f2d878d0000]
java.lang.Thread.State: RUNNABLE
at org.apache.cassandra.dht.Range.isWrapAround(Range.java:260)
at org.apache.cassandra.dht.Range.contains(Range.java:51)
at org.apache.cassandra.dht.Range.contains(Range.java:110)
at
org.apache.cassandra.locator.TokenMetadata.pendingEndpointsFor(TokenMetadata.java:916)
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:775)
at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:541)
at
org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:616)
at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1101)
at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1083)
at
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:976)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3996)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3980)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
