Re: Is my cluster normal?

[daemeon reiydelle]

Those numbers, as I suspected, line up pretty well with your AWS
configuration and network latencies within AWS. It is clear that this is a
WRITE ONLY test. You might want to do a mixed (e.g. 50% read, 50% write)
test for sanity. Note that the test will populate the data BEFORE it begins
doing the read/write tests.

In a dedicated environment at a recent client, with 10gbit links (just
grabbing one casstest run from my archives) I see less than twice the
above. Note your latency max is the result of a stop-the-world garbage
collection. There were huge problems below because this particular run was
using 24gb (Cassandra 2.x) java heap.

op rate   : 21567 [WRITE:21567]
partition rate: 21567 [WRITE:21567]
row rate  : 21567 [WRITE:21567]
latency mean  : 9.3 [WRITE:9.3]
latency median: 7.7 [WRITE:7.7]
latency 95th percentile   : 13.2 [WRITE:13.2]
latency 99th percentile   : 32.6 [WRITE:32.6]
latency 99.9th percentile : 97.2 [WRITE:97.2]
latency max   : 14906.1 [WRITE:14906.1]
Total partitions  : 8333 [WRITE:8333]
Total errors  : 0 [WRITE:0]
total gc count: 705
total gc mb   : 1691132
total gc time (s) : 30
avg gc time(ms)   : 43
stdev gc time(ms) : 13
Total operation time  : 01:04:23


*...*



*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Thu, Jul 7, 2016 at 2:51 PM, Yuan Fang  wrote:

> Yes, here is my stress test result:
> Results:
> op rate   : 12200 [WRITE:12200]
> partition rate: 12200 [WRITE:12200]
> row rate  : 12200 [WRITE:12200]
> latency mean  : 16.4 [WRITE:16.4]
> latency median: 7.1 [WRITE:7.1]
> latency 95th percentile   : 38.1 [WRITE:38.1]
> latency 99th percentile   : 204.3 [WRITE:204.3]
> latency 99.9th percentile : 465.9 [WRITE:465.9]
> latency max   : 1408.4 [WRITE:1408.4]
> Total partitions  : 100 [WRITE:100]
> Total errors  : 0 [WRITE:0]
> total gc count: 0
> total gc mb   : 0
> total gc time (s) : 0
> avg gc time(ms)   : NaN
> stdev gc time(ms) : 0
> Total operation time  : 00:01:21
> END
>
> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:
>
>> Lots of variables you're leaving out.
>>
>> Depends on write size, if you're using logged batch or not, what
>> consistency level, what RF, if the writes come in bursts, etc, etc.
>> However, that's all sort of moot for determining "normal" really you need a
>> baseline as all those variables end up mattering a huge amount.
>>
>> I would suggest using Cassandra stress as a baseline and go from there
>> depending on what those numbers say (just pick the defaults).
>>
>> Sent from my iPhone
>>
>> On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:
>>
>> yes, it is about 8k writes per node.
>>
>>
>>
>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle 
>> wrote:
>>
>>> Are you saying 7k writes per node? or 30k writes per node?
>>>
>>>
>>> *...*
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>>
>>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:
>>>
 writes 30k/second is the main thing.


 On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
 wrote:

> Assuming you meant 100k, that likely for something with 16mb of
> storage (probably way small) where the data is more that 64k hence will 
> not
> fit into the row cache.
>
>
> *...*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
> <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang 
> wrote:
>
>>
>>
>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and
>> 600GB ssd EBS).
>> I can reach a cluster wide write requests of 30k/second and read
>> request about 100/second. The cluster OS load constantly above 10. Are
>> those normal?
>>
>> Thanks!
>>
>>
>> Best,
>>
>> Yuan
>>
>>
>

>>>
>>
>

Re: Is my cluster normal?

[Jeff Jirsa]

EBS iops scale with volume size.

 

A 600G EBS volume only guarantees 1800 iops – if you’re exhausting those on 
writes, you’re going to suffer on reads.

 

You have a 16G server, and probably a good chunk of that allocated to heap. 
Consequently, you have almost no page cache, so your reads are going to hit the 
disk. Your reads being very low is not uncommon if you have no page cache – the 
default settings for Cassandra (64k compression chunks) are really inefficient 
for small reads served off of disk. If you drop the compression chunk size (4k, 
for example), you’ll probably see your read throughput increase significantly, 
which will give you more iops for commitlog, so write throughput likely goes 
up, too.

 

 

 

From: Jonathan Haddad 
Reply-To: "user@cassandra.apache.org" 
Date: Thursday, July 7, 2016 at 6:54 PM
To: "user@cassandra.apache.org" 
Subject: Re: Is my cluster normal?

 

What's your CPU looking like? If it's low, check your IO with iostat or dstat. 
I know some people have used Ebs and say it's fine but ive been burned too many 
times. 

On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang  wrote:

Hi Riccardo, 

 

Very low IO-wait. About 0.3%.

No stolen CPU. It is a casssandra only instance. I did not see any dropped 
messages.

 

 

ubuntu@cassandra1:/mnt/data$ nodetool tpstats

Pool NameActive   Pending  Completed   Blocked  All 
time blocked

MutationStage 1 1  929509244 0  
   0

ViewMutationStage 0 0  0 0  
   0

ReadStage 4 04021570 0  
   0

RequestResponseStage  0 0  731477999 0  
   0

ReadRepairStage   0 0 165603 0  
   0

CounterMutationStage  0 0  0 0  
   0

MiscStage 0 0  0 0  
   0

CompactionExecutor255  92022 0  
   0

MemtableReclaimMemory 0 0   1736 0  
   0

PendingRangeCalculator0 0  6 0  
   0

GossipStage   0 0 345474 0  
   0

SecondaryIndexManagement  0 0  0 0  
   0

HintsDispatcher   0 0  4 0  
   0

MigrationStage0 0 35 0  
   0

MemtablePostFlush 0 0   1973 0  
   0

ValidationExecutor0 0  0 0  
   0

Sampler   0 0  0 0  
   0

MemtableFlushWriter   0 0   1736 0  
   0

InternalResponseStage 0 0   5311 0  
   0

AntiEntropyStage  0 0  0 0  
   0

CacheCleanupExecutor  0 0  0 0  
   0

Native-Transport-Requests   128   128  347508531 2  
15891862

 

Message type   Dropped

READ 0

RANGE_SLICE  0

_TRACE   0

HINT 0

MUTATION 0

COUNTER_MUTATION 0

BATCH_STORE  0

BATCH_REMOVE 0

REQUEST_RESPONSE 0

PAGED_RANGE  0

READ_REPAIR  0

 

 

 

 

 

On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari  wrote:

Hi Yuan, 

 

You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside from 
any Cassandra specific discussion a system load of 10 on a 4 threads machine is 
way too much in my opinion. If that is the running average system load I would 
look deeper into system details. Is that IO wait? Is that CPU Stolen? Is that a 
Cassandra only instance or are there other processes pushing the load?

What does your "nodetool tpstats" say? Hoe many dropped messages do you have?

 

Best,

 

On Fri, Jul 8, 2016 at 12:34 AM, Yuan Fang  wrote:

Thanks Ben! For the post, it seems they got a little better but similar result 
than i did. Good to know it. 

I am not sure if a little fine tuning of heap memory will help or not.  

 

 

On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater  wrote:

Hi Yuan, 

 

You might find this blog post a useful comparison:

Re: [RELEASE] Apache Cassandra 3.0.8 released

[Jake Luciani]

Sorry, I totally missed that.  Uploading now.

On Thu, Jul 7, 2016 at 4:51 AM, horschi  wrote:

> Same for 2.2.7.
>
> On Thu, Jul 7, 2016 at 10:49 AM, Julien Anguenot 
> wrote:
>
>> Hey,
>>
>> The Debian packages do not seem to have been published. Normal?
>>
>> Thank you.
>>
>>J.
>>
>> On Jul 6, 2016, at 4:20 PM, Jake Luciani  wrote:
>>
>> The Cassandra team is pleased to announce the release of Apache Cassandra
>> version 3.0.8.
>>
>> Apache Cassandra is a fully distributed database. It is the right choice
>> when you need scalability and high availability without compromising
>> performance.
>>
>>  http://cassandra.apache.org/
>>
>> Downloads of source and binary distributions are listed in our download
>> section:
>>
>>  http://cassandra.apache.org/download/
>>
>> This version is a bug fix release[1] on the 3.0 series. As always, please
>> pay
>> attention to the release notes[2] and Let us know[3] if you were to
>> encounter
>> any problem.
>>
>> Enjoy!
>>
>> [1]: http://goo.gl/DQpe4d (CHANGES.txt)
>> [2]: http://goo.gl/UISX1K (NEWS.txt)
>> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>>
>>
>>
>

Re: Is my cluster normal?

[Jonathan Haddad]

What's your CPU looking like? If it's low, check your IO with iostat or
dstat. I know some people have used Ebs and say it's fine but ive been
burned too many times.
On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang  wrote:

> Hi Riccardo,
>
> Very low IO-wait. About 0.3%.
> No stolen CPU. It is a casssandra only instance. I did not see any dropped
> messages.
>
>
> ubuntu@cassandra1:/mnt/data$ nodetool tpstats
> Pool NameActive   Pending  Completed   Blocked
>  All time blocked
> MutationStage 1 1  929509244 0
> 0
> ViewMutationStage 0 0  0 0
> 0
> ReadStage 4 04021570 0
> 0
> RequestResponseStage  0 0  731477999 0
> 0
> ReadRepairStage   0 0 165603 0
> 0
> CounterMutationStage  0 0  0 0
> 0
> MiscStage 0 0  0 0
> 0
> CompactionExecutor255  92022 0
> 0
> MemtableReclaimMemory 0 0   1736 0
> 0
> PendingRangeCalculator0 0  6 0
> 0
> GossipStage   0 0 345474 0
> 0
> SecondaryIndexManagement  0 0  0 0
> 0
> HintsDispatcher   0 0  4 0
> 0
> MigrationStage0 0 35 0
> 0
> MemtablePostFlush 0 0   1973 0
> 0
> ValidationExecutor0 0  0 0
> 0
> Sampler   0 0  0 0
> 0
> MemtableFlushWriter   0 0   1736 0
> 0
> InternalResponseStage 0 0   5311 0
> 0
> AntiEntropyStage  0 0  0 0
> 0
> CacheCleanupExecutor  0 0  0 0
> 0
> Native-Transport-Requests   128   128  347508531 2
>  15891862
>
> Message type   Dropped
> READ 0
> RANGE_SLICE  0
> _TRACE   0
> HINT 0
> MUTATION 0
> COUNTER_MUTATION 0
> BATCH_STORE  0
> BATCH_REMOVE 0
> REQUEST_RESPONSE 0
> PAGED_RANGE  0
> READ_REPAIR  0
>
>
>
>
>
> On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari 
> wrote:
>
>> Hi Yuan,
>>
>> You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside
>> from any Cassandra specific discussion a system load of 10 on a 4 threads
>> machine is way too much in my opinion. If that is the running average
>> system load I would look deeper into system details. Is that IO wait? Is
>> that CPU Stolen? Is that a Cassandra only instance or are there other
>> processes pushing the load?
>> What does your "nodetool tpstats" say? Hoe many dropped messages do you
>> have?
>>
>> Best,
>>
>> On Fri, Jul 8, 2016 at 12:34 AM, Yuan Fang  wrote:
>>
>>> Thanks Ben! For the post, it seems they got a little better but similar
>>> result than i did. Good to know it.
>>> I am not sure if a little fine tuning of heap memory will help or not.
>>>
>>>
>>> On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater 
>>> wrote:
>>>
 Hi Yuan,

 You might find this blog post a useful comparison:

 https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/

 Although the focus is on Spark and Cassandra and multi-DC there are
 also some single DC benchmarks of m4.xl clusters plus some discussion of
 how we went about benchmarking.

 Cheers
 Ben


 On Fri, 8 Jul 2016 at 07:52 Yuan Fang  wrote:

> Yes, here is my stress test result:
> Results:
> op rate   : 12200 [WRITE:12200]
> partition rate: 12200 [WRITE:12200]
> row rate  : 12200 [WRITE:12200]
> latency mean  : 16.4 [WRITE:16.4]
> latency median: 7.1 [WRITE:7.1]
> latency 95th percentile   : 38.1 [WRITE:38.1]
> latency 99th percentile   : 204.3 [WRITE:204.3]
> latency 99.9th percentile : 465.9 [WRITE:465.9]
> latency max   : 1408.4 [WRITE:1408.4]
> Total partitions  : 100 [WRITE:100]
> Total errors  : 0 

Re: How to get current value of commitlog_segment_size_in_mb?

[Stone Fang]

commitlog_segment_size_in_mb is configed in cassandra.yaml.dont  think that
it wolud be stored in Cassandra system table.
the following is the introduction of Cassandra System table.
https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_query_system_c.html


On Fri, Jul 8, 2016 at 4:23 AM, Jaydeep Chovatia  wrote:

> Hi,
>
> In my project I need to read current value for
> "commitlog_segment_size_in_mb", I am looking for CQL query to do this. Any
> idea if this information gets stored in any of the Cassandra system table?
>
> Thanks,
> Jaydeep
>

Re: Is my cluster normal?

[Yuan Fang]

Hi Riccardo,

Very low IO-wait. About 0.3%.
No stolen CPU. It is a casssandra only instance. I did not see any dropped
messages.


ubuntu@cassandra1:/mnt/data$ nodetool tpstats
Pool NameActive   Pending  Completed   Blocked  All
time blocked
MutationStage 1 1  929509244 0
0
ViewMutationStage 0 0  0 0
0
ReadStage 4 04021570 0
0
RequestResponseStage  0 0  731477999 0
0
ReadRepairStage   0 0 165603 0
0
CounterMutationStage  0 0  0 0
0
MiscStage 0 0  0 0
0
CompactionExecutor255  92022 0
0
MemtableReclaimMemory 0 0   1736 0
0
PendingRangeCalculator0 0  6 0
0
GossipStage   0 0 345474 0
0
SecondaryIndexManagement  0 0  0 0
0
HintsDispatcher   0 0  4 0
0
MigrationStage0 0 35 0
0
MemtablePostFlush 0 0   1973 0
0
ValidationExecutor0 0  0 0
0
Sampler   0 0  0 0
0
MemtableFlushWriter   0 0   1736 0
0
InternalResponseStage 0 0   5311 0
0
AntiEntropyStage  0 0  0 0
0
CacheCleanupExecutor  0 0  0 0
0
Native-Transport-Requests   128   128  347508531 2
 15891862

Message type   Dropped
READ 0
RANGE_SLICE  0
_TRACE   0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE  0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE  0
READ_REPAIR  0





On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari  wrote:

> Hi Yuan,
>
> You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside
> from any Cassandra specific discussion a system load of 10 on a 4 threads
> machine is way too much in my opinion. If that is the running average
> system load I would look deeper into system details. Is that IO wait? Is
> that CPU Stolen? Is that a Cassandra only instance or are there other
> processes pushing the load?
> What does your "nodetool tpstats" say? Hoe many dropped messages do you
> have?
>
> Best,
>
> On Fri, Jul 8, 2016 at 12:34 AM, Yuan Fang  wrote:
>
>> Thanks Ben! For the post, it seems they got a little better but similar
>> result than i did. Good to know it.
>> I am not sure if a little fine tuning of heap memory will help or not.
>>
>>
>> On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater 
>> wrote:
>>
>>> Hi Yuan,
>>>
>>> You might find this blog post a useful comparison:
>>>
>>> https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/
>>>
>>> Although the focus is on Spark and Cassandra and multi-DC there are also
>>> some single DC benchmarks of m4.xl clusters plus some discussion of how we
>>> went about benchmarking.
>>>
>>> Cheers
>>> Ben
>>>
>>>
>>> On Fri, 8 Jul 2016 at 07:52 Yuan Fang  wrote:
>>>
 Yes, here is my stress test result:
 Results:
 op rate   : 12200 [WRITE:12200]
 partition rate: 12200 [WRITE:12200]
 row rate  : 12200 [WRITE:12200]
 latency mean  : 16.4 [WRITE:16.4]
 latency median: 7.1 [WRITE:7.1]
 latency 95th percentile   : 38.1 [WRITE:38.1]
 latency 99th percentile   : 204.3 [WRITE:204.3]
 latency 99.9th percentile : 465.9 [WRITE:465.9]
 latency max   : 1408.4 [WRITE:1408.4]
 Total partitions  : 100 [WRITE:100]
 Total errors  : 0 [WRITE:0]
 total gc count: 0
 total gc mb   : 0
 total gc time (s) : 0
 avg gc time(ms)   : NaN
 stdev gc time(ms) : 0
 Total operation time  : 00:01:21
 END

 On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:

> Lots of variables you're leaving out.
>
> Depends on write size, if you're using logged batch or not, 

Re: Is my cluster normal?

[Riccardo Ferrari]

Hi Yuan,

You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside
from any Cassandra specific discussion a system load of 10 on a 4 threads
machine is way too much in my opinion. If that is the running average
system load I would look deeper into system details. Is that IO wait? Is
that CPU Stolen? Is that a Cassandra only instance or are there other
processes pushing the load?
What does your "nodetool tpstats" say? Hoe many dropped messages do you
have?

Best,

On Fri, Jul 8, 2016 at 12:34 AM, Yuan Fang  wrote:

> Thanks Ben! For the post, it seems they got a little better but similar
> result than i did. Good to know it.
> I am not sure if a little fine tuning of heap memory will help or not.
>
>
> On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater 
> wrote:
>
>> Hi Yuan,
>>
>> You might find this blog post a useful comparison:
>>
>> https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/
>>
>> Although the focus is on Spark and Cassandra and multi-DC there are also
>> some single DC benchmarks of m4.xl clusters plus some discussion of how we
>> went about benchmarking.
>>
>> Cheers
>> Ben
>>
>>
>> On Fri, 8 Jul 2016 at 07:52 Yuan Fang  wrote:
>>
>>> Yes, here is my stress test result:
>>> Results:
>>> op rate   : 12200 [WRITE:12200]
>>> partition rate: 12200 [WRITE:12200]
>>> row rate  : 12200 [WRITE:12200]
>>> latency mean  : 16.4 [WRITE:16.4]
>>> latency median: 7.1 [WRITE:7.1]
>>> latency 95th percentile   : 38.1 [WRITE:38.1]
>>> latency 99th percentile   : 204.3 [WRITE:204.3]
>>> latency 99.9th percentile : 465.9 [WRITE:465.9]
>>> latency max   : 1408.4 [WRITE:1408.4]
>>> Total partitions  : 100 [WRITE:100]
>>> Total errors  : 0 [WRITE:0]
>>> total gc count: 0
>>> total gc mb   : 0
>>> total gc time (s) : 0
>>> avg gc time(ms)   : NaN
>>> stdev gc time(ms) : 0
>>> Total operation time  : 00:01:21
>>> END
>>>
>>> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:
>>>
 Lots of variables you're leaving out.

 Depends on write size, if you're using logged batch or not, what
 consistency level, what RF, if the writes come in bursts, etc, etc.
 However, that's all sort of moot for determining "normal" really you need a
 baseline as all those variables end up mattering a huge amount.

 I would suggest using Cassandra stress as a baseline and go from there
 depending on what those numbers say (just pick the defaults).

 Sent from my iPhone

 On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:

 yes, it is about 8k writes per node.



 On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle 
 wrote:

> Are you saying 7k writes per node? or 30k writes per node?
>
>
> *...*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
> <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang 
> wrote:
>
>> writes 30k/second is the main thing.
>>
>>
>> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle > > wrote:
>>
>>> Assuming you meant 100k, that likely for something with 16mb of
>>> storage (probably way small) where the data is more that 64k hence will 
>>> not
>>> fit into the row cache.
>>>
>>>
>>> *...*
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>>
>>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang 
>>> wrote:
>>>


 I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and
 600GB ssd EBS).
 I can reach a cluster wide write requests of 30k/second and read
 request about 100/second. The cluster OS load constantly above 10. Are
 those normal?

 Thanks!


 Best,

 Yuan


>>>
>>
>

>>> --
>> 
>> Ben Slater
>> Chief Product Officer
>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>> +61 437 929 798
>>
>
>

Re: Is my cluster normal?

[Yuan Fang]

Thanks Ben! For the post, it seems they got a little better but similar
result than i did. Good to know it.
I am not sure if a little fine tuning of heap memory will help or not.

On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater 
wrote:

> Hi Yuan,
>
> You might find this blog post a useful comparison:
>
> https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/
>
> Although the focus is on Spark and Cassandra and multi-DC there are also
> some single DC benchmarks of m4.xl clusters plus some discussion of how we
> went about benchmarking.
>
> Cheers
> Ben
>
>
> On Fri, 8 Jul 2016 at 07:52 Yuan Fang  wrote:
>
>> Yes, here is my stress test result:
>> Results:
>> op rate   : 12200 [WRITE:12200]
>> partition rate: 12200 [WRITE:12200]
>> row rate  : 12200 [WRITE:12200]
>> latency mean  : 16.4 [WRITE:16.4]
>> latency median: 7.1 [WRITE:7.1]
>> latency 95th percentile   : 38.1 [WRITE:38.1]
>> latency 99th percentile   : 204.3 [WRITE:204.3]
>> latency 99.9th percentile : 465.9 [WRITE:465.9]
>> latency max   : 1408.4 [WRITE:1408.4]
>> Total partitions  : 100 [WRITE:100]
>> Total errors  : 0 [WRITE:0]
>> total gc count: 0
>> total gc mb   : 0
>> total gc time (s) : 0
>> avg gc time(ms)   : NaN
>> stdev gc time(ms) : 0
>> Total operation time  : 00:01:21
>> END
>>
>> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:
>>
>>> Lots of variables you're leaving out.
>>>
>>> Depends on write size, if you're using logged batch or not, what
>>> consistency level, what RF, if the writes come in bursts, etc, etc.
>>> However, that's all sort of moot for determining "normal" really you need a
>>> baseline as all those variables end up mattering a huge amount.
>>>
>>> I would suggest using Cassandra stress as a baseline and go from there
>>> depending on what those numbers say (just pick the defaults).
>>>
>>> Sent from my iPhone
>>>
>>> On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:
>>>
>>> yes, it is about 8k writes per node.
>>>
>>>
>>>
>>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle 
>>> wrote:
>>>
 Are you saying 7k writes per node? or 30k writes per node?


 *...*



 *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
 <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
 <%28%2B44%29%20%280%29%2020%208144%209872>*

 On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang 
 wrote:

> writes 30k/second is the main thing.
>
>
> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
> wrote:
>
>> Assuming you meant 100k, that likely for something with 16mb of
>> storage (probably way small) where the data is more that 64k hence will 
>> not
>> fit into the row cache.
>>
>>
>> *...*
>>
>>
>>
>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>
>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang 
>> wrote:
>>
>>>
>>>
>>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and
>>> 600GB ssd EBS).
>>> I can reach a cluster wide write requests of 30k/second and read
>>> request about 100/second. The cluster OS load constantly above 10. Are
>>> those normal?
>>>
>>> Thanks!
>>>
>>>
>>> Best,
>>>
>>> Yuan
>>>
>>>
>>
>

>>>
>> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>

Re: Is my cluster normal?

[Yuan Fang]

Hi Ryan,


The version of cassandra is 3.0.6 and
java version "1.8.0_91"

Yuan

On Thu, Jul 7, 2016 at 3:11 PM, Ryan Svihla  wrote:

> what version of cassandra and java?
>
> Regards,
>
> Ryan Svihla
>
> On Jul 7, 2016, at 4:51 PM, Yuan Fang  wrote:
>
> Yes, here is my stress test result:
> Results:
> op rate   : 12200 [WRITE:12200]
> partition rate: 12200 [WRITE:12200]
> row rate  : 12200 [WRITE:12200]
> latency mean  : 16.4 [WRITE:16.4]
> latency median: 7.1 [WRITE:7.1]
> latency 95th percentile   : 38.1 [WRITE:38.1]
> latency 99th percentile   : 204.3 [WRITE:204.3]
> latency 99.9th percentile : 465.9 [WRITE:465.9]
> latency max   : 1408.4 [WRITE:1408.4]
> Total partitions  : 100 [WRITE:100]
> Total errors  : 0 [WRITE:0]
> total gc count: 0
> total gc mb   : 0
> total gc time (s) : 0
> avg gc time(ms)   : NaN
> stdev gc time(ms) : 0
> Total operation time  : 00:01:21
> END
>
> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:
>
>> Lots of variables you're leaving out.
>>
>> Depends on write size, if you're using logged batch or not, what
>> consistency level, what RF, if the writes come in bursts, etc, etc.
>> However, that's all sort of moot for determining "normal" really you need a
>> baseline as all those variables end up mattering a huge amount.
>>
>> I would suggest using Cassandra stress as a baseline and go from there
>> depending on what those numbers say (just pick the defaults).
>>
>> Sent from my iPhone
>>
>> On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:
>>
>> yes, it is about 8k writes per node.
>>
>>
>>
>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle 
>> wrote:
>>
>>> Are you saying 7k writes per node? or 30k writes per node?
>>>
>>>
>>> *...*
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>>
>>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:
>>>
 writes 30k/second is the main thing.


 On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
 wrote:

> Assuming you meant 100k, that likely for something with 16mb of
> storage (probably way small) where the data is more that 64k hence will 
> not
> fit into the row cache.
>
>
> *...*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
> <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang 
> wrote:
>
>>
>>
>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and
>> 600GB ssd EBS).
>> I can reach a cluster wide write requests of 30k/second and read
>> request about 100/second. The cluster OS load constantly above 10. Are
>> those normal?
>>
>> Thanks!
>>
>>
>> Best,
>>
>> Yuan
>>
>>
>

>>>
>>
>

Re: Is my cluster normal?

[Ryan Svihla]

what version of cassandra and java?

Regards,

Ryan Svihla

> On Jul 7, 2016, at 4:51 PM, Yuan Fang  wrote:
> 
> Yes, here is my stress test result:
> Results:
> op rate   : 12200 [WRITE:12200]
> partition rate: 12200 [WRITE:12200]
> row rate  : 12200 [WRITE:12200]
> latency mean  : 16.4 [WRITE:16.4]
> latency median: 7.1 [WRITE:7.1]
> latency 95th percentile   : 38.1 [WRITE:38.1]
> latency 99th percentile   : 204.3 [WRITE:204.3]
> latency 99.9th percentile : 465.9 [WRITE:465.9]
> latency max   : 1408.4 [WRITE:1408.4]
> Total partitions  : 100 [WRITE:100]
> Total errors  : 0 [WRITE:0]
> total gc count: 0
> total gc mb   : 0
> total gc time (s) : 0
> avg gc time(ms)   : NaN
> stdev gc time(ms) : 0
> Total operation time  : 00:01:21
> END
> 
>> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:
>> Lots of variables you're leaving out.
>> 
>> Depends on write size, if you're using logged batch or not, what consistency 
>> level, what RF, if the writes come in bursts, etc, etc. However, that's all 
>> sort of moot for determining "normal" really you need a baseline as all 
>> those variables end up mattering a huge amount.
>> 
>> I would suggest using Cassandra stress as a baseline and go from there 
>> depending on what those numbers say (just pick the defaults).
>> 
>> Sent from my iPhone
>> 
>>> On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:
>>> 
>>> yes, it is about 8k writes per node.
>>> 
>>> 
>>> 
 On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle  
 wrote:
 Are you saying 7k writes per node? or 30k writes per node?
 
 
 ...
 
 Daemeon C.M. Reiydelle
 USA (+1) 415.501.0198
 London (+44) (0) 20 8144 9872
 
> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:
> writes 30k/second is the main thing.
> 
> 
>> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle  
>> wrote:
>> Assuming you meant 100k, that likely for something with 16mb of storage 
>> (probably way small) where the data is more that 64k hence will not fit 
>> into the row cache.
>> 
>> 
>> ...
>> 
>> Daemeon C.M. Reiydelle
>> USA (+1) 415.501.0198
>> London (+44) (0) 20 8144 9872
>> 
>>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang  wrote:
>>> 
>>> 
>>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB 
>>> ssd EBS).
>>> I can reach a cluster wide write requests of 30k/second and read 
>>> request about 100/second. The cluster OS load constantly above 10. Are 
>>> those normal?
>>> 
>>> Thanks!
>>> 
>>> 
>>> Best,
>>> 
>>> Yuan 
>>> 
>> 
> 
 
>>> 
> 

unsubscribe

[Shashi Yachavaram]

Re: Is my cluster normal?

[Ben Slater]

Hi Yuan,

You might find this blog post a useful comparison:
https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/

Although the focus is on Spark and Cassandra and multi-DC there are also
some single DC benchmarks of m4.xl clusters plus some discussion of how we
went about benchmarking.

Cheers
Ben


On Fri, 8 Jul 2016 at 07:52 Yuan Fang  wrote:

> Yes, here is my stress test result:
> Results:
> op rate   : 12200 [WRITE:12200]
> partition rate: 12200 [WRITE:12200]
> row rate  : 12200 [WRITE:12200]
> latency mean  : 16.4 [WRITE:16.4]
> latency median: 7.1 [WRITE:7.1]
> latency 95th percentile   : 38.1 [WRITE:38.1]
> latency 99th percentile   : 204.3 [WRITE:204.3]
> latency 99.9th percentile : 465.9 [WRITE:465.9]
> latency max   : 1408.4 [WRITE:1408.4]
> Total partitions  : 100 [WRITE:100]
> Total errors  : 0 [WRITE:0]
> total gc count: 0
> total gc mb   : 0
> total gc time (s) : 0
> avg gc time(ms)   : NaN
> stdev gc time(ms) : 0
> Total operation time  : 00:01:21
> END
>
> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:
>
>> Lots of variables you're leaving out.
>>
>> Depends on write size, if you're using logged batch or not, what
>> consistency level, what RF, if the writes come in bursts, etc, etc.
>> However, that's all sort of moot for determining "normal" really you need a
>> baseline as all those variables end up mattering a huge amount.
>>
>> I would suggest using Cassandra stress as a baseline and go from there
>> depending on what those numbers say (just pick the defaults).
>>
>> Sent from my iPhone
>>
>> On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:
>>
>> yes, it is about 8k writes per node.
>>
>>
>>
>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle 
>> wrote:
>>
>>> Are you saying 7k writes per node? or 30k writes per node?
>>>
>>>
>>> *...*
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>>
>>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:
>>>
 writes 30k/second is the main thing.


 On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
 wrote:

> Assuming you meant 100k, that likely for something with 16mb of
> storage (probably way small) where the data is more that 64k hence will 
> not
> fit into the row cache.
>
>
> *...*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
> <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang 
> wrote:
>
>>
>>
>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and
>> 600GB ssd EBS).
>> I can reach a cluster wide write requests of 30k/second and read
>> request about 100/second. The cluster OS load constantly above 10. Are
>> those normal?
>>
>> Thanks!
>>
>>
>> Best,
>>
>> Yuan
>>
>>
>

>>>
>>
> --

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798

Re: Is my cluster normal?

[Yuan Fang]

Yes, here is my stress test result:
Results:
op rate   : 12200 [WRITE:12200]
partition rate: 12200 [WRITE:12200]
row rate  : 12200 [WRITE:12200]
latency mean  : 16.4 [WRITE:16.4]
latency median: 7.1 [WRITE:7.1]
latency 95th percentile   : 38.1 [WRITE:38.1]
latency 99th percentile   : 204.3 [WRITE:204.3]
latency 99.9th percentile : 465.9 [WRITE:465.9]
latency max   : 1408.4 [WRITE:1408.4]
Total partitions  : 100 [WRITE:100]
Total errors  : 0 [WRITE:0]
total gc count: 0
total gc mb   : 0
total gc time (s) : 0
avg gc time(ms)   : NaN
stdev gc time(ms) : 0
Total operation time  : 00:01:21
END

On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla  wrote:

> Lots of variables you're leaving out.
>
> Depends on write size, if you're using logged batch or not, what
> consistency level, what RF, if the writes come in bursts, etc, etc.
> However, that's all sort of moot for determining "normal" really you need a
> baseline as all those variables end up mattering a huge amount.
>
> I would suggest using Cassandra stress as a baseline and go from there
> depending on what those numbers say (just pick the defaults).
>
> Sent from my iPhone
>
> On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:
>
> yes, it is about 8k writes per node.
>
>
>
> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle 
> wrote:
>
>> Are you saying 7k writes per node? or 30k writes per node?
>>
>>
>> *...*
>>
>>
>>
>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>
>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:
>>
>>> writes 30k/second is the main thing.
>>>
>>>
>>> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
>>> wrote:
>>>
 Assuming you meant 100k, that likely for something with 16mb of storage
 (probably way small) where the data is more that 64k hence will not fit
 into the row cache.


 *...*



 *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
 <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
 <%28%2B44%29%20%280%29%2020%208144%209872>*

 On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang 
 wrote:

>
>
> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and
> 600GB ssd EBS).
> I can reach a cluster wide write requests of 30k/second and read
> request about 100/second. The cluster OS load constantly above 10. Are
> those normal?
>
> Thanks!
>
>
> Best,
>
> Yuan
>
>

>>>
>>
>

Re: Is my cluster normal?

[Ryan Svihla]

Lots of variables you're leaving out.

Depends on write size, if you're using logged batch or not, what consistency 
level, what RF, if the writes come in bursts, etc, etc. However, that's all 
sort of moot for determining "normal" really you need a baseline as all those 
variables end up mattering a huge amount.

I would suggest using Cassandra stress as a baseline and go from there 
depending on what those numbers say (just pick the defaults).

Sent from my iPhone

> On Jul 7, 2016, at 4:39 PM, Yuan Fang  wrote:
> 
> yes, it is about 8k writes per node.
> 
> 
> 
>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle  wrote:
>> Are you saying 7k writes per node? or 30k writes per node?
>> 
>> 
>> ...
>> 
>> Daemeon C.M. Reiydelle
>> USA (+1) 415.501.0198
>> London (+44) (0) 20 8144 9872
>> 
>>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:
>>> writes 30k/second is the main thing.
>>> 
>>> 
 On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle  
 wrote:
 Assuming you meant 100k, that likely for something with 16mb of storage 
 (probably way small) where the data is more that 64k hence will not fit 
 into the row cache.
 
 
 ...
 
 Daemeon C.M. Reiydelle
 USA (+1) 415.501.0198
 London (+44) (0) 20 8144 9872
 
> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang  wrote:
> 
> 
> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB 
> ssd EBS).
> I can reach a cluster wide write requests of 30k/second and read request 
> about 100/second. The cluster OS load constantly above 10. Are those 
> normal?
> 
> Thanks!
> 
> 
> Best,
> 
> Yuan 
> 

Re: Is my cluster normal?

[Yuan Fang]

yes, it is about 8k writes per node.



On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle 
wrote:

> Are you saying 7k writes per node? or 30k writes per node?
>
>
> *...*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
> <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:
>
>> writes 30k/second is the main thing.
>>
>>
>> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
>> wrote:
>>
>>> Assuming you meant 100k, that likely for something with 16mb of storage
>>> (probably way small) where the data is more that 64k hence will not fit
>>> into the row cache.
>>>
>>>
>>> *...*
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>>
>>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang  wrote:
>>>


 I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB
 ssd EBS).
 I can reach a cluster wide write requests of 30k/second and read
 request about 100/second. The cluster OS load constantly above 10. Are
 those normal?

 Thanks!


 Best,

 Yuan


>>>
>>
>

Re: Is my cluster normal?

[daemeon reiydelle]

Are you saying 7k writes per node? or 30k writes per node?


*...*



*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang  wrote:

> writes 30k/second is the main thing.
>
>
> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
> wrote:
>
>> Assuming you meant 100k, that likely for something with 16mb of storage
>> (probably way small) where the data is more that 64k hence will not fit
>> into the row cache.
>>
>>
>> *...*
>>
>>
>>
>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>
>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang  wrote:
>>
>>>
>>>
>>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB
>>> ssd EBS).
>>> I can reach a cluster wide write requests of 30k/second and read request
>>> about 100/second. The cluster OS load constantly above 10. Are those normal?
>>>
>>> Thanks!
>>>
>>>
>>> Best,
>>>
>>> Yuan
>>>
>>>
>>
>

Re: Is my cluster normal?

[Yuan Fang]

writes 30k/second is the main thing.


On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle 
wrote:

> Assuming you meant 100k, that likely for something with 16mb of storage
> (probably way small) where the data is more that 64k hence will not fit
> into the row cache.
>
>
> *...*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
> <%28%2B44%29%20%280%29%2020%208144%209872>*
>
> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang  wrote:
>
>>
>>
>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB
>> ssd EBS).
>> I can reach a cluster wide write requests of 30k/second and read request
>> about 100/second. The cluster OS load constantly above 10. Are those normal?
>>
>> Thanks!
>>
>>
>> Best,
>>
>> Yuan
>>
>>
>

Re: Is my cluster normal?

[daemeon reiydelle]

Assuming you meant 100k, that likely for something with 16mb of storage
(probably way small) where the data is more that 64k hence will not fit
into the row cache.


*...*



*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang  wrote:

>
>
> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB
> ssd EBS).
> I can reach a cluster wide write requests of 30k/second and read request
> about 100/second. The cluster OS load constantly above 10. Are those normal?
>
> Thanks!
>
>
> Best,
>
> Yuan
>
>

Is my cluster normal?

[Yuan Fang]

I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB ssd
EBS).
I can reach a cluster wide write requests of 30k/second and read request
about 100/second. The cluster OS load constantly above 10. Are those normal?

Thanks!


Best,

Yuan

How to get current value of commitlog_segment_size_in_mb?

[Jaydeep Chovatia]

Hi,

In my project I need to read current value for
"commitlog_segment_size_in_mb", I am looking for CQL query to do this. Any
idea if this information gets stored in any of the Cassandra system table?

Thanks,
Jaydeep

Re: Debugging high tail read latencies (internal timeout)

[daemeon reiydelle]

Hmm. Would you mind looking at your network interface (appropriate netstat
commands). if I am right you will be seeing packet errors, drops, retries,
packet out of window receives, etc.

What you may be missing is that you reported zero DROPPED latency. Not mean
LATENCY. Check your netstats. ANY VALUE CHANGE IS BAD (except total
read/write byte counts). If your network guys say otherwise, escalate to
someone who undertands tcp retry and sliding window.



*...*



*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Thu, Jul 7, 2016 at 11:35 AM, Bryan Cheng  wrote:

> Hi Nimi,
>
> My suspicions would probably lie somewhere between GC and large partitions.
>
> The first tool would probably be a trace but if you experience full client
> timeouts from dropped messages you may find it hard to find the issue. You
> can try running the trace with cqlsh's timeouts cranked all the way against
> the local node with CL=ONE to try to force the local machine to answer.
>
> What does nodetool tpstats report for dropped message counts? Are they
> very high? Primarily restricted to READ, or including MUTATION, etc. ?
>
> Are there specific PK's that trigger this behavior, either all the time or
> more consistently? That would finger either very large partition sizes or
> potentially bad hardware on a node. cfhistograms will show you various
> percentile partition sizes and your max as well.
>
> GC should be accessible via JMX and also you should have GCInspector logs
> in cassandra/system.log that should give you per-collection breakdowns.
>
> --Bryan
>
>
> On Wed, Jul 6, 2016 at 6:22 PM, Nimi Wariboko Jr 
> wrote:
>
>> Hi,
>>
>> I've begun experiencing very high tail latencies across my clusters.
>> While Cassandra's internal metrics report <1ms read latencies, measuring
>> responses from within the driver in my applications (roundtrips of
>> query/execute frames), have 90% round trip times of up to a second for very
>> basic queries (SELECT a,b FROM table WHERE pk=x).
>>
>> I've been studying the logs to try and get a handle on what could be
>> going wrong. I don't think there are GC issues, but the logs mention
>> dropped messages due to timeouts while the threadpools are nearly empty -
>>
>> https://gist.github.com/nemothekid/28b2a8e8353b3e60d7bbf390ed17987c
>>
>> Relevant line:
>> REQUEST_RESPONSE messages were dropped in last 5000 ms: 1 for internal
>> timeout and 0 for cross node timeout. Mean internal dropped latency: 54930
>> ms and Mean cross-node dropped latency: 0 ms
>>
>> Are there any tools I can use to start to understand what is causing
>> these issues?
>>
>> Nimi
>>
>>
>

Re: Debugging high tail read latencies (internal timeout)

[Bryan Cheng]

Hi Nimi,

My suspicions would probably lie somewhere between GC and large partitions.

The first tool would probably be a trace but if you experience full client
timeouts from dropped messages you may find it hard to find the issue. You
can try running the trace with cqlsh's timeouts cranked all the way against
the local node with CL=ONE to try to force the local machine to answer.

What does nodetool tpstats report for dropped message counts? Are they very
high? Primarily restricted to READ, or including MUTATION, etc. ?

Are there specific PK's that trigger this behavior, either all the time or
more consistently? That would finger either very large partition sizes or
potentially bad hardware on a node. cfhistograms will show you various
percentile partition sizes and your max as well.

GC should be accessible via JMX and also you should have GCInspector logs
in cassandra/system.log that should give you per-collection breakdowns.

--Bryan


On Wed, Jul 6, 2016 at 6:22 PM, Nimi Wariboko Jr 
wrote:

> Hi,
>
> I've begun experiencing very high tail latencies across my clusters. While
> Cassandra's internal metrics report <1ms read latencies, measuring
> responses from within the driver in my applications (roundtrips of
> query/execute frames), have 90% round trip times of up to a second for very
> basic queries (SELECT a,b FROM table WHERE pk=x).
>
> I've been studying the logs to try and get a handle on what could be going
> wrong. I don't think there are GC issues, but the logs mention dropped
> messages due to timeouts while the threadpools are nearly empty -
>
> https://gist.github.com/nemothekid/28b2a8e8353b3e60d7bbf390ed17987c
>
> Relevant line:
> REQUEST_RESPONSE messages were dropped in last 5000 ms: 1 for internal
> timeout and 0 for cross node timeout. Mean internal dropped latency: 54930
> ms and Mean cross-node dropped latency: 0 ms
>
> Are there any tools I can use to start to understand what is causing these
> issues?
>
> Nimi
>
>

Divergence from Cassandra partition_count and partition keys count

[Alexandre Santana]

Hello,

Im trying to use spark with cassandra and it was oddly generating several
spark jobs because spark follow the guidelines generated by
partitions_count and mean_partition_size. The problem is that I have a very
small table (300MB) with only 16 distinct partition keys running on a
single C* node.

Even though, when I query partitions_count and mean_partition_size I get
the following numbers:
 91959383 and 256 respectively. That's way higher than what I have and
"nodetools cfstats" shows me that the tables are correcly way smaller.

Can someone explain me why this happens or, if it is an error, how to fix
it?

Obs: Im using cassandra 2.2.6

Thanks in Advance,
Alexandre Santana

Re: DTCS SSTable count issue

[Jeff Jirsa]

48 sstables isn’t unreasonable in a DTCS table. It will continue to grow over 
time, but ideally data will expire as it nears your 90 day TTL and those tables 
should start dropping away as they age.

 

3.0.7 introduces an alternative to DTCS you may find easier to use called TWCS. 
It will almost certainly help address the growing sstable count.  

 

 

 

From: Riccardo Ferrari 
Reply-To: "user@cassandra.apache.org" 
Date: Thursday, July 7, 2016 at 6:49 AM
To: "user@cassandra.apache.org" 
Subject: DTCS SSTable count issue

 

Hi everyone, 

 

This is my first question, apologize may I do something wrong.

 

I have a small Cassandra cluster build upon 3 nodes. Originally born as 2.0.X 
cluster was upgraded to 2.0.15 then 2.1.13 and finally to 3.0.4 recently 3.0.6. 
Ubuntu is the OS.

 

There are few tables that have DateTieredCompactionStrategy and are suffering 
of constantly growing SSTable count. I have the feeling this has something to 
do with the upgrade however I need some hint on how to debug this issue.

 

Tables are created like:

CREATE TABLE  (

 ...

PRIMARY KEY (...)

) WITH CLUSTERING ORDER BY (...)

AND bloom_filter_fp_chance = 0.01

AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

AND comment = ''

AND compaction = {'class': 
'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}

AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}

AND crc_check_chance = 1.0

AND dclocal_read_repair_chance = 0.1

AND default_time_to_live = 7776000

AND gc_grace_seconds = 864000

AND max_index_interval = 2048

AND memtable_flush_period_in_ms = 0

AND min_index_interval = 128

AND read_repair_chance = 0.0

AND speculative_retry = '99PERCENTILE';

 

and this is the "nodetool cfstats" output for that table:

Read Count: 39

Read Latency: 85.03307692307692 ms.

Write Count: 9845275

Write Latency: 0.09604882382665797 ms.

Pending Flushes: 0

Table: 

SSTable count: 48

Space used (live): 19566109394

Space used (total): 19566109394

Space used by snapshots (total): 109796505570

Off heap memory used (total): 11317941

SSTable Compression Ratio: 0.22632301701483284

Number of keys (estimate): 2557

Memtable cell count: 0

Memtable data size: 0

Memtable off heap memory used: 0

Memtable switch count: 828

Local read count: 39

Local read latency: 93.051 ms

Local write count: 9845275

Local write latency: 0.106 ms

Pending flushes: 0

Bloom filter false positives: 2

Bloom filter false ratio: 0.0

Bloom filter space used: 10200

Bloom filter off heap memory used: 9816

Index summary off heap memory used: 4677

Compression metadata off heap memory used: 11303448

Compacted partition minimum bytes: 150

Compacted partition maximum bytes: 4139110981

Compacted partition mean bytes: 13463937

Average live cells per slice (last five minutes): 59.69230769230769

Maximum live cells per slice (last five minutes): 149

Average tombstones per slice (last five minutes): 8.564102564102564

Maximum tombstones per slice (last five minutes): 42

 

According to the "nodetool compactionhistory ."

the oldest timestamp is "Thu, 30 Jun 2016 13:14:23 GMT"

and the most recent one is "Thu, 07 Jul 2016 12:15:50 GMT" (THAT IS TODAY)

 

However the table count is still very high compared to tables that have a 
different compaction strategy. If I run a "nodetool compact " the 
SSTable count decrease dramatically to a reasonable number.

I read many articles including: 
http://www.datastax.com/dev/blog/datetieredcompactionstrategy however I can not 
really tell if this is an expected behavior.

What concerns me is that I have an high tombstone read count despite those are 
insert only tables. Compacting the table make the tombstone issue disappear. 
Yes, we are using TTL to expire data after 3 months and I have not touch the GC 
grace period.

Looking at the file system I see the very first *-Data.db file that is 15GB 
then there are all the other 43 *-Data.db files that are ranging from 50 to 
150MB in size.

 

How can I debug this mis-compaction issue? Any help is much appreciated

Best,



smime.p7s
Description: S/MIME cryptographic signature

Blog post on Cassandra's inner workings and performance - feedback welcome (URL fixed)

[Manuel Kiessling]

Hi all,

I'm currently in the process of understanding the inner workings of
Cassandra with regards to network and local storage mechanisms and
operations. In order to do so, I've written a blog post about it which is
now in a "first final" version.

Any feedback, especially corrections regarding misunderstandings on my
side, would be highly appreciated. The post really represents my very
subjective view on how Cassandra works under the hood, which makes it prone
to errors of course.

You can access the current version at
http://journeymonitor.com:4000/tutorials/2016/02/29/cassandra-inner-workings-and-how-this-relates-to-performance/

Thanks,
-- 
 Manuel

DTCS SSTable count issue

[Riccardo Ferrari]

Hi everyone,

This is my first question, apologize may I do something wrong.

I have a small Cassandra cluster build upon 3 nodes. Originally born as
2.0.X cluster was upgraded to 2.0.15 then 2.1.13 and finally to 3.0.4
recently 3.0.6. Ubuntu is the OS.

There are few tables that have DateTieredCompactionStrategy and are
suffering of constantly growing SSTable count. I have the feeling this has
something to do with the upgrade however I need some hint on how to debug
this issue.

Tables are created like:
CREATE TABLE  (
 ...
PRIMARY KEY (...)
) WITH CLUSTERING ORDER BY (...)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 7776000
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

and this is the "nodetool cfstats" output for that table:
Read Count: 39
Read Latency: 85.03307692307692 ms.
Write Count: 9845275
Write Latency: 0.09604882382665797 ms.
Pending Flushes: 0
Table: 
SSTable count: 48
Space used (live): 19566109394
Space used (total): 19566109394
Space used by snapshots (total): 109796505570
Off heap memory used (total): 11317941
SSTable Compression Ratio: 0.22632301701483284
Number of keys (estimate): 2557
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 828
Local read count: 39
Local read latency: 93.051 ms
Local write count: 9845275
Local write latency: 0.106 ms
Pending flushes: 0
Bloom filter false positives: 2
Bloom filter false ratio: 0.0
Bloom filter space used: 10200
Bloom filter off heap memory used: 9816
Index summary off heap memory used: 4677
Compression metadata off heap memory used: 11303448
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 4139110981
Compacted partition mean bytes: 13463937
Average live cells per slice (last five minutes): 59.69230769230769
Maximum live cells per slice (last five minutes): 149
Average tombstones per slice (last five minutes): 8.564102564102564
Maximum tombstones per slice (last five minutes): 42

According to the "nodetool compactionhistory ."
the oldest timestamp is "Thu, 30 Jun 2016 13:14:23 GMT"
and the most recent one is "Thu, 07 Jul 2016 12:15:50 GMT" (THAT IS TODAY)

However the table count is still very high compared to tables that have a
different compaction strategy. If I run a "nodetool compact " the
SSTable count decrease dramatically to a reasonable number.
I read many articles including:
http://www.datastax.com/dev/blog/datetieredcompactionstrategy however I can
not really tell if this is an expected behavior.
What concerns me is that I have an high tombstone read count despite those
are insert only tables. Compacting the table make the tombstone issue
disappear. Yes, we are using TTL to expire data after 3 months and I have
not touch the GC grace period.
Looking at the file system I see the very first *-Data.db file that is 15GB
then there are all the other 43 *-Data.db files that are ranging from 50 to
150MB in size.

How can I debug this mis-compaction issue? Any help is much appreciated
Best,

Re: Blog post on Cassandra's inner workings and performance - feedback welcome

[Manuel Kiessling]

Awfully sorry, here is the correct URI to the blog post:
http://journeymonitor.com:4000/tutorials/2016/02/29/cassandra-inner-workings-and-how-this-relates-to-performance/

Regards,
Manuel


On Thu, Jul 7, 2016 at 3:11 PM, Manuel Kiessling 
wrote:

> Hi all,
>
> I'm currently in the process of understanding the inner workings of
> Cassandra with regards to network and local storage mechanisms and
> operations. In order to do so, I've written a blog post about it which is
> now in a "first final" version.
>
> Any feedback, especially corrections regarding misunderstandings on my
> side, would be highly appreciated. The post really represents my very
> subjective view on how Cassandra works under the hood, which makes it prone
> to errors of course.
>
> You can access the current version at
> http://localhost:4000/tutorials/2016/02/29/cassandra-inner-workings-and-how-this-relates-to-performance/
>
> Thanks,
> --
>  Manuel
>
>


-- 
 Manuel

Re: Blog post on Cassandra's inner workings and performance - feedback welcome

[Rakesh Kumar]

http://localhost:4000/tutorials/2016/02/29/cassandra-inner-workings-and-how-this-relates-to-performance/

This is not a valid address.

On Thu, Jul 7, 2016 at 9:11 AM, Manuel Kiessling  wrote:
> Hi all,
>
> I'm currently in the process of understanding the inner workings of
> Cassandra with regards to network and local storage mechanisms and
> operations. In order to do so, I've written a blog post about it which is
> now in a "first final" version.
>
> Any feedback, especially corrections regarding misunderstandings on my side,
> would be highly appreciated. The post really represents my very subjective
> view on how Cassandra works under the hood, which makes it prone to errors
> of course.
>
> You can access the current version at
> http://localhost:4000/tutorials/2016/02/29/cassandra-inner-workings-and-how-this-relates-to-performance/
>
> Thanks,
> --
>  Manuel
>

Re: Blog post on Cassandra's inner workings and performance - feedback welcome

[Steven Choo]

Hi Manuel,

Nice initiative, but I think you posted a link to your local machine.

Best regards,


Steven Choo





ste...@humanswitch.io
www.humanswitch.io

www.iqnomy.com

Office:

HumanSwitch
Bijsterveldenlaan 5
5045 ZZ Tilburg
Netherlands

0031-(0)13-303 11 60

On Thu, Jul 7, 2016 at 3:11 PM, Manuel Kiessling 
wrote:

> Hi all,
>
> I'm currently in the process of understanding the inner workings of
> Cassandra with regards to network and local storage mechanisms and
> operations. In order to do so, I've written a blog post about it which is
> now in a "first final" version.
>
> Any feedback, especially corrections regarding misunderstandings on my
> side, would be highly appreciated. The post really represents my very
> subjective view on how Cassandra works under the hood, which makes it prone
> to errors of course.
>
> You can access the current version at
> http://localhost:4000/tutorials/2016/02/29/cassandra-inner-workings-and-how-this-relates-to-performance/
>
> Thanks,
> --
>  Manuel
>
>

Blog post on Cassandra's inner workings and performance - feedback welcome

[Manuel Kiessling]

Hi all,

I'm currently in the process of understanding the inner workings of
Cassandra with regards to network and local storage mechanisms and
operations. In order to do so, I've written a blog post about it which is
now in a "first final" version.

Any feedback, especially corrections regarding misunderstandings on my
side, would be highly appreciated. The post really represents my very
subjective view on how Cassandra works under the hood, which makes it prone
to errors of course.

You can access the current version at
http://localhost:4000/tutorials/2016/02/29/cassandra-inner-workings-and-how-this-relates-to-performance/

Thanks,
-- 
 Manuel

Re: whats the default .yaml file that cassandra-stress uses

[Daiyue Weng]

I added

concurrent_writes: 48

to the cqlstress-example.yaml, which is recommended as No. of cores * 8.
But no improvement is made when trying 100 writes test.

Since cassandra.yaml specifies concurrent_writes, but cqlstress-example.yaml
doesn't.

BTW, the following warning was shown as the first line of the test.

WARN  10:25:44 You listed localhost/0:0:0:0:0:0:0:1:9042 in your contact
points, but it wasn't found in the control host's system.peers at startup

how to fix it?

many thanks

On 7 July 2016 at 09:45, Stone Fang  wrote:

> there is a simple config file for testing."cqlstress-example.yaml" under
> the tool folder.
> you can customize this file to achieve your test
>
> stone.
>
> On Thu, Jul 7, 2016 at 4:28 PM, Daiyue Weng  wrote:
>
>> Hi, I am wondering what's the default .yaml file that cassandra-stress
>> uses when testing writes and reads when command 'profile=' is not
>> specified. Is it the cassandra.yaml? Does it affect the performance of
>> cassandra-stress test by modifying/tuning it?
>>
>> ps.I am running cassandra instances on Linux, the path to
>> cassandra.yaml that I found is
>>
>> /etc/cassandra/cassandra.yaml
>>
>> thanks
>>
>
>

Re: [RELEASE] Apache Cassandra 3.0.8 released

[horschi]

Same for 2.2.7.

On Thu, Jul 7, 2016 at 10:49 AM, Julien Anguenot 
wrote:

> Hey,
>
> The Debian packages do not seem to have been published. Normal?
>
> Thank you.
>
>J.
>
> On Jul 6, 2016, at 4:20 PM, Jake Luciani  wrote:
>
> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 3.0.8.
>
> Apache Cassandra is a fully distributed database. It is the right choice
> when you need scalability and high availability without compromising
> performance.
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions are listed in our download
> section:
>
>  http://cassandra.apache.org/download/
>
> This version is a bug fix release[1] on the 3.0 series. As always, please
> pay
> attention to the release notes[2] and Let us know[3] if you were to
> encounter
> any problem.
>
> Enjoy!
>
> [1]: http://goo.gl/DQpe4d (CHANGES.txt)
> [2]: http://goo.gl/UISX1K (NEWS.txt)
> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>
>
>

Re: [RELEASE] Apache Cassandra 3.0.8 released

[Julien Anguenot]

Hey, 

The Debian packages do not seem to have been published. Normal?

Thank you.

   J. 

> On Jul 6, 2016, at 4:20 PM, Jake Luciani  wrote:
> 
> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 3.0.8.
> 
> Apache Cassandra is a fully distributed database. It is the right choice
> when you need scalability and high availability without compromising
> performance.
> 
>  http://cassandra.apache.org/ 
> 
> Downloads of source and binary distributions are listed in our download
> section:
> 
>  http://cassandra.apache.org/download/ 
> 
> This version is a bug fix release[1] on the 3.0 series. As always, please pay
> attention to the release notes[2] and Let us know[3] if you were to encounter
> any problem.
> 
> Enjoy!
> 
> [1]: http://goo.gl/DQpe4d  (CHANGES.txt)
> [2]: http://goo.gl/UISX1K  (NEWS.txt)
> [3]: https://issues.apache.org/jira/browse/CASSANDRA 
> 
> 

Re: whats the default .yaml file that cassandra-stress uses

[Stone Fang]

there is a simple config file for testing."cqlstress-example.yaml" under
the tool folder.
you can customize this file to achieve your test

stone.

On Thu, Jul 7, 2016 at 4:28 PM, Daiyue Weng  wrote:

> Hi, I am wondering what's the default .yaml file that cassandra-stress
> uses when testing writes and reads when command 'profile=' is not
> specified. Is it the cassandra.yaml? Does it affect the performance of
> cassandra-stress test by modifying/tuning it?
>
> ps.I am running cassandra instances on Linux, the path to
> cassandra.yaml that I found is
>
> /etc/cassandra/cassandra.yaml
>
> thanks
>

whats the default .yaml file that cassandra-stress uses

[Daiyue Weng]

Hi, I am wondering what's the default .yaml file that cassandra-stress uses
when testing writes and reads when command 'profile=' is not specified. Is
it the cassandra.yaml? Does it affect the performance of
cassandra-stress test by modifying/tuning it?

ps.I am running cassandra instances on Linux, the path to
cassandra.yaml that I found is

/etc/cassandra/cassandra.yaml

thanks