iops for commitlog, so write throughput likely goes
up, too. From: Jonathan Haddad <j...@jonhaddad.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Thursday, July 7, 2016 at 6:54 PM
To: "user@cassandra.apache.org" <user@cassandra.a
>>
>>>>>>
>>>>>>
>>>>>> A 600G EBS volume only guarantees 1800 iops – if you’re exhausting
>>>>>> those on writes, you’re going to suffer on reads.
>>>>>>
>>>>>>
>>>>>>
>>>>>> EBS iops scale with volume size.
>>>>>>
>>>>>>
>>>>>>
>>>>>> A 600G EBS volume only guarantees 1800 iops – if you’re exhausting
>>>>>> those on writes, you’re going to suffer on reads
d probably a good chunk of that allocated to
>>>>> heap. Consequently, you have almost no page cache, so your reads are going
>>>>> to hit the disk. Your reads being very low is not uncommon if you have no
>>>>> page cache – the default settings
ge cache – the default settings for Cassandra (64k compression chunks)
>>>> are really inefficient for small reads served off of disk. If you drop the
>>>> compression chunk size (4k, for example), you’ll probably see your read
>>>> throughput increase significantly, which will
ge cache – the default settings for Cassandra (64k compression chunks)
>>>> are really inefficient for small reads served off of disk. If you drop the
>>>> compression chunk size (4k, for example), you’ll probably see your read
>>>> throughput increase significantly, which will
), you’ll probably see your read
>>> throughput increase significantly, which will give you more iops for
>>> commitlog, so write throughput likely goes up, too.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From: *Jonatha
>> commitlog, so write throughput likely goes up, too.
>>
>>
>>
>>
>>
>>
>>
>> *From: *Jonathan Haddad <j...@jonhaddad.com>
>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Date: *Thursday, Ju
> *Date: *Thursday, July 7, 2016 at 6:54 PM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *Re: Is my cluster normal?
>
>
>
> What's your CPU looking like? If it's low, check your IO with iostat or
> dstat. I know some people have u
Hi Jonathan,
The IOs are like below. I am not sure why one node always has a much bigger
KB_read/s than other nodes. It seems not good.
==
avg-cpu: %user %nice %system %iowait %steal %idle
54.78 24.489.350.960.08 10.35
Device:tps
Those numbers, as I suspected, line up pretty well with your AWS
configuration and network latencies within AWS. It is clear that this is a
WRITE ONLY test. You might want to do a mixed (e.g. 50% read, 50% write)
test for sanity. Note that the test will populate the data BEFORE it begins
doing the
6 at 6:54 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Is my cluster normal?
What's your CPU looking like? If it's low, check your IO with iostat or dstat.
I know some people have used Ebs and say it's fine but ive been burned too many
times.
On Th
What's your CPU looking like? If it's low, check your IO with iostat or
dstat. I know some people have used Ebs and say it's fine but ive been
burned too many times.
On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang wrote:
> Hi Riccardo,
>
> Very low IO-wait. About 0.3%.
> No
Hi Riccardo,
Very low IO-wait. About 0.3%.
No stolen CPU. It is a casssandra only instance. I did not see any dropped
messages.
ubuntu@cassandra1:/mnt/data$ nodetool tpstats
Pool NameActive Pending Completed Blocked All
time blocked
MutationStage
Hi Yuan,
You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside
from any Cassandra specific discussion a system load of 10 on a 4 threads
machine is way too much in my opinion. If that is the running average
system load I would look deeper into system details. Is that IO wait? Is
Thanks Ben! For the post, it seems they got a little better but similar
result than i did. Good to know it.
I am not sure if a little fine tuning of heap memory will help or not.
On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater
wrote:
> Hi Yuan,
>
> You might find this
Hi Ryan,
The version of cassandra is 3.0.6 and
java version "1.8.0_91"
Yuan
On Thu, Jul 7, 2016 at 3:11 PM, Ryan Svihla wrote:
> what version of cassandra and java?
>
> Regards,
>
> Ryan Svihla
>
> On Jul 7, 2016, at 4:51 PM, Yuan Fang wrote:
>
>
what version of cassandra and java?
Regards,
Ryan Svihla
> On Jul 7, 2016, at 4:51 PM, Yuan Fang wrote:
>
> Yes, here is my stress test result:
> Results:
> op rate : 12200 [WRITE:12200]
> partition rate: 12200 [WRITE:12200]
> row rate
Hi Yuan,
You might find this blog post a useful comparison:
https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/
Although the focus is on Spark and Cassandra and multi-DC there are also
some single DC benchmarks of m4.xl clusters plus some
Yes, here is my stress test result:
Results:
op rate : 12200 [WRITE:12200]
partition rate: 12200 [WRITE:12200]
row rate : 12200 [WRITE:12200]
latency mean : 16.4 [WRITE:16.4]
latency median: 7.1 [WRITE:7.1]
latency 95th
Lots of variables you're leaving out.
Depends on write size, if you're using logged batch or not, what consistency
level, what RF, if the writes come in bursts, etc, etc. However, that's all
sort of moot for determining "normal" really you need a baseline as all those
variables end up
yes, it is about 8k writes per node.
On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle
wrote:
> Are you saying 7k writes per node? or 30k writes per node?
>
>
> *...*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
> <%28%2B1%29%20415.501.0198>London (+44) (0) 20
Are you saying 7k writes per node? or 30k writes per node?
*...*
*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang wrote:
> writes 30k/second is the main thing.
>
>
> On Thu, Jul 7, 2016 at 1:51
writes 30k/second is the main thing.
On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle
wrote:
> Assuming you meant 100k, that likely for something with 16mb of storage
> (probably way small) where the data is more that 64k hence will not fit
> into the row cache.
>
>
>
Assuming you meant 100k, that likely for something with 16mb of storage
(probably way small) where the data is more that 64k hence will not fit
into the row cache.
*...*
*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang
I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB ssd
EBS).
I can reach a cluster wide write requests of 30k/second and read request
about 100/second. The cluster OS load constantly above 10. Are those normal?
Thanks!
Best,
Yuan
26 matches
Mail list logo