Re: [Ann] Ansible base Cassandra stress framework for EC2

[Tzach Livyatan]

For anyone interested, the framework can now deploy and stress Cassandra on
multi EC2 regions.
https://github.com/scylladb/cassandra-test-and-deploy

Cheers
Tzach


On Thu, Sep 3, 2015 at 9:21 PM, Tzach Livyatan 
wrote:

> I'm please to share a framework for running Cassandra stress tests on EC2.
> https://github.com/cloudius-systems/ansible-cassandra-cluster-stress
>
> The framework is a collection of Ansible playbooks and scripts, allowing
> to:
> - Create a Cassandra cluster (setting server type, version, etc)
> - Launch any number of loaders
> - Run cassandra-stress on all loaders and collect the results
> - Add nodes to a running cluster
> - Stop and starts nodes
> - Clean old data from servers before each test
> - Collect and display relevant metrics on a Collectd+Graphite+Tessera
> 
>  server
>
> Use cases I tested using this framework:
> * Stress with multiple loaders
> * Out Scale
> 
>  (adding
> server under stress)
> * Testing Cassandra Repair
> 
>
> More info in README
> 
> the  and Wiki
>
> Some of my future plans include adding YCSB, run on other providers and
> more.
> Contributions and suggestions will be appreciated!
>
> Tzach
>

Re: Doubt regarding consistency-level in Cassandra-2.1.10

[Bryan Cheng]

What Eric means is that SERIAL consistency is a special type of consistency
that is only invoked for a subset of operations: those that use
CAS/lightweight transactions, for example "IF NOT EXISTS" queries.

The differences between CAS operations and standard operations are
significant and there are large repercussions for tunable consistency. The
amount of time such an operation takes is greatly increased as well; you
may need to increase your internal node-to-node timeouts .

On Mon, Nov 2, 2015 at 8:01 PM, Ajay Garg  wrote:

> Hi Eric,
>
> I am sorry, but I don't understand.
>
> If there had been some issue in the configuration, then the
> consistency-issue would be seen everytime (I guess).
> As of now, the error is seen sometimes (probably 30% of times).
>
> On Mon, Nov 2, 2015 at 10:24 PM, Eric Stevens  wrote:
>
>> Serial consistency gets invoked at the protocol level when doing
>> lightweight transactions such as CAS operations.  If you're expecting that
>> your topology is RF=2, N=2, it seems like some keyspace has RF=3, and so
>> there aren't enough nodes available to satisfy serial consistency.
>>
>> See
>> http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_ltwt_transaction_c.html
>>
>> On Mon, Nov 2, 2015 at 1:29 AM Ajay Garg  wrote:
>>
>>> Hi All.
>>>
>>> I have a 2*2 Network-Topology Replication setup, and I run my
>>> application via DataStax-driver.
>>>
>>> I frequently get the errors of type ::
>>> *Cassandra timeout during write query at consistency SERIAL (3 replica
>>> were required but only 0 acknowledged the write)*
>>>
>>> I have already tried passing a "write-options with LOCAL_QUORUM
>>> consistency-level" in all create/save statements, but I still get this
>>> error.
>>>
>>> Does something else need to be changed in /etc/cassandra/cassandra.yaml
>>> too?
>>> Or may be some another place?
>>>
>>>
>>> --
>>> Regards,
>>> Ajay
>>>
>>
>
>
> --
> Regards,
> Ajay
>

Re: compression cpu overhead

[Tushar Agrawal]

For writes it's negligible. For reads it makes a significant difference for 
high tps and low latency workload. You would see up to 3x higher cpu with LZ4 
vs no compression. It would be different for different h/w configurations.


Thanks,
Tushar
(Sent from iPhone)

> On Nov 3, 2015, at 5:51 PM, Dan Kinder  wrote:
> 
> Most concerned about write since that's where most of the cost is, but perf 
> numbers for a any workload mix would be helpful. 
> 
>> On Tue, Nov 3, 2015 at 3:48 PM, Graham Sanderson  wrote:
>> On read or write?
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-7039 and friends in 2.2 
>> should make some difference, I didn’t immediately find perf numbers though.
>> 
>>> On Nov 3, 2015, at 5:42 PM, Dan Kinder  wrote:
>>> 
>>> Hey all,
>>> 
>>> Just wondering if anyone has done seen or done any benchmarking for the 
>>> actual CPU overhead added by various compression algorithms in Cassandra 
>>> (at least LZ4) vs no compression. Clearly this is going to be workload 
>>> dependent but even a rough gauge would be helpful (ex. "Turning on LZ4 
>>> compression increases my CPU load by ~2x")
>>> 
>>> -dan
> 
> 
> 
> -- 
> Dan Kinder
> Senior Software Engineer
> Turnitin – www.turnitin.com
> dkin...@turnitin.com

compression cpu overhead

[Dan Kinder]

Hey all,

Just wondering if anyone has done seen or done any benchmarking for the
actual CPU overhead added by various compression algorithms in Cassandra
(at least LZ4) vs no compression. Clearly this is going to be workload
dependent but even a rough gauge would be helpful (ex. "Turning on LZ4
compression increases my CPU load by ~2x")

-dan

Re: compression cpu overhead

[Dan Kinder]

Most concerned about write since that's where most of the cost is, but perf
numbers for a any workload mix would be helpful.

On Tue, Nov 3, 2015 at 3:48 PM, Graham Sanderson  wrote:

> On read or write?
>
> https://issues.apache.org/jira/browse/CASSANDRA-7039 and friends in 2.2
> should make some difference, I didn’t immediately find perf numbers though.
>
> On Nov 3, 2015, at 5:42 PM, Dan Kinder  wrote:
>
> Hey all,
>
> Just wondering if anyone has done seen or done any benchmarking for the
> actual CPU overhead added by various compression algorithms in Cassandra
> (at least LZ4) vs no compression. Clearly this is going to be workload
> dependent but even a rough gauge would be helpful (ex. "Turning on LZ4
> compression increases my CPU load by ~2x")
>
> -dan
>
>
>


-- 
Dan Kinder
Senior Software Engineer
Turnitin – www.turnitin.com
dkin...@turnitin.com

Re: compression cpu overhead

[Jon Haddad]

You won't see any overhead on writes because you don't actually write to 
sstables when performing a write.  Just the commit log & memtable.  Memtables 
are flushes asynchronously.

> On Nov 4, 2015, at 1:57 AM, Tushar Agrawal  wrote:
> 
> For writes it's negligible. For reads it makes a significant difference for 
> high tps and low latency workload. You would see up to 3x higher cpu with LZ4 
> vs no compression. It would be different for different h/w configurations.
> 
> 
> Thanks,
> Tushar
> (Sent from iPhone)
> 
> On Nov 3, 2015, at 5:51 PM, Dan Kinder  > wrote:
> 
>> Most concerned about write since that's where most of the cost is, but perf 
>> numbers for a any workload mix would be helpful. 
>> 
>> On Tue, Nov 3, 2015 at 3:48 PM, Graham Sanderson > > wrote:
>> On read or write?
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-7039 
>>  and friends in 2.2 
>> should make some difference, I didn’t immediately find perf numbers though.
>> 
>>> On Nov 3, 2015, at 5:42 PM, Dan Kinder >> > wrote:
>>> 
>>> Hey all,
>>> 
>>> Just wondering if anyone has done seen or done any benchmarking for the 
>>> actual CPU overhead added by various compression algorithms in Cassandra 
>>> (at least LZ4) vs no compression. Clearly this is going to be workload 
>>> dependent but even a rough gauge would be helpful (ex. "Turning on LZ4 
>>> compression increases my CPU load by ~2x")
>>> 
>>> -dan
>> 
>> 
>> 
>> 
>> -- 
>> Dan Kinder
>> Senior Software Engineer
>> Turnitin – www.turnitin.com 
>> dkin...@turnitin.com 

Re: compression cpu overhead

[Graham Sanderson]

On read or write?

https://issues.apache.org/jira/browse/CASSANDRA-7039 
 and friends in 2.2 
should make some difference, I didn’t immediately find perf numbers though.

> On Nov 3, 2015, at 5:42 PM, Dan Kinder  wrote:
> 
> Hey all,
> 
> Just wondering if anyone has done seen or done any benchmarking for the 
> actual CPU overhead added by various compression algorithms in Cassandra (at 
> least LZ4) vs no compression. Clearly this is going to be workload dependent 
> but even a rough gauge would be helpful (ex. "Turning on LZ4 compression 
> increases my CPU load by ~2x")
> 
> -dan



smime.p7s
Description: S/MIME cryptographic signature

Re: Doubt regarding consistency-level in Cassandra-2.1.10

[Ajay Garg]

Hmm... ok.

Ideally, we require ::

a)
The intra-DC-node-syncing takes place at the statement/query level.

b)
The inter-DC-node-syncing takes place at cassandra level.


That way, we don't spend too much delay at the statement/query level.


For the so-called CAS/lightweight transactions, the above are impossible
then?

On Wed, Nov 4, 2015 at 5:58 AM, Bryan Cheng  wrote:

> What Eric means is that SERIAL consistency is a special type of
> consistency that is only invoked for a subset of operations: those that use
> CAS/lightweight transactions, for example "IF NOT EXISTS" queries.
>
> The differences between CAS operations and standard operations are
> significant and there are large repercussions for tunable consistency. The
> amount of time such an operation takes is greatly increased as well; you
> may need to increase your internal node-to-node timeouts .
>
> On Mon, Nov 2, 2015 at 8:01 PM, Ajay Garg  wrote:
>
>> Hi Eric,
>>
>> I am sorry, but I don't understand.
>>
>> If there had been some issue in the configuration, then the
>> consistency-issue would be seen everytime (I guess).
>> As of now, the error is seen sometimes (probably 30% of times).
>>
>> On Mon, Nov 2, 2015 at 10:24 PM, Eric Stevens  wrote:
>>
>>> Serial consistency gets invoked at the protocol level when doing
>>> lightweight transactions such as CAS operations.  If you're expecting that
>>> your topology is RF=2, N=2, it seems like some keyspace has RF=3, and so
>>> there aren't enough nodes available to satisfy serial consistency.
>>>
>>> See
>>> http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_ltwt_transaction_c.html
>>>
>>> On Mon, Nov 2, 2015 at 1:29 AM Ajay Garg  wrote:
>>>
 Hi All.

 I have a 2*2 Network-Topology Replication setup, and I run my
 application via DataStax-driver.

 I frequently get the errors of type ::
 *Cassandra timeout during write query at consistency SERIAL (3 replica
 were required but only 0 acknowledged the write)*

 I have already tried passing a "write-options with LOCAL_QUORUM
 consistency-level" in all create/save statements, but I still get this
 error.

 Does something else need to be changed in /etc/cassandra/cassandra.yaml
 too?
 Or may be some another place?


 --
 Regards,
 Ajay

>>>
>>
>>
>> --
>> Regards,
>> Ajay
>>
>
>


-- 
Regards,
Ajay

Re: Error code=1000

[Tyler Hobbs]

When you say "I am using cassandra standalone", do you mean that you're
running a single-node cluster?  If that's the case, then I'm guessing your
problem is that the replication factor for the keyspace is set to 2 or 3
(instead of 1).

On Sat, Oct 31, 2015 at 3:00 PM, Ricardo Sancho 
wrote:

> One or more of your nodes, depending on your replication factor, is not
> answering in time. Either they are down or have too much load that they are
> not able to answer every request before the timeout expires.
> On 31 Oct 2015 20:35, "Eduardo Alfaia"  wrote:
>
>> Hi guys,
>>
>> Could you help me with this error?
>>
>> cassandra.Unavailable: code=1000 [Unavailable exception] message="Cannot
>> achieve consistency level LOCAL_QUORUM" info={'required_replicas': 2,
>> 'alive_replicas': 1, 'consistency': 'LOCAL_QUORUM’}
>>
>> I am using cassandra standalone
>>
>> Thanks
>>
>>


-- 
Tyler Hobbs
DataStax