Re: Cassandra cluster: could not reach linear scalability

[onmstester onmstester]

Thanks for your help,

I'been biased with Cassandra server and forget about the client completely!


Sent using Zoho Mail






 On Mon, 19 Feb 2018 15:21:03 +0330 Lucas Benevides 
lu...@maurobenevides.com.br wrote 




Why did you set the number of 1000 threads?

Does it show to be the more performatic when threads = auto?



I have used stress tool in a larger test bed (10 nodes) and my optimal setup 
was 24 threads. 

To check this you must monitor the stress node, both the CPU and I/O. And give 
it a try with fewer threads.



Lucas Benevides

Ipea





2018-02-18 8:29 GMT-03:00 onmstester onmstester onmstes...@zoho.com:








I've configured a simple cluster using two PC with identical spec:

 cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with 
iperf, it really is!) 
using the common configs described in many sites including datastax itself:

cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: 
org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 
"192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: 
GossipingPropertyFileSnitch 
Running stress tool:

cassandra-stress write n=100 -rate threads=1000 -mode native cql3 -node 
192.168.1.1,192.168.1.2 
Over each node it shows 39 K writes/seconds, but running the same stress tool 
command on cluster of both nodes shows 45 K writes/seconds. I've done all the 
tuning mentioned by apache and datastax. There are many use cases on the net 
proving Cassandra linear Scalability So what is wrong with my cluster?



Sent using Zoho Mail

Re: Cassandra cluster: could not reach linear scalability

[Lucas Benevides]

Why did you set the number of 1000 threads?
Does it show to be the more performatic when threads = auto?

I have used stress tool in a larger test bed (10 nodes) and my optimal
setup was 24 threads.
To check this you must monitor the stress node, both the CPU and I/O. And
give it a try with fewer threads.

Lucas Benevides
Ipea

2018-02-18 8:29 GMT-03:00 onmstester onmstester :

> I've configured a simple cluster using two PC with identical spec:
>
>   cpu core i5
>RAM: 8GB ddr3
>Disk: 1TB 5400rpm
>Network: 1 G (I've test it with iperf, it really is!)
>
> using the common configs described in many sites including datastax itself:
>
> cluster_name: 'MyCassandraCluster'
> num_tokens: 256
> seed_provider:
>   - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
>  - seeds: "192.168.1.1,192.168.1.2"
> listen_address:
> rpc_address: 0.0.0.0
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Running stress tool:
>
> cassandra-stress write n=100 -rate threads=1000 -mode native cql3 -node 
> 192.168.1.1,192.168.1.2
>
> Over each node it shows 39 K writes/seconds, but running the same stress
> tool command on cluster of both nodes shows 45 K writes/seconds. I've done
> all the tuning mentioned by apache and datastax. There are many use cases
> on the net proving Cassandra linear Scalability So what is wrong with my
> cluster?
>
> Sent using Zoho Mail 
>
>
>

Re: Cassandra cluster: could not reach linear scalability

[Rahul Singh]

If that is the case you could also try to run more stress from another machine 
as well.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Feb 18, 2018, 2:37 PM -0500, Jeff Jirsa , wrote:
> Stress client may be cpu bound as well
>
> --
> Jeff Jirsa
>
>
> On Feb 18, 2018, at 7:40 AM, onmstester onmstester  
> wrote:
>
> > I'm running tests on separate machine (not member of the cluster)
> > I'm using the default data model of cassandra-stress tool : keyspace1 and 
> > table: standard1. nothing special on network or data traffic. Network 
> > capable of 1 G and tested it with ipperf.
> > iftop shows maximum of 48Mbit traffic between nodes in cluster.
> > Have not seen any warning in log files.
> > I'm monitoring cassandra during runtime using jvisual vm and never saw any 
> > GC chokepoints, cpu is below 40% always. I just cant understand why 
> > cassandra is limmiting the throughput?!
> > using top, fps and write per seconds are not showing any problems
> >
> > Sent using Zoho Mail
> >
> >
> >  On Sun, 18 Feb 2018 18:42:48 +0330 Rahul Singh 
> >  wrote 
> >
> > > Got it.
> > >
> > > Here are some other questions.
> > >
> > > Are you running the test on separate machine or one of the cluster 
> > > members?
> > >
> > > When configuring Cassandra stress what data model did you end up using? ( 
> > > do you see Data or traffic skew?)
> > >
> > > Do you see any wide partitions or Tombstone warnings on either node.
> > >
> > > Have you visualized the GC logs using something like VisualVM or hubspots 
> > > GC visualizer? This is to see if there are chokepoints in the GC cycle.
> > >
> > >
> > > --
> > > Rahul Singh
> > > rahul.si...@anant.us
> > >
> > > Anant Corporation
> > >
> > > On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester 
> > > , wrote:
> > >
> > > > But monitoring cassandra with jmx using jvisualVM shows no problem, 
> > > > less than 30% of heap size used
> > > >
> > > > Sent using Zoho Mail
> > > >
> > > >
> > > >  On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh 
> > > >  wrote 
> > > >
> > > > > You don’t don’t have enough memory. That’s just a start.
> > > > >
> > > > > --
> > > > > Rahul Singh
> > > > > rahul.si...@anant.us
> > > > >
> > > > > Anant Corporation
> > > > >
> > > > > On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester 
> > > > > , wrote:
> > > > >
> > > > > > I've configured a simple cluster using two PC with identical spec:
> > > > > >  cpu core i5
> > > > > >   RAM: 8GB ddr3
> > > > > >   Disk: 1TB 5400rpm
> > > > > >   Network: 1 G (I've test it with iperf, it really is!)
> > > > > >
> > > > > > using the common configs described in many sites including datastax 
> > > > > > itself:
> > > > > > cluster_name: 'MyCassandraCluster'
> > > > > > num_tokens: 256
> > > > > > seed_provider:
> > > > > >  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> > > > > >parameters:
> > > > > > - seeds: "192.168.1.1,192.168.1.2"
> > > > > > listen_address:
> > > > > > rpc_address: 0.0.0.0
> > > > > > endpoint_snitch: GossipingPropertyFileSnitch
> > > > > >
> > > > > > Running stress tool:
> > > > > > cassandra-stress write n=100 -rate threads=1000 -mode native 
> > > > > > cql3 -node 192.168.1.1,192.168.1.2
> > > > > >
> > > > > > Over each node it shows 39 K writes/seconds, but running the same 
> > > > > > stress tool command on cluster of both nodes shows 45 K 
> > > > > > writes/seconds. I've done all the tuning mentioned by apache and 
> > > > > > datastax. There are many use cases on the net proving Cassandra 
> > > > > > linear Scalability So what is wrong with my cluster?
> > > > > >
> > > > > > Sent using Zoho Mail
> > > > > >
> > > >
> >
> >

Re: Cassandra cluster: could not reach linear scalability

[Jeff Jirsa]

Stress client may be cpu bound as well

-- 
Jeff Jirsa


> On Feb 18, 2018, at 7:40 AM, onmstester onmstester  
> wrote:
> 
> I'm running tests on separate machine (not member of the cluster)
> I'm using the default data model of cassandra-stress tool : keyspace1 and 
> table: standard1. nothing special on network or data traffic. Network capable 
> of 1 G and tested it with ipperf.
> iftop shows maximum of 48Mbit traffic between nodes in cluster.
> Have not seen any warning in log files.
> I'm monitoring cassandra during runtime using jvisual vm and never saw any GC 
> chokepoints, cpu is below 40% always. I just cant understand why cassandra is 
> limmiting the throughput?!
> using top, fps and write per seconds are not showing any problems
> 
> Sent using Zoho Mail
> 
> 
> 
>  On Sun, 18 Feb 2018 18:42:48 +0330 Rahul Singh 
>  wrote 
> 
> Got it.
> 
> Here are some other questions.
> 
> Are you running the test on separate machine or one of the cluster members?
> 
> When configuring Cassandra stress what data model did you end up using? ( do 
> you see Data or traffic skew?)
> 
> Do you see any wide partitions or Tombstone warnings on either node.
> 
> Have you visualized the GC logs using something like VisualVM or hubspots GC 
> visualizer? This is to see if there are chokepoints in the GC cycle.
> 
> 
> --
> Rahul Singh
> rahul.si...@anant.us
> 
> Anant Corporation
> 
> On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester , 
> wrote: 
> 
> But monitoring cassandra with jmx using jvisualVM shows no problem, less than 
> 30% of heap size used
> 
> Sent using Zoho Mail
> 
> 
> 
>  On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh 
>  wrote 
> 
> You don’t don’t have enough memory. That’s just a start.
> 
> --
> Rahul Singh
> rahul.si...@anant.us
> 
> Anant Corporation
> 
> On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester , 
> wrote:
> 
> I've configured a simple cluster using two PC with identical spec:
> 
>   cpu core i5
>RAM: 8GB ddr3
>Disk: 1TB 5400rpm
>Network: 1 G (I've test it with iperf, it really is!)
> 
> using the common configs described in many sites including datastax itself:
> 
> cluster_name: 'MyCassandraCluster'
> num_tokens: 256
> seed_provider:
>   - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
>  - seeds: "192.168.1.1,192.168.1.2"
> listen_address:
> rpc_address: 0.0.0.0
> endpoint_snitch: GossipingPropertyFileSnitch
> 
> Running stress tool:
> 
> cassandra-stress write n=100 -rate threads=1000 -mode native cql3 -node 
> 192.168.1.1,192.168.1.2
> 
> Over each node it shows 39 K writes/seconds, but running the same stress tool 
> command on cluster of both nodes shows 45 K writes/seconds. I've done all the 
> tuning mentioned by apache and datastax. There are many use cases on the net 
> proving Cassandra linear Scalability So what is wrong with my cluster?
> 
> 
> Sent using Zoho Mail
> 
> 
> 
> 
> 

Re: Cassandra cluster: could not reach linear scalability

[onmstester onmstester]

I'm running tests on separate machine (not member of the cluster)

I'm using the default data model of cassandra-stress tool : keyspace1 and 
table: standard1. nothing special on network or data traffic. Network capable 
of 1 G and tested it with ipperf.

iftop shows maximum of 48Mbit traffic between nodes in cluster.

Have not seen any warning in log files.

I'm monitoring cassandra during runtime using jvisual vm and never saw any GC 
chokepoints, cpu is below 40% always. I just cant understand why cassandra is 
limmiting the throughput?!

using top, fps and write per seconds are not showing any problems



Sent using Zoho Mail






 On Sun, 18 Feb 2018 18:42:48 +0330 Rahul Singh 
rahul.xavier.si...@gmail.com wrote 




Got it.

 

 Here are some other questions.

 

 Are you running the test on separate machine or one of the cluster members?

 

 When configuring Cassandra stress what data model did you end up using? ( do 
you see Data or traffic skew?)

 

 Do you see any wide partitions or Tombstone warnings on either node.

 

 Have you visualized the GC logs using something like VisualVM or hubspots GC 
visualizer? This is to see if there are chokepoints in the GC cycle.

 




--

 Rahul Singh

 rahul.si...@anant.us

 

 Anant Corporation




On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester 
onmstes...@zoho.com, wrote: 





But monitoring cassandra with jmx using jvisualVM shows no problem, less than 
30% of heap size used



Sent using Zoho Mail






 On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh 
rahul.xavier.si...@gmail.com wrote 




You don’t don’t have enough memory. That’s just a start.



--

Rahul Singh

rahul.si...@anant.us



Anant Corporation




On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester 
onmstes...@zoho.com, wrote:





I've configured a simple cluster using two PC with identical spec:

 cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with 
iperf, it really is!) 
using the common configs described in many sites including datastax itself:

cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: 
org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 
"192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: 
GossipingPropertyFileSnitch 
Running stress tool:

cassandra-stress write n=100 -rate threads=1000 -mode native cql3 -node 
192.168.1.1,192.168.1.2 
Over each node it shows 39 K writes/seconds, but running the same stress tool 
command on cluster of both nodes shows 45 K writes/seconds. I've done all the 
tuning mentioned by apache and datastax. There are many use cases on the net 
proving Cassandra linear Scalability So what is wrong with my cluster?



Sent using Zoho Mail

Re: Cassandra cluster: could not reach linear scalability

[Rahul Singh]

Got it.

Here are some other questions.

Are you running the test on separate machine or one of the cluster members?

When configuring Cassandra stress what data model did you end up using? ( do 
you see Data or traffic skew?)

Do you see any wide partitions or Tombstone warnings on either node.

Have you visualized the GC logs using something like VisualVM or hubspots GC 
visualizer? This is to see if there are chokepoints in the GC cycle.


--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Feb 18, 2018, 9:23 AM -0500, onmstester onmstester , 
wrote:
> But monitoring cassandra with jmx using jvisualVM shows no problem, less than 
> 30% of heap size used
>
> Sent using Zoho Mail
>
>
>  On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh 
>  wrote 
>
> > You don’t don’t have enough memory. That’s just a start.
> >
> > --
> > Rahul Singh
> > rahul.si...@anant.us
> >
> > Anant Corporation
> >
> > On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester 
> > , wrote:
> >
> > > I've configured a simple cluster using two PC with identical spec:
> > >  cpu core i5
> > >   RAM: 8GB ddr3
> > >   Disk: 1TB 5400rpm
> > >   Network: 1 G (I've test it with iperf, it really is!)
> > >
> > > using the common configs described in many sites including datastax 
> > > itself:
> > > cluster_name: 'MyCassandraCluster'
> > > num_tokens: 256
> > > seed_provider:
> > >  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> > >parameters:
> > > - seeds: "192.168.1.1,192.168.1.2"
> > > listen_address:
> > > rpc_address: 0.0.0.0
> > > endpoint_snitch: GossipingPropertyFileSnitch
> > >
> > > Running stress tool:
> > > cassandra-stress write n=100 -rate threads=1000 -mode native cql3 
> > > -node 192.168.1.1,192.168.1.2
> > >
> > > Over each node it shows 39 K writes/seconds, but running the same stress 
> > > tool command on cluster of both nodes shows 45 K writes/seconds. I've 
> > > done all the tuning mentioned by apache and datastax. There are many use 
> > > cases on the net proving Cassandra linear Scalability So what is wrong 
> > > with my cluster?
> > >
> > > Sent using Zoho Mail
> > >
>
>

Re: Cassandra cluster: could not reach linear scalability

[onmstester onmstester]

But monitoring cassandra with jmx using jvisualVM shows no problem, less than 
30% of heap size used


Sent using Zoho Mail






 On Sun, 18 Feb 2018 17:26:59 +0330 Rahul Singh 
rahul.xavier.si...@gmail.com wrote 




You don’t don’t have enough memory. That’s just a start.



--

 Rahul Singh

 rahul.si...@anant.us

 

 Anant Corporation




On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester 
onmstes...@zoho.com, wrote: 





I've configured a simple cluster using two PC with identical spec:

 cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with 
iperf, it really is!) 
using the common configs described in many sites including datastax itself:

cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: 
org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 
"192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: 
GossipingPropertyFileSnitch 
Running stress tool:

cassandra-stress write n=100 -rate threads=1000 -mode native cql3 -node 
192.168.1.1,192.168.1.2 
Over each node it shows 39 K writes/seconds, but running the same stress tool 
command on cluster of both nodes shows 45 K writes/seconds. I've done all the 
tuning mentioned by apache and datastax. There are many use cases on the net 
proving Cassandra linear Scalability So what is wrong with my cluster?



Sent using Zoho Mail

Re: Cassandra cluster: could not reach linear scalability

[Rahul Singh]

You don’t don’t have enough memory. That’s just a start.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Feb 18, 2018, 6:29 AM -0500, onmstester onmstester , 
wrote:
> I've configured a simple cluster using two PC with identical spec:
>  cpu core i5
>   RAM: 8GB ddr3
>   Disk: 1TB 5400rpm
>   Network: 1 G (I've test it with iperf, it really is!)
>
> using the common configs described in many sites including datastax itself:
> cluster_name: 'MyCassandraCluster'
> num_tokens: 256
> seed_provider:
>  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>parameters:
> - seeds: "192.168.1.1,192.168.1.2"
> listen_address:
> rpc_address: 0.0.0.0
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Running stress tool:
> cassandra-stress write n=100 -rate threads=1000 -mode native cql3 -node 
> 192.168.1.1,192.168.1.2
>
> Over each node it shows 39 K writes/seconds, but running the same stress tool 
> command on cluster of both nodes shows 45 K writes/seconds. I've done all the 
> tuning mentioned by apache and datastax. There are many use cases on the net 
> proving Cassandra linear Scalability So what is wrong with my cluster?
>
> Sent using Zoho Mail
>
>

Cassandra cluster: could not reach linear scalability

[onmstester onmstester]

I've configured a simple cluster using two PC with identical spec:

 cpu core i5 RAM: 8GB ddr3 Disk: 1TB 5400rpm Network: 1 G (I've test it with 
iperf, it really is!) 
using the common configs described in many sites including datastax itself:

cluster_name: 'MyCassandraCluster' num_tokens: 256 seed_provider: - class_name: 
org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 
"192.168.1.1,192.168.1.2" listen_address: rpc_address: 0.0.0.0 endpoint_snitch: 
GossipingPropertyFileSnitch 
Running stress tool:

cassandra-stress write n=100 -rate threads=1000 -mode native cql3 -node 
192.168.1.1,192.168.1.2 
Over each node it shows 39 K writes/seconds, but running the same stress tool 
command on cluster of both nodes shows 45 K writes/seconds. I've done all the 
tuning mentioned by apache and datastax. There are many use cases on the net 
proving Cassandra linear Scalability So what is wrong with my cluster?



Sent using Zoho Mail