Reversed read write performance.

[Roger Warner]

I’m confused about read vs write performance.   I was expecting to see higher 
write than read perf.   I’m seeing the opposite by nearly 2X
Please help.  Am I doing/configuring something wrong or do I have the wrong 
expectations.   I am very new to Cassandra.   And this is not using Datastax. .

I have a new cluster I’m trying to estimate capacity of.

· 6 nodes, 2 datacenters, 3:3  – but on the same physical network as of 
now

· 500G SATA SSD drives benched at 250MB/s on these machines

· 32 core,  2Ghz, 60G of RAM / node

· Cassandra 3.11

Cassandra-stress being run on the same build as these.

· Cl=LOCAL_ONE

· Mixed test 4:1

· Defaults otherwise

· Testing  using –nodes from dci.

Schema

· Tweaked Cassandra-stress schema

· NetworkToplogyStragety

· 2 DC

· 3 replicas

· SizeTieredCompactionStrategy



I’ve run 4:1 read to write I see ~325Kops/s
I see about write to read I see ~150Kops/s

Any suggestions a very much appreciated.

Re: Reversed read write performance.

[Roger Warner]

30G java heap.   The dataset is the usual Cassandra-test size

How do I tell if compaction has completed?

I will add more iterations/time to the test.

Thank you




Date: Friday, July 14, 2017 at 2:21 PM
To: Roger Warner <rwar...@pandora.com>
Subject: Re: Reversed read write performance.

Pls add info about caching. Probably your reads are cached (what's the dataset 
size?).
Pls also make sure that you run these tests after previous compaction finished 
and run
them long enough (even few hours).

Cheers,
d...@scylladb.com<mailto:d...@scylladb.com>

On Sat, Jul 15, 2017 at 12:11 AM, Roger Warner 
<rwar...@pandora.com<mailto:rwar...@pandora.com>> wrote:
I’m confused about read vs write performance.   I was expecting to see higher 
write than read perf.   I’m seeing the opposite by nearly 2X
Please help.  Am I doing/configuring something wrong or do I have the wrong 
expectations.   I am very new to Cassandra.   And this is not using Datastax. .

I have a new cluster I’m trying to estimate capacity of.

• 6 nodes, 2 datacenters, 3:3  – but on the same physical network as of 
now

• 500G SATA SSD drives benched at 250MB/s on these machines

• 32 core,  2Ghz, 60G of RAM / node

• Cassandra 3.11

Cassandra-stress being run on the same build as these.

• Cl=LOCAL_ONE

• Mixed test 4:1

• Defaults otherwise

• Testing  using –nodes from dci.

Schema

• Tweaked Cassandra-stress schema

• NetworkToplogyStragety

• 2 DC

• 3 replicas

• SizeTieredCompactionStrategy



I’ve run 4:1 read to write I see ~325Kops/s
I see about write to read I see ~150Kops/s

Any suggestions a very much appreciated.

Data Loss irreparabley so

[Roger Warner]

This is a quick informational question. I know that Cassandra can detect 
failures of nodes and repair them given replication and multiple DC.

My question is can Cassandra tell if data was lost after a failure and node(s) 
“fixed” and resumed operation?

If so where would it log or flag it?  Or are we just supposed to figure it out?

R

Re: read/write request counts and write size of each write

[Roger Warner]

You need to also have the mx4j jar in your Cassandra lib directory.   Double 
checking you did that – its not included with the distro.You have to 
download it.

http://mx4j.sourceforge.net/

R

From: Nitan Kainth 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, July 26, 2017 at 8:22 AM
To: "User cassandra.apache.org" 
Subject: Re: read/write request counts and write size of each write

Thank you very much Kurt.

I am not a java guy, need one small help. I initiated JMX connection but I am 
getting some exception:


java -jar ~/jmxterm-1.0-alpha-4-uber.jar --url localhost:7199

Welcome to JMX terminal. Type "help" for available commands.

$>run -b org.apache.cassandra.metrics:type=ClientRequest scope=CASWrite 
name=MutationSizeHistogram

#InstanceNotFoundException: org.apache.cassandra.metrics:type=ClientRequest


I verified, Cassandra is running on my machine.



On Tue, Jul 25, 2017 at 9:36 PM, kurt greaves 
> wrote:
Looks like you can collect MutationSizeHistogram for each write as well from 
the coordinator, in regards to write request size. See the Write request 
section under 
https://cassandra.apache.org/doc/latest/operating/metrics.html#client-request-metrics

Re: read/write request counts and write size of each write

[Roger Warner]

I think that is not the correct lib directory.You want it under 
$CASSANDRA_HOME/lib.   Ie wherever you deployed cassandra distro into / lib

That directory should be loaded with *.jar files.   That is the directory you 
want.

Roger

From: Nitan Kainth <ni...@bamlabs.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, July 26, 2017 at 1:42 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: read/write request counts and write size of each write

Hey Roger,

I downloaded and saved the file in /var/lib/cassandra, but getting same error:

 java -jar /tmp/jmxterm-1.0-alpha-4-uber.jar --url localhost:7199
Welcome to JMX terminal. Type "help" for available commands.
$>run -b org.apache.cassandra.metrics:type=ClientRequest scope=CASWrite 
name=MutationSizeHistogram
#InstanceNotFoundException: org.apache.cassandra.metrics:type=ClientRequest


I tried using mx4 jar file:


 java -jar /var/lib/cassandra/mx4j-3.0.1.jar url localhost:7199
no main manifest attribute, in /var/lib/cassandra/mx4j-3.0.1.jar






On Jul 26, 2017, at 11:50 AM, Roger Warner 
<rwar...@pandora.com<mailto:rwar...@pandora.com>> wrote:

You need to also have the mx4j jar in your Cassandra lib directory.   Double 
checking you did that – its not included with the distro.You have to 
download it.

http://mx4j.sourceforge.net/<https://urldefense.proofpoint.com/v2/url?u=http-3A__mx4j.sourceforge.net_=DwMFaQ=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw=_V9IzBxaV864n6VoHCzfArKKHY9Xqk-DWw-wAB9JU7E=rd2FJwCKPP5z8NHhUlnT38ZikdsXjyW01dIzgTpVNZI=ALRm4TAh0JSWovaZA0ijrCAmzPz_H1hBZEJ_psmMGyU=>

R

From: Nitan Kainth <ni...@sleepiqlabs.com<mailto:ni...@sleepiqlabs.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Wednesday, July 26, 2017 at 8:22 AM
To: "User 
cassandra.apache.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org=DwMFaQ=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw=_V9IzBxaV864n6VoHCzfArKKHY9Xqk-DWw-wAB9JU7E=rd2FJwCKPP5z8NHhUlnT38ZikdsXjyW01dIzgTpVNZI=K_gfKLpT6YRBplNhlYsM7Rgtx7FEz-Q6bvXUom6GMks=>"
 <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: read/write request counts and write size of each write

Thank you very much Kurt.

I am not a java guy, need one small help. I initiated JMX connection but I am 
getting some exception:

java -jar ~/jmxterm-1.0-alpha-4-uber.jar --url localhost:7199
Welcome to JMX terminal. Type "help" for available commands.
$>run -b org.apache.cassandra.metrics:type=ClientRequest scope=CASWrite 
name=MutationSizeHistogram
#InstanceNotFoundException: org.apache.cassandra.metrics:type=ClientRequest



I verified, Cassandra is running on my machine.




On Tue, Jul 25, 2017 at 9:36 PM, kurt greaves 
<k...@instaclustr.com<mailto:k...@instaclustr.com>> wrote:
Looks like you can collect MutationSizeHistogram for each write as well from 
the coordinator, in regards to write request size. See the Write request 
section 
underhttps://cassandra.apache.org/doc/latest/operating/metrics.html#client-request-metrics<https://urldefense.proofpoint.com/v2/url?u=https-3A__cassandra.apache.org_doc_latest_operating_metrics.html-23client-2Drequest-2Dmetrics=DwMFaQ=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw=_V9IzBxaV864n6VoHCzfArKKHY9Xqk-DWw-wAB9JU7E=_TOyM3vR40V4Xvh1nWnClVq3MYuybHfhdtWfkIGvVlk=vgnwngzAkTiksz5qoRNA71R9UZTCiYLkjw4qXe5nK74=>

Multi datacenter node loss

[Roger Warner]

Hi

I’m a little dim on what multi datacenter implies in the 1 replica case. I 
know about replica recovery, how about “node recovery”

As I understand if there a node failure or disk crash with a single node 
cluster with replication factor 1 I lose data.Easy.

nodetool tells me each node in my 3 node X 2 datacenters is responsible for 
~1/3 of the data.If in this cluster with RF=1 a node fails in dc1 what 
happens ? in 1 dc with data loss can the node be “restored” from a node in 
dc2?.  Automatically?

I’m also asking tangentially how does the data map from nodes in dc1 to dc2.

I hope I made that coherent.

Roger

Nodetool repair on read only cluster

[Roger Warner]

What would running a repair on a cluster do when there are no deletes nor have 
there ever been?I have no deletes yet on my data.Yet running a repair 
took over 9 hours on a 5 node cluster?

Roger?