Re: 3.0.15 or 3.11.1

[Mick Semb Wever]

>
> Can you please provide dome JIRAs for superior fixes and performance
> improvements which are present in 3.11.1 but are missing in 3.0.15.
>


Some that come to mind…

Cassandra Storage Engine: CASSANDRA-12269, CASSANDRA-12731

Streaming and Compaction: CASSANDRA-11206, CASSANDRA-9766, CASSANDRA-11623,

Reintroduce off heap memtables –  CASSANDRA-9472

Re: NVMe SSD benchmarking with Cassandra

[Nate McCall]

>
>
>
>
> In regards to setting read ahead, how is this set for nvme drives? Also,
> below is our compression settings for the table… It’s the same as our tests
> that we are doing against SAS SSDs so I don’t think the compression
> settings would be the issue…
>
>
>

Check blockdev --report between the old and the new servers to see if there
is a difference. Are there other deltas in the disk layouts between the old
and new servers (ie. LVM, mdadm, etc.)?

You can control read ahead via 'blockdev --setra' or via poking the kernel:
/sys/block/[YOUR DRIVE]/queue/read_ahead_kb

In both cases, changes are instantaneous so you can do it on a canary and
monitor for effect.

Also, i'd be curious to know (since you have this benchmark setup) if you
got the degradation you are currently seeing if you set concurrent_reads
and concurrent_writes back to their defaults.


-- 
-
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

RE: NVMe SSD benchmarking with Cassandra

[Justin Sanciangco]

Hi Jeff,

In regards to setting read ahead, how is this set for nvme drives? Also, below 
is our compression settings for the table… It’s the same as our tests that we 
are doing against SAS SSDs so I don’t think the compression settings would be 
the issue…

CREATE KEYSPACE ycsb WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;

CREATE TABLE ycsb.usertable (
y_id text PRIMARY KEY,
field0 text,
field1 text,
field2 text,
field3 text,
field4 text,
field5 text,
field6 text,
field7 text,
field8 text,
field9 text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

Below are the metrics as far as TPS output from the YCSB benchmark...

DBWrapper: report latency for each error is false and specific error codes to 
track for latency are: []
2018-01-08 21:50:49:100 10 sec: 1048634 operations; 104863.4 current ops/sec; 
est completion in 15 minutes [INSERT: Count=1048634, Max=291071, Min=194, 
Avg=417.22, 90=463, 99=947, 99.9=5531, 99.99=136831]
2018-01-08 21:50:59:100 20 sec: 2159133 operations; 111049.9 current ops/sec; 
est completion in 15 minutes [INSERT: Count=1110545, Max=409087, Min=194, 
Avg=434.17, 90=450, 99=612, 99.9=3409, 99.99=294911]
2018-01-08 21:51:09:101 30 sec: 3092963 operations; 93383 current ops/sec; est 
completion in 15 minutes [INSERT: Count=933938, Max=460287, Min=193, Avg=511.4, 
90=470, 99=750, 99.9=6055, 99.99=429823]
2018-01-08 21:51:19:100 40 sec: 4153712 operations; 106074.9 current ops/sec; 
est completion in 15 minutes [INSERT: Count=1060595, Max=388095, Min=194, 
Avg=434.08, 90=457, 99=604, 99.9=3261, 99.99=335103]
2018-01-08 21:51:29:100 50 sec: 5165150 operations; 101143.8 current ops/sec; 
est completion in 15 minutes [INSERT: Count=1011537, Max=419839, Min=189, 
Avg=488.41, 90=462, 99=666, 99.9=4057, 99.99=397823]
2018-01-08 21:51:39:100 60 sec: 6151282 operations; 98613.2 current ops/sec; 
est completion in 15 minutes [INSERT: Count=986033, Max=408575, Min=196, 
Avg=474.68, 90=467, 99=671, 99.9=5463, 99.99=375807]
2018-01-08 21:51:49:100 70 sec: 7171184 operations; 101990.2 current ops/sec; 
est completion in 15 minutes [INSERT: Count=1019962, Max=406783, Min=189, 
Avg=477.11, 90=468, 99=725, 99.9=4855, 99.99=364031]
2018-01-08 21:51:59:100 80 sec: 8154478 operations; 98329.4 current ops/sec; 
est completion in 15 minutes [INSERT: Count=983234, Max=391423, Min=188, 
Avg=473.52, 90=465, 99=653, 99.9=4751, 99.99=346623]
2018-01-08 21:52:09:100 90 sec: 9204270 operations; 104979.2 current ops/sec; 
est completion in 14 minutes [INSERT: Count=1049855, Max=366335, Min=194, 
Avg=465.83, 90=466, 99=690, 99.9=4207, 99.99=347391]
2018-01-08 21:52:19:100 100 sec: 10191251 operations; 98698.1 current ops/sec; 
est completion in 14 minutes [INSERT: Count=986982, Max=337663, Min=191, 
Avg=483.67, 90=466, 99=707, 99.9=5495, 99.99=323583]
2018-01-08 21:52:29:100 110 sec: 8897 operations; 92764.6 current ops/sec; 
est completion in 14 minutes [INSERT: Count=927649, Max=324607, Min=195, 
Avg=514.77, 90=490, 99=798, 99.9=7939, 99.99=314111]
2018-01-08 21:52:39:100 120 sec: 12106226 operations; 98732.9 current ops/sec; 
est completion in 14 minutes [INSERT: Count=987327, Max=327423, Min=191, 
Avg=483.53, 90=475, 99=749, 99.9=6303, 99.99=291583]
2018-01-08 21:52:49:100 130 sec: 12406781 operations; 30055.5 current ops/sec; 
est completion in 15 minutes [INSERT: Count=300545, Max=2267135, Min=195, 
Avg=1594.21, 90=551, 99=1412, 99.9=268031, 99.99=2059263]
2018-01-08 21:52:59:100 140 sec: 12455737 operations; 4895.6 current ops/sec; 
est completion in 16 minutes [INSERT: Count=48901, Max=2637823, Min=208, 
Avg=8719.47, 90=570, 99=1775, 99.9=2435071, 99.99=2637823]
2018-01-08 21:53:09:100 150 sec: 12545132 operations; 8939.5 current ops/sec; 
est completion in 17 minutes [INSERT: Count=89395, Max=2040831, Min=196, 
Avg=5236.29, 90=603, 99=3103, 99.9=1419263, 99.99=2039807]
2018-01-08 21:53:19:100 160 sec: 12713856 operations; 16872.4 current ops/sec; 
est completion in 18 minutes [INSERT: Count=168724, Max=3260415, Min=201, 
Avg=3212.66, 90=505, 99=825, 99.9=1442815, 99.99=3256319]
2018-01-08 21:53:29:100 170 sec: 13014136 operations; 30028 current ops/sec; 
est completion in 18 minutes [INSERT: Count=300280, Max=3291135, Min=195, 
Avg=1398.45, 90=486, 99=722, 

Re: how to check C* partition size

[Ahmed Eljami]

​>Nodetool tablestats gives you a general idea.

Since C* 3.X :)

RE: how to check C* partition size

[Meg Mara]

Nodetool tablestats gives you a general idea.

Meg Mara


From: Peng Xiao [mailto:2535...@qq.com]
Sent: Sunday, January 07, 2018 9:26 AM
To: user 
Subject: how to check C* partition size

Hi guys,

Could anyone please help on this simple question?
How to check C* partition size and related information.
looks nodetool ring only shows the token distribution.

Thanks

Re: Rebuild to a new DC fails every time

[Martin Mačura]

None of the files is listed more than once in the logs:

java.lang.RuntimeException: Transfer of file
/fs3/cassandra/data//event_group-3b5782d08e4411e6842917253f111990/mc-116042-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs0/cassandra/data//event_group-3b5782d08e4411e6842917253f111990/mc-111370-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs3/cassandra/data//event_alert-13d78e3f11e6a6cbe1698349da4d/mc-8659-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs4/cassandra/data//event_alert-13d78e3f11e6a6cbe1698349da4d/mc-9133-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs2/cassandra/data//event_alert-13d78e3f11e6a6cbe1698349da4d/mc-3997-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs1/cassandra/data///event_group-3b5782d08e4411e6842917253f111990/mc-152979-big-Data.db
already completed or aborted (perhaps session failed?).




On Mon, Jan 8, 2018 at 2:21 AM, kurt greaves  wrote:
> If you're on 3.9 it's likely unrelated as streaming_socket_timeout_in_ms is
> 48 hours. Appears rebuild is trying to stream the same file twice. Are there
> other exceptions in the logs related to the file, or can you find out if
> it's previously been sent by the same session? Search the logs for the file
> that failed and post back any exceptions.
>
> On 29 December 2017 at 10:18, Martin Mačura  wrote:
>>
>> Is this something that can be resolved by CASSANDRA-11841 ?
>>
>> Thanks,
>>
>> Martin
>>
>> On Thu, Dec 21, 2017 at 3:02 PM, Martin Mačura  wrote:
>> > Hi all,
>> > we are trying to add a new datacenter to the existing cluster, but the
>> > 'nodetool rebuild' command always fails after a couple of hours.
>> >
>> > We're on Cassandra 3.9.
>> >
>> > Example 1:
>> >
>> > 172.24.16.169 INFO  [STREAM-IN-/172.25.16.125:55735] 2017-12-13
>> > 23:55:38,840 StreamResultFuture.java:174 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
>> > Receiving 0 files(0.000KiB), sending 9844 files(885.587GiB)
>> > 172.25.16.125 INFO  [STREAM-IN-/172.24.16.169:7000] 2017-12-13
>> > 23:55:38,858 StreamResultFuture.java:174 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
>> > Receiving 9844 files(885.587GiB), sending 0 files(0.000KiB)
>> >
>> > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:55735] 2017-12-14
>> > 04:28:09,064 StreamSession.java:533 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.io.IOException: Connection reset by peer
>> >
>> > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:49412] 2017-12-14
>> > 07:26:26,832 StreamSession.java:533 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.lang.RuntimeException: Transfer of file
>> > -13d78e3f11e6a6cbe1698349da4d/mc-8659-big-Data.db
>> > already completed or aborted (perhaps session failed?).
>> > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-14
>> > 07:26:50,004 StreamSession.java:533 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
>> > session with peer 172.24.16.169
>> > 172.25.16.125 java.io.IOException: Connection reset by peer
>> >
>> > Example 2:
>> >
>> > 172.24.16.169 INFO  [STREAM-IN-/172.25.16.125:35202] 2017-12-18
>> > 03:24:31,423 StreamResultFuture.java:174 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
>> > Receiving 0 files(0.000KiB), sending 12312 files(895.973GiB)
>> > 172.25.16.125 INFO  [STREAM-IN-/172.24.16.169:7000] 2017-12-18
>> > 03:24:31,441 StreamResultFuture.java:174 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
>> > Receiving 12312 files(895.973GiB), sending 0 files(0.000KiB)
>> >
>> > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:35202] 2017-12-18
>> > 06:39:42,049 StreamSession.java:533 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.io.IOException: Connection reset by peer
>> >
>> > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:42744] 2017-12-18
>> > 09:25:36,188 StreamSession.java:533 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.lang.RuntimeException: Transfer of file
>> > -3b5782d08e4411e6842917253f111990/mc-152979-big-Data.db
>> > already completed or aborted (perhaps session failed?).
>> > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-18
>> > 09:25:59,447 StreamSession.java:533 - [Stream

Re: Compaction: ThreadPool Metrics vs Compaction Metrics

[Lucas Benevides]

Hello Ahmed,

I have questioned about the compaction metrics in 27/10/2017.
You may see the conversation here

:

Lucas Benevides

2018-01-05 14:13 GMT-02:00 Ahmed Eljami :

> ​Hello,
>
> ​​Could someone explain me the difference between the values of the two
> following metrics​:
>
> *​ThreadPool Metrics:​CompactionExecutor:CompletedTasks* vs *Compaction
> Metrics:CompletedTasks*
>
> I do not the same value when I query JMX!
>
> Thanks
>
>
>