[
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656030#comment-16656030
]
Ariel Weisberg edited comment on CASSANDRA-13241 at 10/19/18 1:35 PM:
----------------------------------------------------------------------
Running
{noformat}
#!/bin/sh
echo "drop keyspace keyspace1;" | ../../bin/cqlsh
./cassandra-stress write no-warmup n=100000000 -pop seq=1...100000000 -schema
compression=LZ4Compressor
./cassandra-stress read no-warmup n=10000000 -pop dist=UNIFORM\(1...100000000\)
-rate threads=32
{noformat}
64k load
{noformat}
Results:
Op rate : 42,237 op/s [WRITE: 42,254 op/s]
Partition rate : 42,237 pk/s [WRITE: 42,254 pk/s]
Row rate : 42,237 row/s [WRITE: 42,254 row/s]
Latency mean : 4.7 ms [WRITE: 4.7 ms]
Latency median : 1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.2 ms [WRITE: 13.2 ms]
Latency 99th percentile : 85.3 ms [WRITE: 85.3 ms]
Latency 99.9th percentile : 230.0 ms [WRITE: 230.0 ms]
Latency max : 629.1 ms [WRITE: 629.1 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:39:27
{noformat}
{noformat}
Op rate : 45,504 op/s [WRITE: 45,504 op/s]
Partition rate : 45,504 pk/s [WRITE: 45,504 pk/s]
Row rate : 45,504 row/s [WRITE: 45,504 row/s]
Latency mean : 4.4 ms [WRITE: 4.4 ms]
Latency median : 1.5 ms [WRITE: 1.5 ms]
Latency 95th percentile : 12.0 ms [WRITE: 12.0 ms]
Latency 99th percentile : 77.7 ms [WRITE: 77.7 ms]
Latency 99.9th percentile : 212.7 ms [WRITE: 212.7 ms]
Latency max : 501.5 ms [WRITE: 501.5 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:36:37
{noformat}
64k read
{noformat}
Results:
Op rate : 6,576 op/s [READ: 6,576 op/s]
Partition rate : 6,576 pk/s [READ: 6,576 pk/s]
Row rate : 6,576 row/s [READ: 6,576 row/s]
Latency mean : 4.8 ms [READ: 4.8 ms]
Latency median : 3.0 ms [READ: 3.0 ms]
Latency 95th percentile : 12.9 ms [READ: 12.9 ms]
Latency 99th percentile : 32.6 ms [READ: 32.6 ms]
Latency 99.9th percentile : 100.8 ms [READ: 100.8 ms]
Latency max : 14982.1 ms [READ: 14,982.1 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:25:20
{noformat}
{noformat}
Op rate : 6,523 op/s [READ: 6,523 op/s]
Partition rate : 6,523 pk/s [READ: 6,523 pk/s]
Row rate : 6,523 row/s [READ: 6,523 row/s]
Latency mean : 4.9 ms [READ: 4.9 ms]
Latency median : 3.2 ms [READ: 3.2 ms]
Latency 95th percentile : 12.9 ms [READ: 12.9 ms]
Latency 99th percentile : 32.1 ms [READ: 32.1 ms]
Latency 99.9th percentile : 94.1 ms [READ: 94.1 ms]
Latency max : 283.4 ms [READ: 283.4 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:25:33
{noformat}
32k load:
{noformat}
Op rate : 44,463 op/s [WRITE: 44,463 op/s]
Partition rate : 44,463 pk/s [WRITE: 44,463 pk/s]
Row rate : 44,463 row/s [WRITE: 44,463 row/s]
Latency mean : 4.5 ms [WRITE: 4.5 ms]
Latency median : 1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 12.5 ms [WRITE: 12.5 ms]
Latency 99th percentile : 76.7 ms [WRITE: 76.7 ms]
Latency 99.9th percentile : 200.7 ms [WRITE: 200.7 ms]
Latency max : 626.0 ms [WRITE: 626.0 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:37:29
{noformat}
32k read:
{noformat}
Op rate : 8,324 op/s [READ: 8,324 op/s]
Partition rate : 8,324 pk/s [READ: 8,324 pk/s]
Row rate : 8,324 row/s [READ: 8,324 row/s]
Latency mean : 3.8 ms [READ: 3.8 ms]
Latency median : 2.7 ms [READ: 2.7 ms]
Latency 95th percentile : 9.3 ms [READ: 9.3 ms]
Latency 99th percentile : 23.1 ms [READ: 23.1 ms]
Latency 99.9th percentile : 71.6 ms [READ: 71.6 ms]
Latency max : 393.5 ms [READ: 393.5 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:20:01
{noformat}
16k load:
{noformat}
Results:
Op rate : 42,266 op/s [WRITE: 42,266 op/s]
Partition rate : 42,266 pk/s [WRITE: 42,266 pk/s]
Row rate : 42,266 row/s [WRITE: 42,266 row/s]
Latency mean : 4.7 ms [WRITE: 4.7 ms]
Latency median : 1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.1 ms [WRITE: 13.1 ms]
Latency 99th percentile : 83.2 ms [WRITE: 83.2 ms]
Latency 99.9th percentile : 218.1 ms [WRITE: 218.1 ms]
Latency max : 886.0 ms [WRITE: 886.0 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:39:25
{noformat}
{noformat}
Op rate : 43,555 op/s [WRITE: 43,555 op/s]
Partition rate : 43,555 pk/s [WRITE: 43,555 pk/s]
Row rate : 43,555 row/s [WRITE: 43,555 row/s]
Latency mean : 4.6 ms [WRITE: 4.6 ms]
Latency median : 1.5 ms [WRITE: 1.5 ms]
Latency 95th percentile : 12.1 ms [WRITE: 12.1 ms]
Latency 99th percentile : 84.7 ms [WRITE: 84.7 ms]
Latency 99.9th percentile : 250.9 ms [WRITE: 250.9 ms]
Latency max : 591.9 ms [WRITE: 591.9 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:38:15
{noformat}
16k read
{noformat}
Op rate : 8,622 op/s [READ: 8,622 op/s]
Partition rate : 8,622 pk/s [READ: 8,622 pk/s]
Row rate : 8,622 row/s [READ: 8,622 row/s]
Latency mean : 3.7 ms [READ: 3.7 ms]
Latency median : 2.6 ms [READ: 2.6 ms]
Latency 95th percentile : 9.0 ms [READ: 9.0 ms]
Latency 99th percentile : 22.2 ms [READ: 22.2 ms]
Latency 99.9th percentile : 63.5 ms [READ: 63.5 ms]
Latency max : 256.8 ms [READ: 256.8 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:19:19
{noformat}
{noformat}
Op rate : 8,660 op/s [READ: 8,660 op/s]
Partition rate : 8,660 pk/s [READ: 8,660 pk/s]
Row rate : 8,660 row/s [READ: 8,660 row/s]
Latency mean : 3.7 ms [READ: 3.7 ms]
Latency median : 2.6 ms [READ: 2.6 ms]
Latency 95th percentile : 8.9 ms [READ: 8.9 ms]
Latency 99th percentile : 22.6 ms [READ: 22.6 ms]
Latency 99.9th percentile : 69.9 ms [READ: 69.9 ms]
Latency max : 715.7 ms [READ: 715.7 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:19:14
{noformat}
8k load:
{noformat}
Op rate : 44,109 op/s [WRITE: 44,109 op/s]
Partition rate : 44,109 pk/s [WRITE: 44,109 pk/s]
Row rate : 44,109 row/s [WRITE: 44,109 row/s]
Latency mean : 4.5 ms [WRITE: 4.5 ms]
Latency median : 1.4 ms [WRITE: 1.4 ms]
Latency 95th percentile : 12.0 ms [WRITE: 12.0 ms]
Latency 99th percentile : 84.7 ms [WRITE: 84.7 ms]
Latency 99.9th percentile : 251.3 ms [WRITE: 251.3 ms]
Latency max : 752.4 ms [WRITE: 752.4 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:37:47
{noformat}
8k read:
{noformat}
Op rate : 9,889 op/s [READ: 9,889 op/s]
Partition rate : 9,889 pk/s [READ: 9,889 pk/s]
Row rate : 9,889 row/s [READ: 9,889 row/s]
Latency mean : 3.2 ms [READ: 3.2 ms]
Latency median : 2.3 ms [READ: 2.3 ms]
Latency 95th percentile : 8.0 ms [READ: 8.0 ms]
Latency 99th percentile : 19.5 ms [READ: 19.5 ms]
Latency 99.9th percentile : 55.0 ms [READ: 55.0 ms]
Latency max : 301.2 ms [READ: 301.2 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:16:51
{noformat}
4k load:
{noformat}
Op rate : 42,132 op/s [WRITE: 42,132 op/s]
Partition rate : 42,132 pk/s [WRITE: 42,132 pk/s]
Row rate : 42,132 row/s [WRITE: 42,132 row/s]
Latency mean : 4.7 ms [WRITE: 4.7 ms]
Latency median : 1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.5 ms [WRITE: 13.5 ms]
Latency 99th percentile : 80.3 ms [WRITE: 80.3 ms]
Latency 99.9th percentile : 219.5 ms [WRITE: 219.5 ms]
Latency max : 1108.3 ms [WRITE: 1,108.3 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:39:33
{noformat}
4k read:
{noformat}
Op rate : 10,649 op/s [READ: 10,649 op/s]
Partition rate : 10,649 pk/s [READ: 10,649 pk/s]
Row rate : 10,649 row/s [READ: 10,649 row/s]
Latency mean : 3.0 ms [READ: 3.0 ms]
Latency median : 2.2 ms [READ: 2.2 ms]
Latency 95th percentile : 7.3 ms [READ: 7.3 ms]
Latency 99th percentile : 17.1 ms [READ: 17.1 ms]
Latency 99.9th percentile : 40.0 ms [READ: 40.0 ms]
Latency max : 256.2 ms [READ: 256.2 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:15:39
{noformat}
This read workload is 2x faster with 16k chunks vs 64k chunks.
was (Author: aweisberg):
Running
{noformat}
#!/bin/sh
echo "drop keyspace keyspace1;" | ../../bin/cqlsh
./cassandra-stress write no-warmup n=100000000 -pop seq=1...100000000 -schema
compression=LZ4Compressor
./cassandra-stress read no-warmup n=10000000 -pop dist=UNIFORM\(1...100000000\)
-rate threads=32
{noformat}
64k load
{noformat}
Results:
Op rate : 42,237 op/s [WRITE: 42,254 op/s]
Partition rate : 42,237 pk/s [WRITE: 42,254 pk/s]
Row rate : 42,237 row/s [WRITE: 42,254 row/s]
Latency mean : 4.7 ms [WRITE: 4.7 ms]
Latency median : 1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.2 ms [WRITE: 13.2 ms]
Latency 99th percentile : 85.3 ms [WRITE: 85.3 ms]
Latency 99.9th percentile : 230.0 ms [WRITE: 230.0 ms]
Latency max : 629.1 ms [WRITE: 629.1 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:39:27
{noformat}
64k read
{noformat}
Results:
Op rate : 6,576 op/s [READ: 6,576 op/s]
Partition rate : 6,576 pk/s [READ: 6,576 pk/s]
Row rate : 6,576 row/s [READ: 6,576 row/s]
Latency mean : 4.8 ms [READ: 4.8 ms]
Latency median : 3.0 ms [READ: 3.0 ms]
Latency 95th percentile : 12.9 ms [READ: 12.9 ms]
Latency 99th percentile : 32.6 ms [READ: 32.6 ms]
Latency 99.9th percentile : 100.8 ms [READ: 100.8 ms]
Latency max : 14982.1 ms [READ: 14,982.1 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:25:20
{noformat}
16k write
{noformat}
Results:
Op rate : 42,266 op/s [WRITE: 42,266 op/s]
Partition rate : 42,266 pk/s [WRITE: 42,266 pk/s]
Row rate : 42,266 row/s [WRITE: 42,266 row/s]
Latency mean : 4.7 ms [WRITE: 4.7 ms]
Latency median : 1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.1 ms [WRITE: 13.1 ms]
Latency 99th percentile : 83.2 ms [WRITE: 83.2 ms]
Latency 99.9th percentile : 218.1 ms [WRITE: 218.1 ms]
Latency max : 886.0 ms [WRITE: 886.0 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:39:25
{noformat}
16k read
{noformat}
Op rate : 8,622 op/s [READ: 8,622 op/s]
Partition rate : 8,622 pk/s [READ: 8,622 pk/s]
Row rate : 8,622 row/s [READ: 8,622 row/s]
Latency mean : 3.7 ms [READ: 3.7 ms]
Latency median : 2.6 ms [READ: 2.6 ms]
Latency 95th percentile : 9.0 ms [READ: 9.0 ms]
Latency 99th percentile : 22.2 ms [READ: 22.2 ms]
Latency 99.9th percentile : 63.5 ms [READ: 63.5 ms]
Latency max : 256.8 ms [READ: 256.8 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:19:19
{noformat}
This read workload is 2x faster with 16k chunks vs 64k chunks.
> Lower default chunk_length_in_kb from 64kb to 4kb
> -------------------------------------------------
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
> Issue Type: Wish
> Components: Core
> Reporter: Benjamin Roth
> Assignee: Ariel Weisberg
> Priority: Major
> Attachments: CompactIntegerSequence.java,
> CompactIntegerSequenceBench.java, CompactSummingIntegerSequence.java
>
>
> Having a too low chunk size may result in some wasted disk space. A too high
> chunk size may lead to massive overreads and may have a critical impact on
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) /
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but
> if the model consists rather of small rows or small resultsets, the read
> overhead with 64kb chunk size is insanely high. This applies for example for
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic
> snitch magic": https://cl.ly/3E0t1T1z2c0J
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
