[
https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346911#comment-15346911
]
Philip Thompson edited comment on CASSANDRA-7019 at 6/23/16 6:22 PM:
---------------------------------------------------------------------
Here is the relevant portion of the stress outputs:
CONTROL:
{code}
Results:
Op rate : 21,050 op/s [columndelete: 1,403 op/s, delete:
702 op/s, insert: 10,524 op/s, read: 7,017 op/s, rowdelete: 1,404 op/s]
Partition rate : 17,540 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 10,524 pk/s, read: 7,016 pk/s, rowdelete: 0 pk/s]
Row rate : 43,872 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 26,324 row/s, read: 17,548 row/s, rowdelete: 0 row/s]
Latency mean : 2.4 ms [columndelete: 2.2 ms, delete: 2.1 ms,
insert: 2.2 ms, read: 2.7 ms, rowdelete: 2.1 ms]
Latency median : 2.0 ms [columndelete: 1.8 ms, delete: 1.8 ms,
insert: 1.9 ms, read: 2.3 ms, rowdelete: 1.8 ms]
Latency 95th percentile : 3.6 ms [columndelete: 3.3 ms, delete: 3.3 ms,
insert: 3.4 ms, read: 4.1 ms, rowdelete: 3.3 ms]
Latency 99th percentile : 5.5 ms [columndelete: 4.8 ms, delete: 4.8 ms,
insert: 4.9 ms, read: 7.2 ms, rowdelete: 4.7 ms]
Latency 99.9th percentile : 75.2 ms [columndelete: 69.9 ms, delete: 69.9 ms,
insert: 72.3 ms, read: 78.6 ms, rowdelete: 69.3 ms]
Latency max : 1032.3 ms [columndelete: 1,004.5 ms, delete:
1,004.5 ms, insert: 1,031.8 ms, read: 1,032.3 ms, rowdelete: 1,003.5 ms]
Total partitions : 378,840,394 [columndelete: 0, delete: 0, insert:
227,304,928, read: 151,535,466, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 13,090
Total GC memory : 15900.717 GiB
Total GC time : 998.3 seconds
Avg GC time : 76.3 ms
StdDev GC time : 16.1 ms
Total operation time : 05:59:58
{code}
NONE:
{code}
Results:
Op rate : 20,729 op/s [columndelete: 1,382 op/s, delete:
690 op/s, insert: 10,366 op/s, read: 6,909 op/s, rowdelete: 1,382 op/s]
Partition rate : 17,274 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 10,366 pk/s, read: 6,908 pk/s, rowdelete: 0 pk/s]
Row rate : 43,206 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 25,929 row/s, read: 17,277 row/s, rowdelete: 0 row/s]
Latency mean : 2.4 ms [columndelete: 2.1 ms, delete: 2.1 ms,
insert: 2.2 ms, read: 2.9 ms, rowdelete: 2.1 ms]
Latency median : 1.9 ms [columndelete: 1.7 ms, delete: 1.7 ms,
insert: 1.8 ms, read: 2.2 ms, rowdelete: 1.7 ms]
Latency 95th percentile : 3.3 ms [columndelete: 2.9 ms, delete: 2.9 ms,
insert: 3.0 ms, read: 3.7 ms, rowdelete: 2.9 ms]
Latency 99th percentile : 4.7 ms [columndelete: 4.1 ms, delete: 4.1 ms,
insert: 4.2 ms, read: 6.0 ms, rowdelete: 4.1 ms]
Latency 99.9th percentile : 47.6 ms [columndelete: 12.0 ms, delete: 13.0 ms,
insert: 14.7 ms, read: 67.9 ms, rowdelete: 13.4 ms]
Latency max : 1055.6 ms [columndelete: 1,006.0 ms, delete:
1,004.4 ms, insert: 1,055.6 ms, read: 1,055.6 ms, rowdelete: 1,031.9 ms]
Total partitions : 373,111,699 [columndelete: 0, delete: 0, insert:
223,905,059, read: 149,206,640, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 14,082
Total GC memory : 17120.316 GiB
Total GC time : 1,005.7 seconds
Avg GC time : 71.4 ms
StdDev GC time : 13.5 ms
Total operation time : 06:00:00
{code}
ROW:
{code}
Results:
Op rate : 16,121 op/s [columndelete: 1,075 op/s, delete:
538 op/s, insert: 8,061 op/s, read: 5,372 op/s, rowdelete: 1,074 op/s]
Partition rate : 13,432 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 8,061 pk/s, read: 5,371 pk/s, rowdelete: 0 pk/s]
Row rate : 33,597 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 20,165 row/s, read: 13,433 row/s, rowdelete: 0 row/s]
Latency mean : 3.1 ms [columndelete: 2.3 ms, delete: 2.3 ms,
insert: 2.4 ms, read: 4.5 ms, rowdelete: 2.3 ms]
Latency median : 2.3 ms [columndelete: 1.8 ms, delete: 1.7 ms,
insert: 1.8 ms, read: 3.5 ms, rowdelete: 1.8 ms]
Latency 95th percentile : 5.3 ms [columndelete: 3.8 ms, delete: 3.8 ms,
insert: 3.8 ms, read: 6.8 ms, rowdelete: 3.8 ms]
Latency 99th percentile : 8.4 ms [columndelete: 6.1 ms, delete: 6.2 ms,
insert: 6.2 ms, read: 10.9 ms, rowdelete: 6.3 ms]
Latency 99.9th percentile : 51.4 ms [columndelete: 16.4 ms, delete: 15.7 ms,
insert: 14.7 ms, read: 58.7 ms, rowdelete: 41.4 ms]
Latency max : 1053.8 ms [columndelete: 1,003.5 ms, delete:
1,053.7 ms, insert: 1,053.7 ms, read: 1,053.8 ms, rowdelete: 1,006.9 ms]
Total partitions : 290,123,117 [columndelete: 0, delete: 0, insert:
174,120,120, read: 116,002,997, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 16,954
Total GC memory : 20848.789 GiB
Total GC time : 819.7 seconds
Avg GC time : 48.3 ms
StdDev GC time : 10.9 ms
Total operation time : 05:59:58
{code}
CELL:
{code}
Results:
Op rate : 5,876 op/s [columndelete: 392 op/s, delete: 196
op/s, insert: 2,937 op/s, read: 1,959 op/s, rowdelete: 392 op/s]
Partition rate : 4,896 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 2,937 pk/s, read: 1,959 pk/s, rowdelete: 0 pk/s]
Row rate : 12,247 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 7,348 row/s, read: 4,899 row/s, rowdelete: 0 row/s]
Latency mean : 8.5 ms [columndelete: 3.3 ms, delete: 3.3 ms,
insert: 3.4 ms, read: 18.8 ms, rowdelete: 3.3 ms]
Latency median : 3.2 ms [columndelete: 1.7 ms, delete: 1.7 ms,
insert: 1.7 ms, read: 14.7 ms, rowdelete: 1.7 ms]
Latency 95th percentile : 25.6 ms [columndelete: 8.0 ms, delete: 7.9 ms,
insert: 8.2 ms, read: 34.5 ms, rowdelete: 8.0 ms]
Latency 99th percentile : 39.6 ms [columndelete: 13.5 ms, delete: 13.6 ms,
insert: 14.2 ms, read: 49.2 ms, rowdelete: 13.6 ms]
Latency 99.9th percentile : 65.2 ms [columndelete: 35.2 ms, delete: 34.4 ms,
insert: 34.5 ms, read: 76.2 ms, rowdelete: 31.3 ms]
Latency max : 1037.8 ms [columndelete: 208.5 ms, delete: 1,002.7
ms, insert: 1,006.2 ms, read: 1,037.8 ms, rowdelete: 231.9 ms]
Total partitions : 105,729,389 [columndelete: 0, delete: 0, insert:
63,434,249, read: 42,295,140, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 14,437
Total GC memory : 17937.624 GiB
Total GC time : 348.4 seconds
Avg GC time : 24.1 ms
StdDev GC time : 7.1 ms
Total operation time : 05:59:54
{code}
All were generated with the following stress command: {{cassandra-stress user
duration=6h cl=LOCAL_ONE
profile=https://gist.githubusercontent.com/ptnapoleon/ce59cc5a2a42ecfd3d27657f311e4615/raw/fc85d9d715eb6ed87dba10796e505e2b5ef2c7f8/7019-none.yaml
ops\(insert=15,read=10,delete=1,rowdelete=2,columndelete=2\) -rate threads=50
-pop seq=1..750000000 -node 172.31.9.18 -graph
file=/home/automaton/fallout_artifacts/delete_heavy-graph.html | tee
/home/automaton/fallout_artifacts/stress-delete_heavy.log "}} with the obvious
disparity that the linked yaml file changes the relevant compaction option in
the schema.
As you can see, the {{none}} and {{control}} options are very similar. {{row}}
sees a ~20% dip in perf, and {{cell}} sees an incredibly large one, roughly 75%
of throughput is lost.
was (Author: philipthompson):
Here is the relevant portion of the stress outputs:
CONTROL:
{code}
Results:
Op rate : 21,050 op/s [columndelete: 1,403 op/s, delete:
702 op/s, insert: 10,524 op/s, read: 7,017 op/s, rowdelete: 1,404 op/s]
Partition rate : 17,540 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 10,524 pk/s, read: 7,016 pk/s, rowdelete: 0 pk/s]
Row rate : 43,872 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 26,324 row/s, read: 17,548 row/s, rowdelete: 0 row/s]
Latency mean : 2.4 ms [columndelete: 2.2 ms, delete: 2.1 ms,
insert: 2.2 ms, read: 2.7 ms, rowdelete: 2.1 ms]
Latency median : 2.0 ms [columndelete: 1.8 ms, delete: 1.8 ms,
insert: 1.9 ms, read: 2.3 ms, rowdelete: 1.8 ms]
Latency 95th percentile : 3.6 ms [columndelete: 3.3 ms, delete: 3.3 ms,
insert: 3.4 ms, read: 4.1 ms, rowdelete: 3.3 ms]
Latency 99th percentile : 5.5 ms [columndelete: 4.8 ms, delete: 4.8 ms,
insert: 4.9 ms, read: 7.2 ms, rowdelete: 4.7 ms]
Latency 99.9th percentile : 75.2 ms [columndelete: 69.9 ms, delete: 69.9 ms,
insert: 72.3 ms, read: 78.6 ms, rowdelete: 69.3 ms]
Latency max : 1032.3 ms [columndelete: 1,004.5 ms, delete:
1,004.5 ms, insert: 1,031.8 ms, read: 1,032.3 ms, rowdelete: 1,003.5 ms]
Total partitions : 378,840,394 [columndelete: 0, delete: 0, insert:
227,304,928, read: 151,535,466, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 13,090
Total GC memory : 15900.717 GiB
Total GC time : 998.3 seconds
Avg GC time : 76.3 ms
StdDev GC time : 16.1 ms
Total operation time : 05:59:58
{code}
NONE:
{code}
Results:
Op rate : 20,729 op/s [columndelete: 1,382 op/s, delete:
690 op/s, insert: 10,366 op/s, read: 6,909 op/s, rowdelete: 1,382 op/s]
Partition rate : 17,274 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 10,366 pk/s, read: 6,908 pk/s, rowdelete: 0 pk/s]
Row rate : 43,206 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 25,929 row/s, read: 17,277 row/s, rowdelete: 0 row/s]
Latency mean : 2.4 ms [columndelete: 2.1 ms, delete: 2.1 ms,
insert: 2.2 ms, read: 2.9 ms, rowdelete: 2.1 ms]
Latency median : 1.9 ms [columndelete: 1.7 ms, delete: 1.7 ms,
insert: 1.8 ms, read: 2.2 ms, rowdelete: 1.7 ms]
Latency 95th percentile : 3.3 ms [columndelete: 2.9 ms, delete: 2.9 ms,
insert: 3.0 ms, read: 3.7 ms, rowdelete: 2.9 ms]
Latency 99th percentile : 4.7 ms [columndelete: 4.1 ms, delete: 4.1 ms,
insert: 4.2 ms, read: 6.0 ms, rowdelete: 4.1 ms]
Latency 99.9th percentile : 47.6 ms [columndelete: 12.0 ms, delete: 13.0 ms,
insert: 14.7 ms, read: 67.9 ms, rowdelete: 13.4 ms]
Latency max : 1055.6 ms [columndelete: 1,006.0 ms, delete:
1,004.4 ms, insert: 1,055.6 ms, read: 1,055.6 ms, rowdelete: 1,031.9 ms]
Total partitions : 373,111,699 [columndelete: 0, delete: 0, insert:
223,905,059, read: 149,206,640, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 14,082
Total GC memory : 17120.316 GiB
Total GC time : 1,005.7 seconds
Avg GC time : 71.4 ms
StdDev GC time : 13.5 ms
Total operation time : 06:00:00
{code}
ROW:
Results:
Op rate : 16,121 op/s [columndelete: 1,075 op/s, delete:
538 op/s, insert: 8,061 op/s, read: 5,372 op/s, rowdelete: 1,074 op/s]
Partition rate : 13,432 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 8,061 pk/s, read: 5,371 pk/s, rowdelete: 0 pk/s]
Row rate : 33,597 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 20,165 row/s, read: 13,433 row/s, rowdelete: 0 row/s]
Latency mean : 3.1 ms [columndelete: 2.3 ms, delete: 2.3 ms,
insert: 2.4 ms, read: 4.5 ms, rowdelete: 2.3 ms]
Latency median : 2.3 ms [columndelete: 1.8 ms, delete: 1.7 ms,
insert: 1.8 ms, read: 3.5 ms, rowdelete: 1.8 ms]
Latency 95th percentile : 5.3 ms [columndelete: 3.8 ms, delete: 3.8 ms,
insert: 3.8 ms, read: 6.8 ms, rowdelete: 3.8 ms]
Latency 99th percentile : 8.4 ms [columndelete: 6.1 ms, delete: 6.2 ms,
insert: 6.2 ms, read: 10.9 ms, rowdelete: 6.3 ms]
Latency 99.9th percentile : 51.4 ms [columndelete: 16.4 ms, delete: 15.7 ms,
insert: 14.7 ms, read: 58.7 ms, rowdelete: 41.4 ms]
Latency max : 1053.8 ms [columndelete: 1,003.5 ms, delete:
1,053.7 ms, insert: 1,053.7 ms, read: 1,053.8 ms, rowdelete: 1,006.9 ms]
Total partitions : 290,123,117 [columndelete: 0, delete: 0, insert:
174,120,120, read: 116,002,997, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 16,954
Total GC memory : 20848.789 GiB
Total GC time : 819.7 seconds
Avg GC time : 48.3 ms
StdDev GC time : 10.9 ms
Total operation time : 05:59:58
{code}
CELL:
{code}
Results:
Op rate : 5,876 op/s [columndelete: 392 op/s, delete: 196
op/s, insert: 2,937 op/s, read: 1,959 op/s, rowdelete: 392 op/s]
Partition rate : 4,896 pk/s [columndelete: 0 pk/s, delete: 0
pk/s, insert: 2,937 pk/s, read: 1,959 pk/s, rowdelete: 0 pk/s]
Row rate : 12,247 row/s [columndelete: 0 row/s, delete: 0
row/s, insert: 7,348 row/s, read: 4,899 row/s, rowdelete: 0 row/s]
Latency mean : 8.5 ms [columndelete: 3.3 ms, delete: 3.3 ms,
insert: 3.4 ms, read: 18.8 ms, rowdelete: 3.3 ms]
Latency median : 3.2 ms [columndelete: 1.7 ms, delete: 1.7 ms,
insert: 1.7 ms, read: 14.7 ms, rowdelete: 1.7 ms]
Latency 95th percentile : 25.6 ms [columndelete: 8.0 ms, delete: 7.9 ms,
insert: 8.2 ms, read: 34.5 ms, rowdelete: 8.0 ms]
Latency 99th percentile : 39.6 ms [columndelete: 13.5 ms, delete: 13.6 ms,
insert: 14.2 ms, read: 49.2 ms, rowdelete: 13.6 ms]
Latency 99.9th percentile : 65.2 ms [columndelete: 35.2 ms, delete: 34.4 ms,
insert: 34.5 ms, read: 76.2 ms, rowdelete: 31.3 ms]
Latency max : 1037.8 ms [columndelete: 208.5 ms, delete: 1,002.7
ms, insert: 1,006.2 ms, read: 1,037.8 ms, rowdelete: 231.9 ms]
Total partitions : 105,729,389 [columndelete: 0, delete: 0, insert:
63,434,249, read: 42,295,140, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0,
read: 0, rowdelete: 0]
Total GC count : 14,437
Total GC memory : 17937.624 GiB
Total GC time : 348.4 seconds
Avg GC time : 24.1 ms
StdDev GC time : 7.1 ms
Total operation time : 05:59:54
{code}
All were generated with the following stress command: {{cassandra-stress user
duration=6h cl=LOCAL_ONE
profile=https://gist.githubusercontent.com/ptnapoleon/ce59cc5a2a42ecfd3d27657f311e4615/raw/fc85d9d715eb6ed87dba10796e505e2b5ef2c7f8/7019-none.yaml
ops\(insert=15,read=10,delete=1,rowdelete=2,columndelete=2\) -rate threads=50
-pop seq=1..750000000 -node 172.31.9.18 -graph
file=/home/automaton/fallout_artifacts/delete_heavy-graph.html | tee
/home/automaton/fallout_artifacts/stress-delete_heavy.log "}} with the obvious
disparity that the linked yaml file changes the relevant compaction option in
the schema.
As you can see, the {{none}} and {{control}} options are very similar. {{row}}
sees a ~20% dip in perf, and {{cell}} sees an incredibly large one, roughly 75%
of throughput is lost.
> Improve tombstone compactions
> -----------------------------
>
> Key: CASSANDRA-7019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7019
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
> Reporter: Marcus Eriksson
> Assignee: Branimir Lambov
> Labels: compaction, fallout
> Fix For: 3.x
>
> Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz,
> control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html
>
>
> When there are no other compactions to do, we trigger a single-sstable
> compaction if there is more than X% droppable tombstones in the sstable.
> In this ticket we should try to include overlapping sstables in those
> compactions to be able to actually drop the tombstones. Might only be doable
> with LCS (with STCS we would probably end up including all sstables)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
