[
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424092#comment-15424092
]
Marcus Eriksson commented on CASSANDRA-10540:
---------------------------------------------
These "benchmarks" have been run using cassandra-stress with
[this|https://paste.fscking.us/display/jKc0X89MLFzHE9jhRqQ5xfvRHeU] yaml (only
modified per run with the different compaction configurations).
cassandra-stress generates 40GB of data and then it compacts those sstables
using 8 threads. All tests were run with 256 tokens on my machine (2 ssds, 32GB
ram):
{code}
./tools/bin/compaction-stress write -d /var/lib/cassandra -d
/home/marcuse/cassandra -g 40 -p blogpost-range.yaml -t 4 -v 256
./tools/bin/compaction-stress compact -d /var/lib/cassandra -d
/home/marcuse/cassandra -p blogpost-range.yaml -t 8 -v 256
{code}
First a base line - it takes about 7 minutes to compact 40GB of data with STCS,
and we get a write amplification (compaction bytes written / size before) of
about 1.46.
* 40GB + STCS
||size before||size after||compaction bytes written||time||number of
compactions||
|42986704571|31305948786|62268272752|7:44|26|
|43017694284|31717603488|62800073327|7:04|26|
|42863193047|31244649872|64673778727|6:44|26|
|42962733336|31842455113|62985984309|6:14|26|
|43107421526|32526047125|61657717328|6:04|26|
With range aware compaction and a small min_range_sstable_size_in_mb we compact
slower, about 2x the time, but the end result is smaller with a tiny bit smaller
write amplification (1.44). The reason for the longer time is that we need to
do a lot more tiny compaction for each vnode. The reason for the smaller size
after the compactions is that we are much more likely to compact overlapping
sstables together as we compact within each vnode.
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 1
||size before||size after||compaction bytes written||time||number of
compactions||
|42944940703|25352795435|61734295478|13:18|286|
|42896304174|25830662102|62049066195|15:45|287|
|43091495756|24811367911|61448601743|12:25|287|
|42961529234|26275106863|63118850488|13:17|284|
|42902111497|25749453764|61529524300|13:54|280|
As we increase the min_range_sstable_size_in_mb the time spent is reduced, the
size after the compaction is increased and the number of compactions is reduced
since we don't promote sstables to the per-vnode-strategies as quickly. With
large enough min_range_sstable_size_in_mb the behaviour will be the same as
STCS (+a small overhead for estimating the size of the next vnode range during
compaction)
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 5
||size before||size after||compaction bytes written||time||number of
compactions||
|43071111106|27586259306|62855258024|10:35|172|
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 10
||size before||size after||compaction bytes written||time||number of
compactions||
|42998501805|28281735688|65469323764|9:45|109|
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 20
||size before||size after||compaction bytes written||time||number of
compactions||
|42801860659|28934194973|66554340039|10:05|48|
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 50
||size before||size after||compaction bytes written||time||number of
compactions||
|42881416448|30352758950|61223610818|7:25|27|
With LCS and a small sstable_size_in_mb we get a huge difference with range
aware due to the amount of compactions we need to do to get the leveling
without range aware compaction. With range aware, we get fewer levels in each
vnode-range and that is much quicker to compact. Write amplification is about
2.0 with range aware and 3.4 without.
* 40GB + LCS + sstable_size_in_mb: 10 + range_aware +
min_range_sstable_size_in_mb: 10
||size before||size after||compaction bytes written||time||number of
compactions||
|43170254812|26511935628|87637370434|19:55|903|
|43015904097|26100197485|83125478305|14:45|854|
|43188886684|25651102691|87520409116|19:55|920|
* 40GB + LCS + sstable_size_in_mb: 10
||size before||size after||compaction bytes written||time||number of
compactions||
|43099495889|23876144309|139000531662|28:25|3751|
|42811000078|24620085107|147722973544|30:35|3909|
|42879141849|24479485292|146194679395|30:46|3882|
If we bump the lcs sstable_size_in_mb to the default we get more similar
results. Write amplification is smaller with range aware compaction but size
after is also bigger. The reason for the bigger size after compaction has
settled is that we run with a bigger min_range_sstable_size_in_mb which means
more data will stay out of the per-range compaction strategies and this means
it is only size tiered. This probably also explains the reduced write
amplification - 2.0 with range aware and 2.3 without.
* 40GB + LCS + sstable_size_in_mb: 160 + range_aware +
min_range_sstable_size_in_mb: 20
||size before||size after||compaction bytes written||time||number of
compactions||
|42970784099|27044941599|85933586287|12:55|180|
|42953512565|26229232777|82158863291|11:36|155|
|43028281629|26025950993|86704157660|11:25|177|
* 40GB + LCS + sstable_size_in_mb: 160
||size before||size after||compaction bytes written||time||number of
compactions||
|43120992697|24487560567|100347633105|12:25|151|
|42854926611|24466503628|102492898148|10:55|155|
|42919253642|24831918330|100902215961|12:15|161|
> RangeAwareCompaction
> --------------------
>
> Key: CASSANDRA-10540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10540
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Marcus Eriksson
> Assignee: Marcus Eriksson
> Labels: compaction, lcs, vnodes
> Fix For: 3.x
>
>
> Broken out from CASSANDRA-6696, we should split sstables based on ranges
> during compaction.
> Requirements;
> * dont create tiny sstables - keep them bunched together until a single vnode
> is big enough (configurable how big that is)
> * make it possible to run existing compaction strategies on the per-range
> sstables
> We should probably add a global compaction strategy parameter that states
> whether this should be enabled or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
