[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602879#comment-16602879 ] Marcus Eriksson commented on CASSANDRA-10540: - this patch also needs to handle transient replication (CASSANDRA-14404) - we should split transient ranges evenly over all disks to avoid unbalance > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: 4.0-feature-freeze-review-requested, compaction, lcs, > vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589731#comment-16589731 ] Marcus Eriksson commented on CASSANDRA-10540: - bq. if that's all it needed we could of had them written by now. well writing tests will most likely require refactoring and fixing issues found, so it is most likely not all that is needed that is the latest branch, correct > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: 4.0-feature-freeze-review-requested, compaction, lcs, > vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589503#comment-16589503 ] Kurt Greaves commented on CASSANDRA-10540: -- We've been asking to help for months now, if that's all it needed we could of had them written by now. [This|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/10540] is your latest branch correct? I'll start looking at writing tests, although there's minimal hope of having anything ready by the first. > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: 4.0-feature-freeze-review-requested, compaction, lcs, > vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588741#comment-16588741 ] Marcus Eriksson commented on CASSANDRA-10540: - [~KurtG] I'm afraid not, the patch has basically no unit tests, I would need to spend a couple of weeks on getting it in an acceptable state > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: 4.0-feature-freeze-review-requested, compaction, lcs, > vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588733#comment-16588733 ] Kurt Greaves commented on CASSANDRA-10540: -- [~krummas] Any hope of getting this in by 4.0? I'd be committed to testing/verifying/benchmarking post freeze if we can get it in. Code-wise seems fine to me, and our testing so far seems to show that it's viable. Any reason we can't get this committed and fix/revert during freeze if issues come up? > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: 4.0-feature-freeze-review-requested, compaction, lcs, > vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550267#comment-16550267 ] Kurt Greaves commented on CASSANDRA-10540: -- [~krummas] We didn't run into any issues after applying the repair patch from CASSANDRA-13938. We'll probably look at doing a few more tests with repair to measure repair timings with RACS enabled. Let me know if there's anything you need help with to get this in by 4.0. > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: compaction, lcs, vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527299#comment-16527299 ] Marcus Eriksson commented on CASSANDRA-10540: - Hey [~Lerh Low] thanks so much for the testing, I will look in to the results soon, I assume you didn't find any issues with the patch? > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: compaction, lcs, vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519900#comment-16519900
]
Lerh Chuan Low commented on CASSANDRA-10540:
Here is another benchmark run. It is still the same stressspec YAML. This time,
the process is to stop one of the nodes in a DC (Same as before, 3 in 1 DC and
2 in the other), and then insert for 10 minutes.
{code:java}
nohup cassandra-stress user no-warmup profile=stressspec.yaml duration=10m
cl=QUORUM ops\(insert=1\) -node file=nodelist.txt -rate threads=50 -log
file=insert.log > nohup.txt &
{code}
After that, trigger a stress but at the same time run a full repair in the DC:
{code:java}
nohup cassandra-stress user no-warmup profile=stressspec.yaml duration=1h
cl=QUORUM ops\(insert=10,simple1=10,range1=1\) -node file=nodelist.txt -rate
threads=50 -log file=mixed.log > nohup.txt &
nohup nodetool repair --full stresscql2 typestest > nohup.txt &
{code}
Here are the results:
|| ||RACS||Non RACS||
|Stress Result|Op rate : 244 op/s [insert: 116 op/s, range1: 12 op/s, simple1:
116 op/s]
Partition rate : 243 pk/s [insert: 116 pk/s, range1: 10 pk/s, simple1: 116
pk/s]
Row rate : 274 row/s [insert: 116 row/s, range1: 41 row/s, simple1: 116 row/s]
Latency mean : 204.6 ms [insert: 2.5 ms, range1: 387.4 ms, simple1: 388.8 ms]
Latency median : 39.7 ms [insert: 2.0 ms, range1: 378.0 ms, simple1: 377.7 ms]
Latency 95th percentile : 706.2 ms [insert: 3.2 ms, range1: 802.2 ms, simple1:
805.3 ms] Latency 99th percentile : 941.6 ms [insert: 19.7 ms, range1: 1,022.9
ms, simple1: 1,022.4 ms] Latency 99.9th percentile : 1183.8 ms [insert: 65.5
ms, range1: 1,232.1 ms, simple1: 1,218.4 ms] Latency max : 7314.9 ms [insert:
550.0 ms, range1: 1,472.2 ms, simple1: 7,314.9 ms]
Total partitions : 874,058 [insert: 419,116, range1: 36,428, simple1: 418,514]
Total errors : 0 [insert: 0, range1: 0, simple1: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 01:00:00|Op rate : 221 op/s [insert: 105 op/s, range1:
11 op/s, simple1: 105 op/s]
Partition rate : 220 pk/s [insert: 105 pk/s, range1: 9 pk/s, simple1: 105 pk/s]
Row rate : 248 row/s [insert: 105 row/s, range1: 38 row/s, simple1: 105 row/s]
Latency mean : 226.2 ms [insert: 2.7 ms, range1: 428.8 ms, simple1: 429.1 ms]
Latency median : 150.3 ms [insert: 2.0 ms, range1: 385.4 ms, simple1: 383.8 ms]
Latency 95th percentile : 716.2 ms [insert: 3.0 ms, range1: 837.3 ms, simple1:
841.5 ms]
Latency 99th percentile : 1047.5 ms [insert: 14.8 ms, range1: 1,210.1 ms,
simple1: 1,230.0 ms]
Latency 99.9th percentile : 1830.8 ms [insert: 57.5 ms, range1: 2,029.0 ms,
simple1: 2,063.6 ms]
Latency max : 7457.5 ms [insert: 6,358.6 ms, range1: 7,159.7 ms, simple1:
7,457.5 ms]
Total partitions : 790,543 [insert: 378,618, range1: 33,908, simple1: 378,017]
Total errors : 0 [insert: 0, range1: 0, simple1: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 01:00:00|
> RangeAwareCompaction
>
>
> Key: CASSANDRA-10540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10540
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Labels: compaction, lcs, vnodes
> Fix For: 4.x
>
>
> Broken out from CASSANDRA-6696, we should split sstables based on ranges
> during compaction.
> Requirements;
> * dont create tiny sstables - keep them bunched together until a single vnode
> is big enough (configurable how big that is)
> * make it possible to run existing compaction strategies on the per-range
> sstables
> We should probably add a global compaction strategy parameter that states
> whether this should be enabled or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496077#comment-16496077
]
Lerh Chuan Low commented on CASSANDRA-10540:
Hi [~krummas],
Sorry for the delay, here are some initial benchmarks. I've only tried it with
LCS, this is the Stressspec YAML, a reasonably stressful test:
```
keyspace: stresscql2
keyspace_definition: |
CREATE KEYSPACE stresscql2 WITH replication = \{'class':
'NetworkTopologyStrategy', 'Waboku': 3, 'Bokusapp': 2};
table: typestest
table_definition: |
CREATE TABLE typestest (
name text,
choice boolean,
date timestamp,
address inet,
dbl double,
lval bigint,
ival int,
uid timeuuid,
value blob,
PRIMARY KEY((name,choice), date, address, dbl, lval, ival, uid)
) WITH compaction = \{ 'class':'LeveledCompactionStrategy',
'range_aware_compaction':'true', 'min_range_sstable_size_in_mb':'15' }
AND comment='A table of many types to test wide rows'
columnspec:
- name: name
size: uniform(1..1000)
population: uniform(1..500M) # the range of unique values to select for the
field (default is 100Billion)
- name: date
cluster: uniform(20..1000)
- name: lval
population: gaussian(1..1000)
cluster: uniform(1..4)
- name: value
size: uniform(100..500)
insert:
partitions: fixed(1) # number of unique partitions to update in a single
operation
batchtype: UNLOGGED # type of batch to use
select: uniform(1..10)/10 # uniform chance any single generated CQL row will
be visited in a partition;
queries:
simple1:
cql: select * from typestest where name = ? and choice = ? LIMIT 1
fields: samerow
range1:
cql: select name, choice, uid from typestest where name = ? and choice = ? and
date >= ? LIMIT 10
fields: multirow
simple2:
cql: select name, choice, uid from typestest where name = ? and choice = ?
LIMIT 1
fields: samerow # samerow or multirow (select arguments from the same row, or
randomly from all rows in the partition)
```
This is done over a multi DC cluster in EC2, 400GB SSD with 3 nodes in 1 and 2
nodes in the other. Stress replicates to both DCs.
For inserts:
```
nohup cassandra-stress user no-warmup profile=stressspec.yaml n=15000
cl=QUORUM ops\(insert=1\) -node file=nodelist.txt -rate threads=100 -log
file=insert.log > nohup.txt &
```
We have
|| ||RACS||NonRACS||
|Stress result|Op rate : 8,784 op/s [insert: 8,784 op/s]
Partition rate : 8,784 pk/s [insert: 8,784 pk/s]
Row rate : 8,784 row/s [insert: 8,784 row/s]
Latency mean : 5.4 ms [insert: 5.4 ms]
Latency median : 4.3 ms [insert: 4.3 ms]
Latency 95th percentile : 8.4 ms [insert: 8.4 ms]
Latency 99th percentile : 39.2 ms [insert: 39.2 ms]
Latency 99.9th percentile : 63.3 ms [insert: 63.3 ms]
Latency max : 1506.8 ms [insert: 1,506.8 ms]
Total partitions : 150,000,000 [insert: 150,000,000]
Total errors : 0 [insert: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 04:44:35|Op rate : 8,730 op/s [insert: 8,730 op/s]
Partition rate : 8,730 pk/s [insert: 8,730 pk/s]
Row rate : 8,730 row/s [insert: 8,730 row/s]
Latency mean : 5.4 ms [insert: 5.4 ms]
Latency median : 4.3 ms [insert: 4.3 ms]
Latency 95th percentile : 8.5 ms [insert: 8.5 ms]
Latency 99th percentile : 39.4 ms [insert: 39.4 ms]
Latency 99.9th percentile : 66.1 ms [insert: 66.1 ms]
Latency max : 944.8 ms [insert: 944.8 ms]
Total partitions : 150,000,000 [insert: 150,000,000]
Total errors : 0 [insert: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 04:46:22|
|SSTable count|1339
1259
1342
1285
1333|743
750
747
737
741|
For mixed workloads, which is done after the insert so reads are not just read
off the OS page cache:
|| ||RACS||Non RACS||
|Stress result |Op rate : 415 op/s [insert: 197 op/s, range1: 20 op/s, simple1:
198 op/s]
Partition rate : 407 pk/s [insert: 197 pk/s, range1: 12 pk/s, simple1: 198 pk/s]
Row rate : 412 row/s [insert: 197 row/s, range1: 17 row/s, simple1: 198 row/s]
Latency mean : 120.4 ms [insert: 2.3 ms, range1: 227.0 ms, simple1: 227.3 ms]
Latency median : 38.0 ms [insert: 2.0 ms, range1: 207.0 ms, simple1: 207.4 ms]
Latency 95th percentile : 454.6 ms [insert: 3.1 ms, range1: 541.1 ms, simple1:
543.2 ms]
Latency 99th percentile : 673.2 ms [insert: 5.1 ms, range1: 739.2 ms, simple1:
741.3 ms]
Latency 99.9th percentile : 918.0 ms [insert: 43.4 ms, range1: 985.1 ms,
simple1: 975.2 ms]
Latency max : 1584.4 ms [insert: 766.0 ms, range1: 1,426.1 ms, simple1: 1,584.4
ms]
Total partitions : 2,930,512 [insert: 1,419,222, range1: 86,021, simple1:
1,425,269]
Total errors : 0 [insert: 0, range1: 0, simple1: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 02:00:01|Op rate : 382 op/s [insert: 182
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445374#comment-16445374 ] Lerh Chuan Low commented on CASSANDRA-10540: Hey Marcus, thanks for getting back so quickly :) The big motivation for us is repairs...because when vnodes are turned on, every SSTable has many vnodes in them...so when a (incremental) repair happens, the range it is interested in gets anticompacted out. After that the next range gets anticompacted out and so on. RACS solves that big pain. Besides making the SSTables per read much nicer, it's like a LCS on steroids. I think there are other benefits of maintaining each SSTable to one token range...but I can't quite remember any more off the top of my head. So I am hoping it doesn't come to grouping the vnodesunless it's a last resort. Currently it looks like you create a RACS for each of the repaired/unrepaired/pending repairs set, and each RACS keeps track of the compaction strategies it is in charge of (which are all of the same class). The CS instances are lazily initiated (so that's a win right there) until needed. It seems to be that the reason why we want so many CS instances is so that each of them can keep track of their own SSTables (which all belong to that single token range). How about if RACS doesn't instantiate the individual CS instances? It keeps track of all the SSTables in the CF like other CS instances - just that the logic on which SSTables to involve in a compaction is done in RACS. So we can make RACS check L0 and if there are none, L1 would involve grouping the SSTables by range and then calling the next background task for the underlying/wrapped CS instances and submitting them. In this way, the downside is calculating the grouping each time we ask for the next background task. We could also store it in memory in the form of a manifest like in LCS? So an array with the SSTables in each of them - beats having 256 instances but we're still going to have a 256 sized array in memory, I guess. It just seems so starkingly similar to a LCS restricted to just L0 and L1. A final thought: Is the memory footprint actually significant enough for us to want to not bite the bullet and further group the vnodes because the gains from having each SSTable as a single range is a lot, simple is a feature and RACS is customizable? Please excuse my ignorance if none of those suggestions made sense/worked, still not very confident with the code base.. > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: compaction, lcs, vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445273#comment-16445273 ] Marcus Eriksson commented on CASSANDRA-10540: - [~Lerh Low] yeah its still on my plate, just not very happy with it at the moment, mostly because of the number of compaction strategy instances it runs with vnodes (#tokens * rf), might need to group vnodes to get it down to a reasonable number or something. > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: compaction, lcs, vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445061#comment-16445061 ] Lerh Chuan Low commented on CASSANDRA-10540: [~krummas] any chance you're still working on this? (It's been a while). I recently updated your branch against trunk and fixed a few merge conflicts: [https://github.com/juiceblender/cassandra/tree/10540] I'm also planning to write more unit tests to help get this in...which is in the works at the moment. Feel free to let me know if there's any other assistance you could use/need in getting this committed? > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: compaction, lcs, vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996150#comment-15996150 ] Cameron Zemek commented on CASSANDRA-10540: --- I have found one issue with the code. It states "To avoid getting very many tiny sstables in the per-range strategies, we keep them outside the strategy until the estimated size of a range-sstable is larger than 'min_range_sstable_size_in_mb'. (estimation usually gets within a few % of the actual value)." However RangeAwareCompactionStrategy::addSSTable does not check that the sstable meets the minimum size. This is potentially an issue with repairs that stream sections of sstables or if memtable only includes a single token range on flush. On a different note, I notice the performance testing so far as looked at write amplification. I suspect RangeAwareCompaction could also improve read performance due to partitions more likely to exist in less sstables (ie.. reduces the sstables per read). It would be interesting to see SSTable leaders for partitions with STCS vs RangeAwareCompaction + STCS. Can get list of sstable leaders with ic-pstats tool have open sourced here, https://github.com/instaclustr/cassandra-sstable-tools > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Labels: compaction, lcs, vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526390#comment-15526390 ] Carl Yeksigian commented on CASSANDRA-10540: I'm +1 on the code here, I'm just waiting on some more testing from [~philipthompson]. Thanks for the ping [~jjirsa]. > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Labels: compaction, lcs, vnodes > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524875#comment-15524875 ] Jeff Jirsa commented on CASSANDRA-10540: [~carlyeks] - this still on your plate to review? > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Labels: compaction, lcs, vnodes > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424092#comment-15424092
]
Marcus Eriksson commented on CASSANDRA-10540:
-
These "benchmarks" have been run using cassandra-stress with
[this|https://paste.fscking.us/display/jKc0X89MLFzHE9jhRqQ5xfvRHeU] yaml (only
modified per run with the different compaction configurations).
cassandra-stress generates 40GB of data and then it compacts those sstables
using 8 threads. All tests were run with 256 tokens on my machine (2 ssds, 32GB
ram):
{code}
./tools/bin/compaction-stress write -d /var/lib/cassandra -d
/home/marcuse/cassandra -g 40 -p blogpost-range.yaml -t 4 -v 256
./tools/bin/compaction-stress compact -d /var/lib/cassandra -d
/home/marcuse/cassandra -p blogpost-range.yaml -t 8 -v 256
{code}
First a base line - it takes about 7 minutes to compact 40GB of data with STCS,
and we get a write amplification (compaction bytes written / size before) of
about 1.46.
* 40GB + STCS
||size before||size after||compaction bytes written||time||number of
compactions||
|42986704571|31305948786|62268272752|7:44|26|
|43017694284|31717603488|62800073327|7:04|26|
|42863193047|31244649872|64673778727|6:44|26|
|4296276|31842455113|62985984309|6:14|26|
|43107421526|32526047125|61657717328|6:04|26|
With range aware compaction and a small min_range_sstable_size_in_mb we compact
slower, about 2x the time, but the end result is smaller with a tiny bit smaller
write amplification (1.44). The reason for the longer time is that we need to
do a lot more tiny compaction for each vnode. The reason for the smaller size
after the compactions is that we are much more likely to compact overlapping
sstables together as we compact within each vnode.
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 1
||size before||size after||compaction bytes written||time||number of
compactions||
|42944940703|25352795435|61734295478|13:18|286|
|42896304174|25830662102|62049066195|15:45|287|
|43091495756|24811367911|61448601743|12:25|287|
|42961529234|26275106863|63118850488|13:17|284|
|42902111497|25749453764|61529524300|13:54|280|
As we increase the min_range_sstable_size_in_mb the time spent is reduced, the
size after the compaction is increased and the number of compactions is reduced
since we don't promote sstables to the per-vnode-strategies as quickly. With
large enough min_range_sstable_size_in_mb the behaviour will be the same as
STCS (+a small overhead for estimating the size of the next vnode range during
compaction)
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 5
||size before||size after||compaction bytes written||time||number of
compactions||
|4307106|27586259306|62855258024|10:35|172|
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 10
||size before||size after||compaction bytes written||time||number of
compactions||
|42998501805|28281735688|65469323764|9:45|109|
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 20
||size before||size after||compaction bytes written||time||number of
compactions||
|42801860659|28934194973|66554340039|10:05|48|
* 40GB + STCS + range_aware + min_range_sstable_size_in_mb: 50
||size before||size after||compaction bytes written||time||number of
compactions||
|42881416448|30352758950|61223610818|7:25|27|
With LCS and a small sstable_size_in_mb we get a huge difference with range
aware due to the amount of compactions we need to do to get the leveling
without range aware compaction. With range aware, we get fewer levels in each
vnode-range and that is much quicker to compact. Write amplification is about
2.0 with range aware and 3.4 without.
* 40GB + LCS + sstable_size_in_mb: 10 + range_aware +
min_range_sstable_size_in_mb: 10
||size before||size after||compaction bytes written||time||number of
compactions||
|43170254812|26511935628|87637370434|19:55|903|
|43015904097|26100197485|83125478305|14:45|854|
|4316684|25651102691|87520409116|19:55|920|
* 40GB + LCS + sstable_size_in_mb: 10
||size before||size after||compaction bytes written||time||number of
compactions||
|43099495889|23876144309|139000531662|28:25|3751|
|4281178|24620085107|147722973544|30:35|3909|
|42879141849|24479485292|146194679395|30:46|3882|
If we bump the lcs sstable_size_in_mb to the default we get more similar
results. Write amplification is smaller with range aware compaction but size
after is also bigger. The reason for the bigger size after compaction has
settled is that we run with a bigger min_range_sstable_size_in_mb which means
more data will stay out of the per-range compaction strategies and this means
it is only size tiered. This probably also explains the reduced write
amplification - 2.0 with range aware and 2.3 without.
* 40GB + LCS + sstable_size_in_mb: 160 + range_aware +
min_range_sstable_size_in_mb: 20
||size before||size after||compaction bytes
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296316#comment-15296316 ] Marcus Eriksson commented on CASSANDRA-10540: - Planning to use CASSANDRA-11844 to write a bunch of stress tests for this, they should be finished before we consider committing this > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268679#comment-15268679 ] Marcus Eriksson commented on CASSANDRA-10540: - Have not measured, will try to do that this week > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268664#comment-15268664 ] Jonathan Ellis commented on CASSANDRA-10540: While waiting for review, what kind of write amplification improvement (i.e. total bytes compacted given constant bytes loaded) are you seeing with STCS? > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137020#comment-15137020 ] Marcus Eriksson commented on CASSANDRA-10540: - [rebased|https://github.com/krummas/cassandra/commits/marcuse/10540] and test runs looks clean: http://cassci.datastax.com/job/krummas-marcuse-10540-dtest/ http://cassci.datastax.com/job/krummas-marcuse-10540-testall/ > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098155#comment-15098155 ] Marcus Eriksson commented on CASSANDRA-10540: - pushed a [new branch|https://github.com/krummas/cassandra/commits/marcuse/10540] dtest: http://cassci.datastax.com/job/krummas-marcuse-10540-dtest/ utest: http://cassci.datastax.com/job/krummas-marcuse-10540-testall/ I see that there are a few tests that seem to break, I'll look into those > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096503#comment-15096503 ] Carl Yeksigian commented on CASSANDRA-10540: [~krummas]: can you rebase this and rerun the tests now that CASSANDRA-6696 is in? > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037537#comment-15037537 ] Marcus Eriksson commented on CASSANDRA-10540: - pushed a new commit with a fix and some timing output to see how long it takes getting new compaction tasks > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.2 > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968973#comment-14968973
]
Marcus Eriksson commented on CASSANDRA-10540:
-
Pushed a new branch
[here|https://github.com/krummas/cassandra/commits/marcuse/rangeawarecompaction]
for early feedback, still needs cleanup and tests
Enable like this:
{code}ALTER TABLE x.y WITH compaction={'class':'LeveledCompactionStrategy',
'range_aware_compaction':'true', 'min_range_sstable_size_in_mb':'15'}{code}
* Run a compaction strategy instance per owned range (with num_tokens=256 and
rf=3, we will have 768 * 2 instances (repaired/unrepaired data)).
* To avoid getting very many tiny sstables in the per-range strategies, we keep
them outside the strategy until the estimated size of a range-sstable is larger
than {{'min_range_sstable_size_in_mb'}}.
([estimation|https://github.com/krummas/cassandra/blob/09c58eb4689230d471ef4319733fb0e85399bd3a/src/java/org/apache/cassandra/db/compaction/writers/RangeAwareCompactionWriter.java#L115]
usually gets within a few % of the actual value).
* We do STCS among the many-range-sstables (called "L0" which might not be
optimal due to LCS)
* We currently prioritize compaction in L0 to get sstables out of there as
quickly as possible
* If an sstable fits within a range, it is added to that corresponding
range-compaction strategy - this should avoid getting a lot of L0 sstables
after streaming for example
* Adds a {{describecompactionstrategy}} nodetool command which displays
information about the configured compaction strategy (like sstables per range
etc). Example with only unrepaired data and 2 data directories - we first split
the owned ranges over those 2 directories, and then we split on a per range
basis, so the first RangeAwareCompactionStrategy is responsible for half the
data and the second one is responsible for the rest: {code} $ bin/nodetool
describecompactionstrategy keyspace1 standard1
-- keyspace1.standard1
--
Strategy=class org.apache.cassandra.db.compaction.RangeAwareCompactionStrategy,
for 167 unrepaired sstables, boundary tokens=min(-9223372036854775808) ->
max(-4095785201827646), location=/home/marcuse/c/d1
Inner strategy: class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy (257 instances,
162 total sstables)
sstable counts:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24 25 26 27 28 29
--
0.. 29 | 1 3 0 0 2 3 0 3 3 0 3 0 2 1 0 1 0 1 0 3 3 4 1
0 3 1 0 0 0 0
30.. 59 | 0 0 0 3 0 2 2 0 3 0 3 3 0 1 3 3 3 0 2 0 1 2 0
0 0 1 0 3 0 0
60.. 89 | 1 0 0 1 1 1 1 0 1 0 2 3 1 0 3 1 2 3 2 0 0 3 2
1 1 0 0 2 3 1
90..119 | 0 1 2 0 0 3 0 3 3 1 0 0 3 0 2 0 2 0 2 1 3 0 2
1 1 3 1 0 3 0
120..149 | 2 0 3 1 3 0 0 3 3 1 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
150..179 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
180..209 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
210..239 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
240..257 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Strategy=class org.apache.cassandra.db.compaction.RangeAwareCompactionStrategy,
for 221 unrepaired sstables, boundary tokens=max(-4095785201827646) ->
max(9223372036854775807), location=/var/lib/c1
Inner strategy: class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy (257 instances,
215 total sstables)
sstable counts:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24 25 26 27 28 29
--
0.. 29 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
30.. 59 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
60.. 89 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
90..119 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
120..149 | 0 0 0 0 0 0 0 0 0 0 1 6 0 0 3 0 3 0 3 3 3 3 1
0 1 0 2 0 3 2
150..179 | 3 3 3 0 0 3 3 0 3 2 3 1 3 3 3 3 0 0 0 3 0 1 1
0 6 3 3 0 3 3
180..209 | 0 1 1 3 1 3 1 3 3 2 3 3 0 3 0 3 1 0 0 1 2 3 0
0 1 1 0 0 3 3
210..239 | 3 3 3 2 0 6 1 3 0 0 3 3 3 1 3 4 3 3 3 0 3 0 3
1 2 2 0 2 0 0
240..257 | 1 0 3 1 0 3
