[
https://issues.apache.org/jira/browse/CASSANDRA-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Lynch updated CASSANDRA-17381:
-----------------------------------
Description:
The existing compaction strategies have a number of drawbacks that make all
three unsuitable as a general use compaction strategy, for example STCS creates
giant files that are hard to back up, mess with the page cache, and led to many
of the early re-open bugs. LCS improved dramatically on this but also has
various issues e.g. lack of performant full compaction or due to the strict
leveling with e.g. bulk loading when writes exceed the rate we can do the L0 -
L1 promotion.
In this
[talk|https://github.com/ngcc/ngcc2019/blob/master/NextGenerationCassandraCompactionGoingBeyondLCS.pdf]
I introduced a novel compaction strategy that aims to expose a single tunable
that the user can control for the read amplification. Raise the
max_read_per_read and you tradeoff read/space performance for write
performance. Since then a proof of concept [patch
|https://github.com/jolynch/cassandra/tree/jolynch_bounded_read_final]has been
published along with some rudimentary [documentation
|https://gist.github.com/jolynch/9118465f32ad5298b4e39d03ccd4370e] but this is
still not tracked in Jira.
The remaining work here is
1. Validate the algorithm is correct via test suites and performance testing
stress testing and benchmarking with OSS tools (e.g. cassandra-stress,
[tlp-stress|https://github.com/thelastpickle/tlp-stress], or
[ndbench|https://github.com/Netflix/ndbench]). When issues are found (there
likely will be issues as the patch is a PoC), devise how to adjust the
algorithm and implementation appropriately. Key metric of success is we can run
Cassandra stably for more than 24 hours while applying sustained load, and
compaction can keep up.
2. Do more in depth experiments measuring performance across a wide range of
workloads (e.g. write heavy, read heavy, balanced, time series, register
update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and
TWCS (time window). Key metrics of success are establishing that as we tune
max_read_per_read we should get more predictable read latency under low system
load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should
match LCS performance under low load while doing better at high load.
Once this is validated a Cassandra blog post reporting on the findings
(positive or negative) may be advisable.
was:
The existing compaction strategies have a number of drawbacks that make all
three unsuitable as a general use compaction strategy, for example STCS creates
giant files that are hard to back up, mess with the page cache, and led to many
of the early re-open bugs. LCS improved dramatically on this but also has
various issues e.g. lack of performant full compaction or due to the strict
leveling with e.g. bulk loading when writes exceed the rate we can do the L0 -
L1 promotion.
In this
[talk|https://github.com/ngcc/ngcc2019/blob/master/NextGenerationCassandraCompactionGoingBeyondLCS.pdf]
I introduced a novel compaction strategy that aims to expose a single tunable
that the user can control for the read amplification. Raise the
max_read_per_read and you tradeoff read/space performance for write
performance. Since then a proof of concept [patch
|https://github.com/jolynch/cassandra/tree/jolynch_bounded_read_final]has been
published along with some rudimentary [documentation
|https://gist.github.com/jolynch/9118465f32ad5298b4e39d03ccd4370e] but this is
still not tracked in Jira.
The remaining work here is
1. Validate the algorithm is correct via test suites and performance testing
stress testing and benchmarking with OSS tools (e.g. cassandra-stress,
[tlp-stress|https://github.com/thelastpickle/tlp-stress], or
[ndbench|https://github.com/Netflix/ndbench]). When issues are found (there
likely will be issues as the patch is a PoC), devise how to adjust the
algorithm and implementation appropriately. Key metric of success is we can run
Cassandra stably for more than 24 hours while applying sustained load, and
compaction can keep up.
2. Do more in depth experiments measuring performance across a wide range of
workloads (e.g. write heavy, read heavy, balanced, time series, register
update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and
TWCS (time window). Key metrics of success are establishing that as we tune
max_read_per_read we should get more predictable read latency under low system
load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should
match LCS performance under low load while doing better at high load.
> Produce and verify BoundedReadCompactionStrategy as a unified general purpose
> compaction algorithm
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-17381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17381
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Compaction
> Reporter: Joey Lynch
> Assignee: Joey Lynch
> Priority: Normal
>
> The existing compaction strategies have a number of drawbacks that make all
> three unsuitable as a general use compaction strategy, for example STCS
> creates giant files that are hard to back up, mess with the page cache, and
> led to many of the early re-open bugs. LCS improved dramatically on this but
> also has various issues e.g. lack of performant full compaction or due to the
> strict leveling with e.g. bulk loading when writes exceed the rate we can do
> the L0 - L1 promotion.
> In this
> [talk|https://github.com/ngcc/ngcc2019/blob/master/NextGenerationCassandraCompactionGoingBeyondLCS.pdf]
> I introduced a novel compaction strategy that aims to expose a single
> tunable that the user can control for the read amplification. Raise the
> max_read_per_read and you tradeoff read/space performance for write
> performance. Since then a proof of concept [patch
> |https://github.com/jolynch/cassandra/tree/jolynch_bounded_read_final]has
> been published along with some rudimentary [documentation
> |https://gist.github.com/jolynch/9118465f32ad5298b4e39d03ccd4370e] but this
> is still not tracked in Jira.
> The remaining work here is
> 1. Validate the algorithm is correct via test suites and performance testing
> stress testing and benchmarking with OSS tools (e.g. cassandra-stress,
> [tlp-stress|https://github.com/thelastpickle/tlp-stress], or
> [ndbench|https://github.com/Netflix/ndbench]). When issues are found (there
> likely will be issues as the patch is a PoC), devise how to adjust the
> algorithm and implementation appropriately. Key metric of success is we can
> run Cassandra stably for more than 24 hours while applying sustained load,
> and compaction can keep up.
> 2. Do more in depth experiments measuring performance across a wide range of
> workloads (e.g. write heavy, read heavy, balanced, time series, register
> update, etc ...) and in comparison with LCS (leveled), STCS (size tiered),
> and TWCS (time window). Key metrics of success are establishing that as we
> tune max_read_per_read we should get more predictable read latency under low
> system load (ρ < 30%) while not degrading at high system load (ρ > 70%), and
> we should match LCS performance under low load while doing better at high
> load.
> Once this is validated a Cassandra blog post reporting on the findings
> (positive or negative) may be advisable.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
