[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-7019: --- Resolution: Fixed Fix Version/s: (was: 3.x) 3.10 Status: Resolved (was: Ready to Commit) and committed! > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout, lcs > Fix For: 3.10 > > Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, > control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-7019: --- Status: Ready to Commit (was: Patch Available) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout, lcs > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, > control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Branimir Lambov updated CASSANDRA-7019: --- Status: Patch Available (was: Awaiting Feedback) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout, lcs > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, > control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Deng updated CASSANDRA-7019: Labels: compaction fallout lcs (was: compaction fallout) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout, lcs > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, > control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-7019: -- Status: Awaiting Feedback (was: Open) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, > control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-7019: -- Status: Open (was: Patch Available) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, > control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-7019: --- Attachment: cell.tar.gz control.tar.gz none.tar.gz row.tar.gz I have attached tarballs with the compaction.log, debug.log, and stress log files for each test. I am not able to produce a functioning stress graph, probably due to their size > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, > control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[
https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-7019:
---
Attachment: temp-plot.html
So I ran some ~day long tests, using the three new options from this ticket,
{{none}}, {{row}}, and {{cell}}, and the parent commit on trunk, which I
labeled {{control}}. I then grabbed all of the compaction throughput results
from the debug.log files, and attached a very basic graph here, as
temp-plot.html.
The 20% delete workload had a mixture of partition, row, and cell deletions. We
saw improved compaction throughput with 7019 when compared to trunk, and then
expected but acceptable drops in compaction performance with the more
aggressive tombstone removal options.
I will begin dumping all the raw data I have, as well as the stress graphs so
you can see how read/write perf is affected as well. I should note, at this
time I have found nothing that would prevent me from being +1 on this ticket.
> Improve tombstone compactions
> -
>
> Key: CASSANDRA-7019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7019
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
> Labels: compaction, fallout
> Fix For: 3.x
>
> Attachments: 7019-2-system.log, 7019-debug.log, temp-plot.html
>
>
> When there are no other compactions to do, we trigger a single-sstable
> compaction if there is more than X% droppable tombstones in the sstable.
> In this ticket we should try to include overlapping sstables in those
> compactions to be able to actually drop the tombstones. Might only be doable
> with LCS (with STCS we would probably end up including all sstables)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-7019: -- Labels: compaction fallout (was: compaction) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction, fallout > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-7019: --- Attachment: 7019-2-system.log > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction > Fix For: 3.x > > Attachments: 7019-2-system.log, 7019-debug.log > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-7019: --- Attachment: 7019-debug.log > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction > Fix For: 3.x > > Attachments: 7019-debug.log > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Branimir Lambov updated CASSANDRA-7019: --- Status: Patch Available (was: In Progress) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction > Fix For: 3.x > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Branimir Lambov updated CASSANDRA-7019: --- Status: In Progress (was: Patch Available) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction > Fix For: 3.x > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Branimir Lambov updated CASSANDRA-7019: --- Component/s: Compaction > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction > Fix For: 3.x > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-7019: --- Assignee: Branimir Lambov (was: Vishy Kasar) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction > Fix For: 3.x > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-7019: -- Assignee: Branimir Lambov (was: Marcus Eriksson) Assigning to Branimir. Marcus, can you summarize what the scope is and where to start? Improve tombstone compactions - Key: CASSANDRA-7019 URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 Project: Cassandra Issue Type: Improvement Reporter: Marcus Eriksson Assignee: Branimir Lambov Labels: compaction Fix For: 3.0 When there are no other compactions to do, we trigger a single-sstable compaction if there is more than X% droppable tombstones in the sstable. In this ticket we should try to include overlapping sstables in those compactions to be able to actually drop the tombstones. Might only be doable with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-7019: --- Summary: Improve tombstone compactions (was: Major tombstone compaction) Improve tombstone compactions - Key: CASSANDRA-7019 URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 Project: Cassandra Issue Type: Improvement Reporter: Marcus Eriksson Assignee: Marcus Eriksson Labels: compaction Fix For: 3.0 It should be possible to do a major tombstone compaction by including all sstables, but writing them out 1:1, meaning that if you have 10 sstables before, you will have 10 sstables after the compaction with the same data, minus all the expired tombstones. We could do this in two ways: # a nodetool command that includes _all_ sstables # once we detect that an sstable has more than x% (20%?) expired tombstones, we start one of these compactions, and include all overlapping sstables that contain older data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-7019: --- Description: When there are no other compactions to do, we trigger a single-sstable compaction if there is more than X% droppable tombstones in the sstable. In this ticket we should try to include overlapping sstables in those compactions to be able to actually drop the tombstones. Might only be doable with LCS (with STCS we would probably end up including all sstables) was: It should be possible to do a major tombstone compaction by including all sstables, but writing them out 1:1, meaning that if you have 10 sstables before, you will have 10 sstables after the compaction with the same data, minus all the expired tombstones. We could do this in two ways: # a nodetool command that includes _all_ sstables # once we detect that an sstable has more than x% (20%?) expired tombstones, we start one of these compactions, and include all overlapping sstables that contain older data. Improve tombstone compactions - Key: CASSANDRA-7019 URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 Project: Cassandra Issue Type: Improvement Reporter: Marcus Eriksson Assignee: Marcus Eriksson Labels: compaction Fix For: 3.0 When there are no other compactions to do, we trigger a single-sstable compaction if there is more than X% droppable tombstones in the sstable. In this ticket we should try to include overlapping sstables in those compactions to be able to actually drop the tombstones. Might only be doable with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
