The reason is described here: https://issues.apache.org/jira/browse/CASSANDRA-5371?focusedCommentId=13621679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621679
/Marcus On Wed, Nov 23, 2016 at 6:31 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > What you’re describing seems very close to what’s discussed in > https://issues.apache.org/jira/browse/CASSANDRA-10979 - worth reading > that ticket a bit. > > > > There does seem to be a check for STCS in L0 before it tries higher > levels: > > https://github.com/apache/cassandra/blob/cassandra-2.2/ > src/java/org/apache/cassandra/db/compaction/LeveledManifest.java#L324-L326 > > > > Why it’s doing that within the for loop (https://github.com/apache/ > cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/ > db/compaction/LeveledManifest.java#L310 ) is unexpected to me, though – > Carl / Marcus, any insight into why it’s within the loop instead of before > it? > > > > > > *From: *Marcus Olsson <marcus.ols...@ericsson.com> > *Organization: *Ericsson AB > *Reply-To: *"dev@cassandra.apache.org" <dev@cassandra.apache.org> > *Date: *Wednesday, November 23, 2016 at 7:52 AM > *To: *"dev@cassandra.apache.org" <dev@cassandra.apache.org> > *Subject: *STCS in L0 behaviour > > > > Hi everyone, > > > TL;DR > Should LCS be changed to always prefer an STCS compaction in L0 if it's > falling behind? Assuming that STCS in L0 is enabled. > Currently LCS seems to check if there is a possible L0->L1 compaction > before checking if it's falling behind, which in our case used between > 15-30% of the compaction thread CPU. > TL;DR > > So first some background: > We have a Apache Cassandra 2.2 cluster running with a high load. In that > cluster there is a table with a moderate amount of writes per second that > is using LeveledCompactionStrategy. The test was to run repair on that > table while we monitored the cluster through JMC and with Flight Recordings > enabled. This resulted in a large amount of sstables for that table, which > I assume others have experienced as well. In this case I think it was > between 15-20k. > > From the Flight Recording one thing we saw was that 15-30% of the CPU time > in each of the compaction threads was spent on "getNextBackgroundTask()" > which retrieves the next compaction job. With some further investigation > this seems to mostly be when it's checking for overlap in L0 sstables > before performing an L0->L1 compaction. There is a JIRA which seems to be > related to this https://issues.apache.org/jira/browse/CASSANDRA-11571 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D11571&d=DgMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=M8Whp4kcFqczCZGE9J1ZG0d1LKXlruo4bEJLrXUM9n0&s=KaBB4zTcZ8lcdDd0rSCaJdO55spqy1lq5xzEtCyK5As&e=> > which we backported to 2.2 and tested. In our testing it seemed to improve > the situation but it was still using noticeable CPU. > > My interpretation of the current logic of LCS is (if STCS in L0 is > enabled): > 1. Check each level (L1+) > - If a L1+ compaction is needed check if L0 is behind and do STCS if > that's the case, otherwise do the L1+ compaction. > 2. Check L0 -> L1 compactions and if none is needed/possible check for > STCS in L0. > > My proposal is to change this behavior to always check if L0 is far behind > first and do a STCS compaction in that case. This would avoid the overlap > check for L0 -> L1 compactions when L0 is behind and I think it makes sense > since we already prefer STCS to L1+ compactions. This would not solve the > repair situation, but it would lower some of the impact that repair has on > LCS. > > For what version this could get in I think trunk would be enough since > compaction is pluggable. > > -- > > > > [image: ricsson] > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ericsson.com_&d=DgMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=M8Whp4kcFqczCZGE9J1ZG0d1LKXlruo4bEJLrXUM9n0&s=6tv3nY7zQR21E67RGe10u35WbYuJQ4gnMD6cao8il-E&e=> > > *MARCUS OLSSON * > Software Developer > > *Ericsson* > Sweden > marcus.ols...@ericsson.com > www.ericsson.com > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ericsson.com&d=DgMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=M8Whp4kcFqczCZGE9J1ZG0d1LKXlruo4bEJLrXUM9n0&s=ozu2nd2H_Rto9M1Sme6640Z2ntrDYfkPOygOsu9uZUU&e=> >
