[
https://issues.apache.org/jira/browse/CASSANDRA-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-16552:
----------------------------------------
Test and Documentation Plan: new test, cci run
Status: Patch Available (was: Open)
patch: https://github.com/krummas/cassandra/commits/marcuse/16552
tests:
https://app.circleci.com/pipelines/github/krummas/cassandra?branch=marcuse%2F16552
Patch tracks the original sstable instances and uses them when removing
sstables from LeveledGenerations. Also makes sure we replace the sstable
instance if we get an added sstable which we already know about.
> Anticompaction appears to race with Compaction, preventing forward compaction
> progress after an incremental repair
> ------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16552
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16552
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Compaction
> Reporter: Joey Lynch
> Assignee: Marcus Eriksson
> Priority: High
> Fix For: 4.0, 4.0-beta
>
> Attachments: CompactionStuck.png, CurrentView.png,
> NeighborsFallingBehind.png, anticompaction_before_issue.txt, debug.log.gz,
> system.log.gz
>
>
> While testing 4.0-rc1 on a 12 i3en.2xlarge x 2 region (AWS us-east-1 and
> eu-west-1) cluster I attempted to run {{nodetool repair}} while the cluster
> was taking moderate read/write load.
> The first time it worked as expected, but when I ran an incremental run the
> second time multiple nodes got stuck trying to compact the unrepaired
> sstables. They are now spinning with:
> {noformat}
> $ nt compactionstats
> pending tasks: 827
> - acceptance_josephl.acceptance_josephl_cass4: 827
> $ nt tpstats
> Pool Name Active Pending Completed Blocked All time blocked
> RequestResponseStage 0 0 422359133 0 0
> MutationStage 0 0 164540628 0 0
> ReadStage 0 0 198857844 0 0
> CompactionExecutor 0 0 60782 0 0
> $ tail system.log
> DEBUG [CompactionExecutor:684] 2021-03-31 15:13:59,902
> LeveledManifest.java:292 - L0 is too far behind, performing size-tiering
> there first
> DEBUG [CompactionExecutor:684] 2021-03-31 15:13:59,908
> LeveledManifest.java:292 - L0 is too far behind, performing size-tiering
> there first
> WARN [CompactionExecutor:684] 2021-03-31 15:13:59,912
> LeveledCompactionStrategy.java:154 - Could not acquire references for
> compacting SSTables
> [BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11eba
> fb40b81cbd6fb3d/na-4826-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4872-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acc
> eptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4849-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4874-big-Data.db'),
> BigTableReader(path='/mnt/dat
> a/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4841-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4897-big-D
> ata.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4924-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e79
> 0917c11ebafb40b81cbd6fb3d/na-4837-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4926-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_j
> osephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4729-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4723-big-Data.db'),
> BigTableReader(path
> ='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4875-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-
> 4922-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4920-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cas
> s4-6144e790917c11ebafb40b81cbd6fb3d/na-4869-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4823-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/ac
> ceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4846-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4873-big-Data.db'),
> BigTableR
> eader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4840-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cb
> d6fb3d/na-4833-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4829-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_j
> osephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4726-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4923-big-Data.db'),
> BigTableReader(path='/mnt/data/cassand
> ra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4925-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4905-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4876-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11eb
> afb40b81cbd6fb3d/na-4901-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4732-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/ac
> ceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4909-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4915-big-Data.db'),
> BigTableReader(path='/mnt/da
> ta/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4921-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4860-big-
> Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4693-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e7
> 90917c11ebafb40b81cbd6fb3d/na-4694-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4692-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_
> josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4691-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4696-big-Data.db'),
> BigTableReader(pat
> h='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4697-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na
> -4700-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4698-big-Data.db'),
>
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_ca
> ss4-6144e790917c11ebafb40b81cbd6fb3d/na-4688-big-Data.db'),
> BigTableReader(path='/mnt/data/cassandra/data/acceptance_josephl/acceptance_josephl_cass4-6144e790917c11ebafb40b81cbd6fb3d/na-4689-big-Data.db')]
> which is not a problem per se,unless it happens
> frequently, in which case it must be reported. Will retry later.
> {noformat}
> I've attached some starting breadcrumbs. I believe the issue is a potential
> race in [marking sstables for
> compaction|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java#L152-L161]
> getting null back from
> [tryModify|https://github.com/apache/cassandra/blob/d42087a63309178b96909c012dd0073fe0b6ea11/src/java/org/apache/cassandra/db/lifecycle/Tracker.java#L100]
> which I think can only happen under a [small number of
> circumstances|https://github.com/apache/cassandra/blob/d42087a63309178b96909c012dd0073fe0b6ea11/src/java/org/apache/cassandra/db/lifecycle/View.java#L269].
> From the initial investigation it does appear that only the unrepaired
> products get into this state.
> I have a heap dump containing the View state but it contains potentially
> sensitive infrastructure details so if you're debugging just message me in
> slack and I can send it to you directly.
> The following mitigation appears to unstick the nodes via a forced full
> compaction:
> {noformat}
> nodetool stop COMPACTION; nodetool compact <ks> <table>
> {noformat}
> I'm not confident in this mitigation though.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
