[
https://issues.apache.org/jira/browse/CASSANDRA-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247293#comment-14247293
]
Rick Branson commented on CASSANDRA-8463:
-----------------------------------------
I applied the logging patch on one node and quickly noticed a suspect
compaction. This contains 130 sstables from L3 and only 1 from L2.
2014-12-15_21:50:10.24752 INFO 21:50:10 Compacting
(5750bed0-84a4-11e4-8177-4571ae47521c)
[SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35274-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35305-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35330-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35409-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35512-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35286-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35323-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35366-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35407-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35558-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35206-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35570-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35247-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35361-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35335-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35261-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35588-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35368-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35545-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35311-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35231-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35416-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35567-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35265-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35500-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35516-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35304-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35358-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35256-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35550-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35393-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35533-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35494-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35538-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35239-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35544-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35401-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35502-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35495-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35553-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35427-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35503-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35559-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35515-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35536-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35340-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35546-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35343-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35519-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35517-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35214-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35489-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35367-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35376-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35350-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35415-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35218-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35294-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35253-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35591-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35522-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35510-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35508-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35339-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35524-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35359-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35384-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35487-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35579-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35551-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35400-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35342-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35488-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35392-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35486-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35291-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35314-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35391-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35526-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35573-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35572-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35418-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35245-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35317-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35531-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35585-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35377-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35493-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35351-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35270-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35560-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35543-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35523-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35226-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35352-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35324-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35333-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35282-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35326-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35578-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35540-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35496-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35385-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35584-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35506-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35331-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35423-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35408-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35235-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35425-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35537-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35222-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35583-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35278-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35511-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35529-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35499-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35566-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35297-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35433-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35552-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35564-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35687-Data.db,
level=2),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35399-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35320-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35429-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35374-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35530-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35576-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35383-Data.db,
level=3),
SSTableReader(path=/data/cassandra/data/ks1/DataByUserID_005/ks1-DataByUserID_005-ka-35589-Data.db,
level=3), ]
> Constant compaction under LCS
> -----------------------------
>
> Key: CASSANDRA-8463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8463
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Hardware is recent 2-socket, 16-core (x2 Hyperthreaded),
> 144G RAM, solid-state storage.
> Platform is Linux 3.2.51, Oracle JDK 64-bit 1.7.0_65.
> Heap is 32G total, 4G newsize.
> 8G/8G on-heap/off-heap memtables, offheap_buffer allocator, 0.5
> memtable_cleanup_threshold
> concurrent_compactors: 20
> Reporter: Rick Branson
> Assignee: Marcus Eriksson
> Fix For: 2.1.3
>
> Attachments: 0001-better-logging.patch, log-for-8463.txt
>
>
> It appears that tables configured with LCS will completely re-compact
> themselves over some period of time after upgrading from 2.0 to 2.1 (2.0.11
> -> 2.1.2, specifically). It starts out with <10 pending tasks for an hour or
> so, then starts building up, now with 50-100 tasks pending across the cluster
> after 12 hours. These nodes are under heavy write load, but were easily able
> to keep up in 2.0 (they rarely had >5 pending compaction tasks), so I don't
> think it's LCS in 2.1 actually being worse, just perhaps some different LCS
> behavior that causes the layout of tables from 2.0 to prompt the compactor to
> reorganize them?
> The nodes flushed ~11MB SSTables under 2.0. They're currently flushing ~36MB
> SSTables due to the improved memtable setup in 2.1. Before I upgraded the
> entire cluster to 2.1, I noticed the problem and tried several variations on
> the flush size, thinking perhaps the larger tables in L0 were causing some
> kind of cascading compactions. Even if they're sized roughly like the 2.0
> flushes were, same behavior occurs. I also tried both enabling & disabling
> STCS in L0 with no real change other than L0 began to back up faster, so I
> left the STCS in L0 enabled.
> Tables are configured with 32MB sstable_size_in_mb, which was found to be an
> improvement on the 160MB table size for compaction performance. Maybe this is
> wrong now? Otherwise, the tables are configured with defaults. Compaction has
> been unthrottled to help them catch-up. The compaction threads stay very
> busy, with the cluster-wide CPU at 45% "nice" time. No nodes have completely
> caught up yet. I'll update JIRA with status about their progress if anything
> interesting happens.
> From a node around 12 hours ago, around an hour after the upgrade, with 19
> pending compaction tasks:
> SSTables in each level: [6/4, 10, 105/100, 268, 0, 0, 0, 0, 0]
> SSTables in each level: [6/4, 10, 106/100, 271, 0, 0, 0, 0, 0]
> SSTables in each level: [1, 16/10, 105/100, 269, 0, 0, 0, 0, 0]
> SSTables in each level: [5/4, 10, 103/100, 272, 0, 0, 0, 0, 0]
> SSTables in each level: [4, 11/10, 105/100, 270, 0, 0, 0, 0, 0]
> SSTables in each level: [1, 12/10, 105/100, 271, 0, 0, 0, 0, 0]
> SSTables in each level: [1, 14/10, 104/100, 267, 0, 0, 0, 0, 0]
> SSTables in each level: [9/4, 10, 103/100, 265, 0, 0, 0, 0, 0]
> Recently, with 41 pending compaction tasks:
> SSTables in each level: [4, 13/10, 106/100, 269, 0, 0, 0, 0, 0]
> SSTables in each level: [4, 12/10, 106/100, 273, 0, 0, 0, 0, 0]
> SSTables in each level: [5/4, 11/10, 106/100, 271, 0, 0, 0, 0, 0]
> SSTables in each level: [4, 12/10, 103/100, 275, 0, 0, 0, 0, 0]
> SSTables in each level: [2, 13/10, 106/100, 273, 0, 0, 0, 0, 0]
> SSTables in each level: [3, 10, 104/100, 275, 0, 0, 0, 0, 0]
> SSTables in each level: [6/4, 11/10, 103/100, 269, 0, 0, 0, 0, 0]
> SSTables in each level: [4, 16/10, 105/100, 264, 0, 0, 0, 0, 0]
> More information about the use case: writes are roughly uniform across these
> tables. The data is "sharded" across these 8 tables by key to improve
> compaction parallelism. Each node receives up to 75,000 writes/sec sustained
> at peak, and a small number of reads. This is a pre-production cluster that's
> being warmed up with new data, so the low volume of reads (~100/sec per node)
> is just from automatic sampled data checks, otherwise we'd just use STCS :)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
