If there were no other messages about anti-compaction similar to: > > SSTable YYY (ranges) will be anticompacted on range [range]
Then no anti-compaction needed to occur and yes, it was not the cause. On 5 April 2018 at 13:52, Dmitry Simonov <dimmobor...@gmail.com> wrote: > Hi, Evelyn! > > I've found the following messages: > > INFO RepairRunnable.java Starting repair command #41, repairing keyspace > XXX with repair options (parallelism: parallel, primary range: false, > incremental: false, job threads: 1, ColumnFamilies: [YYY], dataCenters: [], > hosts: [], # of ranges: 768) > INFO CompactionExecutor:6 CompactionManager.java Starting anticompaction > for XXX.YYY on 5132/5846 sstables > > After that many similar messages go: > SSTable BigTableReader(path='/mnt/cassandra/data/XXX/YYY- > 4c12fd9029e611e8810ac73ddacb37d1/lb-12688-big-Data.db') fully contained > in range (-9223372036854775808,-9223372036854775808], mutating repairedAt > instead of anticompacting > > Does it means that anti-compaction is not the cause? > > 2018-04-05 18:01 GMT+05:00 Evelyn Smith <u5015...@gmail.com>: > >> It might not be what cause it here. But check your logs for >> anti-compactions. >> >> >> On 5 Apr 2018, at 8:35 pm, Dmitry Simonov <dimmobor...@gmail.com> wrote: >> >> Thank you! >> I'll check this out. >> >> 2018-04-05 15:00 GMT+05:00 Alexander Dejanovski <a...@thelastpickle.com>: >> >>> 40 pending compactions is pretty high and you should have way less than >>> that most of the time, otherwise it means that compaction is not keeping up >>> with your write rate. >>> >>> If you indeed have SSDs for data storage, increase your compaction >>> throughput to 100 or 200 (depending on how the CPUs handle the load). You >>> can experiment with compaction throughput using : nodetool >>> setcompactionthroughput 100 >>> >>> You can raise the number of concurrent compactors as well and set it to >>> a value between 4 and 6 if you have at least 8 cores and CPUs aren't >>> overwhelmed. >>> >>> I'm not sure why you ended up with only one node having 6k SSTables and >>> not the others, but you should apply the above changes so that you can >>> lower the number of pending compactions and see if it prevents the issue >>> from happening again. >>> >>> Cheers, >>> >>> >>> On Thu, Apr 5, 2018 at 11:33 AM Dmitry Simonov <dimmobor...@gmail.com> >>> wrote: >>> >>>> Hi, Alexander! >>>> >>>> SizeTieredCompactionStrategy is used for all CFs in problematic >>>> keyspace. >>>> Current compaction throughput is 16 MB/s (default value). >>>> >>>> We always have about 40 pending and 2 active "CompactionExecutor" tasks >>>> in "tpstats". >>>> Mostly because of another (bigger) keyspace in this cluster. >>>> But the situation is the same on each node. >>>> >>>> According to "nodetool compactionhistory", compactions on this CF run >>>> (sometimes several times per day, sometimes one time per day, the last run >>>> was yesterday). >>>> We run "repair -full" regulary for this keyspace (every 24 hours on >>>> each node), because gc_grace_seconds is set to 24 hours. >>>> >>>> Should we consider increasing compaction throughput and >>>> "concurrent_compactors" (as recommended for SSDs) to keep >>>> "CompactionExecutor" pending tasks low? >>>> >>>> 2018-04-05 14:09 GMT+05:00 Alexander Dejanovski <a...@thelastpickle.com >>>> >: >>>> >>>>> Hi Dmitry, >>>>> >>>>> could you tell us which compaction strategy that table is currently >>>>> using ? >>>>> Also, what is the compaction max throughput and is auto-compaction >>>>> correctly enabled on that node ? >>>>> >>>>> Did you recently run repair ? >>>>> >>>>> Thanks, >>>>> >>>>> On Thu, Apr 5, 2018 at 10:53 AM Dmitry Simonov <dimmobor...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hello! >>>>>> >>>>>> Could you please give some ideas on the following problem? >>>>>> >>>>>> We have a cluster with 3 nodes, running Cassandra 2.2.11. >>>>>> >>>>>> We've recently discovered high CPU usage on one cluster node, after >>>>>> some investigation we found that number of sstables for one CF on it is >>>>>> very big: 5800 sstables, on other nodes: 3 sstable. >>>>>> >>>>>> Data size in this keyspace was not very big ~100-200Mb per node. >>>>>> >>>>>> There is no such problem with other CFs of that keyspace. >>>>>> >>>>>> nodetool compact solved the issue as a quick-fix. >>>>>> >>>>>> But I'm wondering, what was the cause? How prevent it from repeating? >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Dmitry Simonov >>>>>> >>>>> -- >>>>> ----------------- >>>>> Alexander Dejanovski >>>>> France >>>>> @alexanderdeja >>>>> >>>>> Consultant >>>>> Apache Cassandra Consulting >>>>> http://www.thelastpickle.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Dmitry Simonov >>>> >>> -- >>> ----------------- >>> Alexander Dejanovski >>> France >>> @alexanderdeja >>> >>> Consultant >>> Apache Cassandra Consulting >>> http://www.thelastpickle.com >>> >> >> >> >> -- >> Best Regards, >> Dmitry Simonov >> >> >> > > > -- > Best Regards, > Dmitry Simonov >
