Re: Speed up compaction
Hi Léo, Major compactions in LCS (and minor as well) are very slow indeed and I'm afraid there's not much you can do to speed things up. There are lots of synchronized sections in the LCS code and it has to do a lot of comparisons between sstables to make sure a partition won't end up in two sstables of the same level. A major compaction will be single threaded for obvious reasons, and while this is happening you might have all the newly flushed SSTables that will pile up in S0 since I don't see how Cassandra could achieve the "one sstable per partition per level except L0" guarantee otherwise. At this point, your best chance might be to switch the table to STCS, run a major compaction using the "-s" flag (split output, which will create one SSTable per size tier instead of a big fat one) and then back to LCS, before or after your migration (whatever works best for you). If you go down that path, I'd also recommend to try it up on one node using JMX to alter the compaction strategy, run the major compaction with nodetool and see if it's indeed faster than the LCS major compaction. Then, proceed node by node using JMX (wait for the major compaction to go through between nodes) and alter the schema only after the last node has been switched to STCS. You can use more "aggressive" compaction settings to limit read fragmentation reducing max_threshold to 3 instead of 4 (the default). Note that doing all this will impact your cluster performance in ways I cannot predict, and should be attempted only if you really need to perform this major compaction and cannot wait for it to go through at the current pace. Cheers, - Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com On Thu, Jun 13, 2019 at 2:07 PM Léo FERLIN SUTTON wrote: > On Thu, Jun 13, 2019 at 12:09 PM Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > >> On Thu, Jun 13, 2019 at 11:28 AM Léo FERLIN SUTTON >> wrote: >> >>> >>> ## Cassandra configuration : >>> 4 concurrent_compactors >>> Current compaction throughput: 150 MB/s >>> Concurrent reads/write are both set to 128. >>> >>> I have also temporarily stopped every repair operations. >>> >>> Any ideas about how I can speed this up ? >>> >> >> Hi, >> >> What is the compaction strategy used by this column family? >> >> Do you observe this behavior on one of the nodes only? Have you tried to >> cancel this compaction and see if a new one is started and makes better >> progress? Can you try to restart the affected node? >> >> Regards, >> -- >> Alex >> >> I can't believe I forgot that information. > > Overall we are talking about a 1.08TB table, using LCS. > > SSTable count: 1047 >> SSTables in each level: [15/4, 10, 103/100, 918, 0, 0, 0, 0, 0] > > SSTable Compression Ratio: 0.5192269874287099 > > Number of partitions (estimate): 7282253587 > > > We have recently (about a month ago) deleted about 25% of the data in that > table. > > Letting Cassandra reclaim the disk space on it's own (via regular > compactions) was too slow for us, so we wanted to force a compaction on the > table to reclaim the disk space faster. > > The speed of the compaction doesn't seem out of the ordinary for the > cluster, only before we haven't had such a big compaction and the speed > alarmed us. > We never have a big compaction backlog, most of the time less than 5 > pending tasks (per node) > > Finally but we are running Cassandra 3.0.18 and plan to upgrade to 3.11 as > soon as our compactions are over. > > Regards, > > Leo >
Re: Speed up compaction
On Thu, Jun 13, 2019 at 2:44 PM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Thu, Jun 13, 2019 at 2:07 PM Léo FERLIN SUTTON > wrote: > >> >> Overall we are talking about a 1.08TB table, using LCS. >> >> SSTable count: 1047 >>> SSTables in each level: [15/4, 10, 103/100, 918, 0, 0, 0, 0, 0] >> >> SSTable Compression Ratio: 0.5192269874287099 >> >> Number of partitions (estimate): 7282253587 >> >> >> We have recently (about a month ago) deleted about 25% of the data in >> that table. >> >> Letting Cassandra reclaim the disk space on it's own (via regular >> compactions) was too slow for us, so we wanted to force a compaction on the >> table to reclaim the disk space faster. >> > > To be clear, that compaction task is running the major compaction for this > column family? I have no experience with Leveled compaction strategy, so > not really sure what behavior to expect from it. I can imagine that with > that many SSTables and a major compaction, there might be quite some > overhead as it does more than an ordinary merge-sort as I would expect from > Size-tiered. > Yes It is running a major compaction. Even with a lot of overhead, we expect it to take more than two weeks :( We are hoping to find some way to speed up things a bit.
Re: Speed up compaction
On Thu, Jun 13, 2019 at 2:07 PM Léo FERLIN SUTTON wrote: > > Overall we are talking about a 1.08TB table, using LCS. > > SSTable count: 1047 >> SSTables in each level: [15/4, 10, 103/100, 918, 0, 0, 0, 0, 0] > > SSTable Compression Ratio: 0.5192269874287099 > > Number of partitions (estimate): 7282253587 > > > We have recently (about a month ago) deleted about 25% of the data in that > table. > > Letting Cassandra reclaim the disk space on it's own (via regular > compactions) was too slow for us, so we wanted to force a compaction on the > table to reclaim the disk space faster. > To be clear, that compaction task is running the major compaction for this column family? I have no experience with Leveled compaction strategy, so not really sure what behavior to expect from it. I can imagine that with that many SSTables and a major compaction, there might be quite some overhead as it does more than an ordinary merge-sort as I would expect from Size-tiered. -- Alex
Re: Speed up compaction
On Thu, Jun 13, 2019 at 12:09 PM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Thu, Jun 13, 2019 at 11:28 AM Léo FERLIN SUTTON > wrote: > >> >> ## Cassandra configuration : >> 4 concurrent_compactors >> Current compaction throughput: 150 MB/s >> Concurrent reads/write are both set to 128. >> >> I have also temporarily stopped every repair operations. >> >> Any ideas about how I can speed this up ? >> > > Hi, > > What is the compaction strategy used by this column family? > > Do you observe this behavior on one of the nodes only? Have you tried to > cancel this compaction and see if a new one is started and makes better > progress? Can you try to restart the affected node? > > Regards, > -- > Alex > > I can't believe I forgot that information. Overall we are talking about a 1.08TB table, using LCS. SSTable count: 1047 > SSTables in each level: [15/4, 10, 103/100, 918, 0, 0, 0, 0, 0] SSTable Compression Ratio: 0.5192269874287099 Number of partitions (estimate): 7282253587 We have recently (about a month ago) deleted about 25% of the data in that table. Letting Cassandra reclaim the disk space on it's own (via regular compactions) was too slow for us, so we wanted to force a compaction on the table to reclaim the disk space faster. The speed of the compaction doesn't seem out of the ordinary for the cluster, only before we haven't had such a big compaction and the speed alarmed us. We never have a big compaction backlog, most of the time less than 5 pending tasks (per node) Finally but we are running Cassandra 3.0.18 and plan to upgrade to 3.11 as soon as our compactions are over. Regards, Leo
Re: Speed up compaction
On Thu, Jun 13, 2019 at 11:28 AM Léo FERLIN SUTTON wrote: > > ## Cassandra configuration : > 4 concurrent_compactors > Current compaction throughput: 150 MB/s > Concurrent reads/write are both set to 128. > > I have also temporarily stopped every repair operations. > > Any ideas about how I can speed this up ? > Hi, What is the compaction strategy used by this column family? Do you observe this behavior on one of the nodes only? Have you tried to cancel this compaction and see if a new one is started and makes better progress? Can you try to restart the affected node? Regards, -- Alex
Speed up compaction
I am currently noticing very very slow compactions on my cluster and wondering if there is any way to speed things up. Right now I have this compaction currently running : 60c1cfc0-8da7-11e9-bc08-3546c703a280Compaction keyspace1 > table1 8.77 GB 1.71 TB bytes 0.50% The speed is abysmal, it only progresses by "0.24%" every _hour_. At this rate it is going to take weeks... When I look at the cpu load and disk load I see : Never more than 10MB/s read and write combined. Never more than 6 load average, most of the cpu time is spent idle. ## Hardware configuration : The node is a n1-highmem-8 (8 vCPUs, 52 GB memory) instance on google cloud engine. The disk performances should be around : 15,000.00 random IOPS (read write) 800MB/s Read 400 MB/s Write ## Cassandra configuration : 4 concurrent_compactors Current compaction throughput: 150 MB/s Concurrent reads/write are both set to 128. I have also temporarily stopped every repair operations. Any ideas about how I can speed this up ? Regards, Leo