Re: Compaction throughput

Thanks! On Friday, July 19, 2019, 10:15:43 PM GMT+3, Jon Haddad wrote: It's a limit on the total compaction throughput.  On Fri, Jul 19, 2019 at 10:39 AM Vlad wrote: Hi, is  'nodetool setcompactionthroughput' sets limit for all compactions on the node, or is it per compaction

Re: Compaction throughput

It's a limit on the total compaction throughput. On Fri, Jul 19, 2019 at 10:39 AM Vlad wrote: > Hi, > > is 'nodetool setcompactionthroughput' sets limit for all compactions on > the node, or is it per compaction thread? > > Thanks. >

Re: Compaction throughput vs. number of compaction threads?

Hi, The compaction throughput is indeed shared by all compactors. I would not advise to go below 8MB/s per compactor as slowing down compactions put more pressure on the heap. When tuning compaction, the first thing to do is evaluate the maximum throughput your disks can sustain without

Re: compaction throughput

Keep in mind that compaction in LCS can only run 1 compaction per level. Even if it wants to run more compactions in L0 it might be blocked because it is already running a compaction in L0. BR Jan On 01/16/2016 01:26 AM, Sebastian Estevez wrote: LCS is IO ontensive but CPU is also relevant.

Re: compaction throughput

Thanks for that clarification Sebastian! That's really good to know! I never took increasing this value in consideration because of my previous experience. In my case I had a table that was compacting over and over... and only one CPU was used. So that made me believe it was not multithreaded (I

Re: compaction throughput

That is interesting... We recently resolved a performance issue solely by increasing concurrent_compactors parameter from default to 64. We have two tables but 90% data is only in 1 table. We got read performance boost of more than 100% just by increasing that param in yaml. Based on what you

Re: compaction throughput

I am using 2.2.4 and have seen multiple compactors running on the same table. The number of compactors seems to be controlled by concurrent_compactors. As of type of compactions, I've seen normal compaction, tombstone compaction. Validation and Anticompaction seem to always be single threaded. On

Re: compaction throughput

>So compaction of one table will NOT spread over different cores. This is not exactly true. You actually can have multiple compactions running at the same time on the same table, it just doesn't happen all that often. You essentially would have to have two sets of sstables that are both eligible

Re: compaction throughput

@penguin There have been steady improvements in the different compaction strategies recently but not major re-writes. All the best, [image: datastax_logo.png] Sebastián Estévez Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com [image:

Re: compaction throughput

I forget to mention I am using C* 2.2.4 On Jan 15, 2016 3:53 PM, "Kai Wang" wrote: > Hi, > > I am trying to figure out the bottleneck of compaction on my node. The > node is CentOS 7 and has SSDs installed. The table is configured to use > LCS. Here is my compaction related

Re: compaction throughput

Compaction is generally CPU bound and relatively slow. Exactly why that is I’m uncertain. > On Jan 15, 2016, at 12:53 PM, Kai Wang wrote: > > Hi, > > I am trying to figure out the bottleneck of compaction on my node. The node > is CentOS 7 and has SSDs installed. The table

Re: compaction throughput

*nodetool setcompactionthroughput 0* Will only affect future compactions, not the ones that are currently running. All the best, [image: datastax_logo.png] Sebastián Estévez Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com [image: linkedin.png]

Re: compaction throughput

With SSDs, the typical recommendation is up to 0.8-1 compactor per core (depending on other load). How many CPU cores do you have? From: Kai Wang Reply-To: "user@cassandra.apache.org" Date: Friday, January 15, 2016 at 12:53 PM To: "user@cassandra.apache.org" Subject: compaction throughput

Re: compaction throughput

Jeff & Sebastian, Thanks for the reply. There are 12 cores but in my case C* only uses one core most of the time. *nodetool compactionstats* shows there's only one compactor running. I can see C* process only uses one core. So I guess I should've asked the question more clearly: 1. Is ~25 M/s a

Re: compaction throughput

Correct. Why are you concerned with the raw throughput, are you accumulating pending compactions? Are you seeing high sstables per read statistics? all the best, Sebastián On Jan 15, 2016 6:18 PM, "Kai Wang" wrote: > Jeff & Sebastian, > > Thanks for the reply. There are 12

Re: compaction throughput

LCS is IO ontensive but CPU is also relevant. On slower disks compaction may not be cpu bound. If you aren't seeing more than one compaction thread at a time, I suspect your system is not compaction bound. all the best, Sebastián On Jan 15, 2016 7:20 PM, "Kai Wang" wrote: >

Re: compaction throughput

Sebastian, Because I have this impression that LCS is IO intensive and it's recommended only on SSDs. So I am curious to see how far it can stress those SSDs. But it turns out the most expensive part about LCS is not IO bound but CUP bound, or more precisely single core speed bound. This is a

RE: compaction throughput rate not even close to 16MB

: Hiller, Dean [mailto:dean.hil...@nrel.gov] Sent: Wednesday, April 24, 2013 23:38 To: user@cassandra.apache.org Subject: Re: compaction throughput rate not even close to 16MB Thanks much!!! Better to hear at least one other person sees the same thing ;). Sometimes these posts just go silent

Re: compaction throughput rate not even close to 16MB

I have noticed the same. I think in the real world your compaction throughput is limited by other things. If I had to speculate I would say that compaction can remove expired tombstones, however doing this requires bloom filter checks, etc. I think that setting is more important with multi

Re: compaction throughput rate not even close to 16MB

On Wed, Apr 24, 2013 at 1:33 PM, Edward Capriolo edlinuxg...@gmail.com wrote: I think that setting is more important with multi threaded compaction and/or more compaction slots. In those cases it may actually throttle something. Or if you're simultaneously doing a repair, which does a

Re: compaction throughput rate not even close to 16MB

@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Wednesday, April 24, 2013 2:33 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: compaction throughput rate not even close to 16MB I have noticed the same. I think in the real

Re: compaction throughput rate not even close to 16MB

, we are on SSD. -Wei From: Hiller, Dean dean.hil...@nrel.gov To: user@cassandra.apache.org user@cassandra.apache.org Sent: Wednesday, April 24, 2013 1:37 PM Subject: Re: compaction throughput rate not even close to 16MB Thanks much!!!  Better to hear at least