There is some point where you simply need more machines. On Mon, Jan 7, 2013 at 5:02 PM, Michael Kjellman <mkjell...@barracuda.com>wrote:
> Right, I guess I'm saying that you should try loading your data with > leveled compaction and see how your compaction load is. > > Your work load sounds like leveled will fit much better than size tiered. > > From: Brian Tarbox <tar...@cabotresearch.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Monday, January 7, 2013 1:58 PM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: help turning compaction..hours of run to get 0% > compaction.... > > The problem I see is that it already takes me more than 24 hours just to > load my data...during which time the logs say I'm spending tons of time > doing compaction. For example in the last 72 hours I'm consumed* 20 hours > * per machine on compaction. > > Can I conclude from that than I should be (perhaps drastically) increasing > my compaction_mb_per_sec on the theory that I'm getting behind? > > The fact that it takes me 3 days or more to run a test means its hard to > just play with values and see what works best, so I'm trying to understand > the behavior in detail. > > Thanks. > > Brain > > > On Mon, Jan 7, 2013 at 4:13 PM, Michael Kjellman > <mkjell...@barracuda.com>wrote: > >> http://www.datastax.com/dev/blog/when-to-use-leveled-compaction >> >> "If you perform at least twice as many reads as you do writes, leveled >> compaction may actually save you disk I/O, despite consuming more I/O for >> compaction. This is especially true if your reads are fairly random and >> don’t focus on a single, hot dataset." >> >> From: Brian Tarbox <tar...@cabotresearch.com> >> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Date: Monday, January 7, 2013 12:56 PM >> To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Subject: Re: help turning compaction..hours of run to get 0% >> compaction.... >> >> I have not specified leveled compaction so I guess I'm defaulting to size >> tiered? My data (in the column family causing the trouble) insert once, >> ready many, update-never. >> >> Brian >> >> >> On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman <mkjell...@barracuda.com >> > wrote: >> >>> Size tiered or leveled compaction? >>> >>> From: Brian Tarbox <tar...@cabotresearch.com> >>> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> >>> Date: Monday, January 7, 2013 12:03 PM >>> To: "user@cassandra.apache.org" <user@cassandra.apache.org> >>> Subject: help turning compaction..hours of run to get 0% compaction.... >>> >>> I have a column family where I'm doing 500 inserts/sec for 12 hours or >>> so at time. At some point my performance falls off a cliff due to time >>> spent doing compactions. >>> >>> I'm seeing row after row of logs saying that after 1 or 2 hours of >>> compactiing it reduced to 100% of 99% of the original. >>> >>> I'm trying to understand what direction this data points me to in term >>> of configuration change. >>> >>> a) increase my compaction_throughput_mb_per_sec because I'm falling >>> behind (am I falling behind?) >>> >>> b) enable multi-threaded compaction? >>> >>> Any help is appreciated. >>> >>> Brian >>> >>> ---------------------------------- >>> Join Barracuda Networks in the fight against hunger. >>> To learn how you can help in your community, please visit: >>> http://on.fb.me/UAdL4f >>> >>> >> >> >> ---------------------------------- >> Join Barracuda Networks in the fight against hunger. >> To learn how you can help in your community, please visit: >> http://on.fb.me/UAdL4f >> >> > > > ---------------------------------- > Join Barracuda Networks in the fight against hunger. > To learn how you can help in your community, please visit: > http://on.fb.me/UAdL4f > >
