Re: help turning compaction..hours of run to get 0% compaction....
One metric to watch is pending compactions (via nodetool compactionstats). This count will give you some idea of whether you are falling behind with compactions. The other measure is how long you are compacting after your inserts have stopped. If I understand correctly, since you never update the data, that would explain why the compaction logging shows 100% of orig. With size-tiered, you are flushing small files, compacting when you get 4 of like size, etc. Since you have no updates, the compaction will not shrink the data. As Aaron said, use iostat –x (or dstat) to see if you are taxing the disks. If so, then leveled compaction may be your option (for reasons already stated). If not taxing the disks, then you might want to increase your compaction throughput, as you suggested. Depending on what version you are using, another thing to possibly tune is the size of sstables when flushed to disk. In your case of insert only, the smaller the flush size, the more times that row is going to be rewritten during a compaction (hence increase I/O). jc From: Edward Capriolo edlinuxg...@gmail.commailto:edlinuxg...@gmail.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 2:33 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction There is some point where you simply need more machines. On Mon, Jan 7, 2013 at 5:02 PM, Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com wrote: Right, I guess I'm saying that you should try loading your data with leveled compaction and see how your compaction load is. Your work load sounds like leveled will fit much better than size tiered. From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 1:58 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction The problem I see is that it already takes me more than 24 hours just to load my data...during which time the logs say I'm spending tons of time doing compaction. For example in the last 72 hours I'm consumed 20 hours per machine on compaction. Can I conclude from that than I should be (perhaps drastically) increasing my compaction_mb_per_sec on the theory that I'm getting behind? The fact that it takes me 3 days or more to run a test means its hard to just play with values and see what works best, so I'm trying to understand the behavior in detail. Thanks. Brain On Mon, Jan 7, 2013 at 4:13 PM, Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com wrote: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction If you perform at least twice as many reads as you do writes, leveled compaction may actually save you disk I/O, despite consuming more I/O for compaction. This is especially true if your reads are fairly random and don’t focus on a single, hot dataset. From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 12:56 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction I have not specified leveled compaction so I guess I'm defaulting to size tiered? My data (in the column family causing the trouble) insert once, ready many, update-never. Brian On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com wrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec
Re: help turning compaction..hours of run to get 0% compaction....
i'll second edward's comment. cassandra is designed to scale horizontally, so if disk I/O is slowing you down then you must scale On Tue, Jan 8, 2013 at 7:10 AM, Jim Cistaro jcist...@netflix.com wrote: One metric to watch is pending compactions (via nodetool compactionstats). This count will give you some idea of whether you are falling behind with compactions. The other measure is how long you are compacting after your inserts have stopped. If I understand correctly, since you never update the data, that would explain why the compaction logging shows 100% of orig. With size-tiered, you are flushing small files, compacting when you get 4 of like size, etc. Since you have no updates, the compaction will not shrink the data. As Aaron said, use iostat –x (or dstat) to see if you are taxing the disks. If so, then leveled compaction may be your option (for reasons already stated). If not taxing the disks, then you might want to increase your compaction throughput, as you suggested. Depending on what version you are using, another thing to possibly tune is the size of sstables when flushed to disk. In your case of insert only, the smaller the flush size, the more times that row is going to be rewritten during a compaction (hence increase I/O). jc From: Edward Capriolo edlinuxg...@gmail.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 2:33 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction There is some point where you simply need more machines. On Mon, Jan 7, 2013 at 5:02 PM, Michael Kjellman mkjell...@barracuda.comwrote: Right, I guess I'm saying that you should try loading your data with leveled compaction and see how your compaction load is. Your work load sounds like leveled will fit much better than size tiered. From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 1:58 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction The problem I see is that it already takes me more than 24 hours just to load my data...during which time the logs say I'm spending tons of time doing compaction. For example in the last 72 hours I'm consumed* 20 hours* per machine on compaction. Can I conclude from that than I should be (perhaps drastically) increasing my compaction_mb_per_sec on the theory that I'm getting behind? The fact that it takes me 3 days or more to run a test means its hard to just play with values and see what works best, so I'm trying to understand the behavior in detail. Thanks. Brain On Mon, Jan 7, 2013 at 4:13 PM, Michael Kjellman mkjell...@barracuda.com wrote: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction If you perform at least twice as many reads as you do writes, leveled compaction may actually save you disk I/O, despite consuming more I/O for compaction. This is especially true if your reads are fairly random and don’t focus on a single, hot dataset. From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:56 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction I have not specified leveled compaction so I guess I'm defaulting to size tiered? My data (in the column family causing the trouble) insert once, ready many, update-never. Brian On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman mkjell...@barracuda.com wrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your
help turning compaction..hours of run to get 0% compaction....
I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian
Re: help turning compaction..hours of run to get 0% compaction....
Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian Southfield Public School students safely access the tech tools they need on and off campus with the Barracuda Web Filter. Quick installation and easy to use- try the Barracuda Web Filter free for 30 days: http://on.fb.me/Vj6JBd
Re: help turning compaction..hours of run to get 0% compaction....
Take a look at iostat -x 5 to see if your disks are dogging it. Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 8/01/2013, at 9:13 AM, Michael Kjellman mkjell...@barracuda.com wrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f
Re: help turning compaction..hours of run to get 0% compaction....
I have not specified leveled compaction so I guess I'm defaulting to size tiered? My data (in the column family causing the trouble) insert once, ready many, update-never. Brian On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman mkjell...@barracuda.comwrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f
Re: help turning compaction..hours of run to get 0% compaction....
http://www.datastax.com/dev/blog/when-to-use-leveled-compaction If you perform at least twice as many reads as you do writes, leveled compaction may actually save you disk I/O, despite consuming more I/O for compaction. This is especially true if your reads are fairly random and don’t focus on a single, hot dataset. From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 12:56 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction I have not specified leveled compaction so I guess I'm defaulting to size tiered? My data (in the column family causing the trouble) insert once, ready many, update-never. Brian On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com wrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f Southfield Public School students safely access the tech tools they need on and off campus with the Barracuda Web Filter. Quick installation and easy to use- try the Barracuda Web Filter free for 30 days: http://on.fb.me/Vj6JBd
Re: help turning compaction..hours of run to get 0% compaction....
The problem I see is that it already takes me more than 24 hours just to load my data...during which time the logs say I'm spending tons of time doing compaction. For example in the last 72 hours I'm consumed* 20 hours*per machine on compaction. Can I conclude from that than I should be (perhaps drastically) increasing my compaction_mb_per_sec on the theory that I'm getting behind? The fact that it takes me 3 days or more to run a test means its hard to just play with values and see what works best, so I'm trying to understand the behavior in detail. Thanks. Brain On Mon, Jan 7, 2013 at 4:13 PM, Michael Kjellman mkjell...@barracuda.comwrote: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction If you perform at least twice as many reads as you do writes, leveled compaction may actually save you disk I/O, despite consuming more I/O for compaction. This is especially true if your reads are fairly random and don’t focus on a single, hot dataset. From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:56 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction I have not specified leveled compaction so I guess I'm defaulting to size tiered? My data (in the column family causing the trouble) insert once, ready many, update-never. Brian On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman mkjell...@barracuda.comwrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f
Re: help turning compaction..hours of run to get 0% compaction....
Right, I guess I'm saying that you should try loading your data with leveled compaction and see how your compaction load is. Your work load sounds like leveled will fit much better than size tiered. From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 1:58 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction The problem I see is that it already takes me more than 24 hours just to load my data...during which time the logs say I'm spending tons of time doing compaction. For example in the last 72 hours I'm consumed 20 hours per machine on compaction. Can I conclude from that than I should be (perhaps drastically) increasing my compaction_mb_per_sec on the theory that I'm getting behind? The fact that it takes me 3 days or more to run a test means its hard to just play with values and see what works best, so I'm trying to understand the behavior in detail. Thanks. Brain On Mon, Jan 7, 2013 at 4:13 PM, Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com wrote: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction If you perform at least twice as many reads as you do writes, leveled compaction may actually save you disk I/O, despite consuming more I/O for compaction. This is especially true if your reads are fairly random and don’t focus on a single, hot dataset. From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 12:56 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction I have not specified leveled compaction so I guess I'm defaulting to size tiered? My data (in the column family causing the trouble) insert once, ready many, update-never. Brian On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman mkjell...@barracuda.commailto:mkjell...@barracuda.com wrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.commailto:tar...@cabotresearch.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f Southfield Public School students safely access the tech tools they need on and off campus with the Barracuda Web Filter. Quick installation and easy to use- try the Barracuda Web Filter free for 30 days: http://on.fb.me/Vj6JBd
Re: help turning compaction..hours of run to get 0% compaction....
There is some point where you simply need more machines. On Mon, Jan 7, 2013 at 5:02 PM, Michael Kjellman mkjell...@barracuda.comwrote: Right, I guess I'm saying that you should try loading your data with leveled compaction and see how your compaction load is. Your work load sounds like leveled will fit much better than size tiered. From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 1:58 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction The problem I see is that it already takes me more than 24 hours just to load my data...during which time the logs say I'm spending tons of time doing compaction. For example in the last 72 hours I'm consumed* 20 hours * per machine on compaction. Can I conclude from that than I should be (perhaps drastically) increasing my compaction_mb_per_sec on the theory that I'm getting behind? The fact that it takes me 3 days or more to run a test means its hard to just play with values and see what works best, so I'm trying to understand the behavior in detail. Thanks. Brain On Mon, Jan 7, 2013 at 4:13 PM, Michael Kjellman mkjell...@barracuda.comwrote: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction If you perform at least twice as many reads as you do writes, leveled compaction may actually save you disk I/O, despite consuming more I/O for compaction. This is especially true if your reads are fairly random and don’t focus on a single, hot dataset. From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:56 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: help turning compaction..hours of run to get 0% compaction I have not specified leveled compaction so I guess I'm defaulting to size tiered? My data (in the column family causing the trouble) insert once, ready many, update-never. Brian On Mon, Jan 7, 2013 at 3:13 PM, Michael Kjellman mkjell...@barracuda.com wrote: Size tiered or leveled compaction? From: Brian Tarbox tar...@cabotresearch.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, January 7, 2013 12:03 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: help turning compaction..hours of run to get 0% compaction I have a column family where I'm doing 500 inserts/sec for 12 hours or so at time. At some point my performance falls off a cliff due to time spent doing compactions. I'm seeing row after row of logs saying that after 1 or 2 hours of compactiing it reduced to 100% of 99% of the original. I'm trying to understand what direction this data points me to in term of configuration change. a) increase my compaction_throughput_mb_per_sec because I'm falling behind (am I falling behind?) b) enable multi-threaded compaction? Any help is appreciated. Brian -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f -- Join Barracuda Networks in the fight against hunger. To learn how you can help in your community, please visit: http://on.fb.me/UAdL4f
