Commitlog files not getting deleted
Hi Users, In our cluster, the commit log is getting filled up as write progresses. It is noticed that once the commit log is flushed to SSTable the commit log files are not removed/deleted. The result of that the commit log volume is getting filled with commit log files. Any reason why commit log is not cleared even after successful flush. Thanks, Jayant K Kenjale
Re: Commitlog files not getting deleted
its DSE 3.1 Cassandra 2.1 On Thu, Aug 22, 2013 at 10:28 AM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 22, 2013 at 10:24 AM, Jay Svc jaytechg...@gmail.com wrote: In our cluster, the commit log is getting filled up as write progresses. It is noticed that once the commit log is flushed to SSTable the commit log files are not removed/deleted. The result of that the commit log volume is getting filled with commit log files. What version of Cassandra? There are some older versions with bugs like this, but haven't heard this symptom in a while... =Rob
Re: How to make compaction run faster?
Thanks Aaron, The parameters I tried above are set one at a time, based on what I observed, the problem at the core is that can compaction catch up with write speed. I have gone up to 30,000 to 35000 writes per second. I do not see number of writes a much issue either. I see compaction is not catching up with a write speed is an issue, in spite of I have more CPU and memory. Because over the period of time, we will see growing number of pending compactions, as write continues, that will degrade my read performance. Do you think STCS is compaction strategy to speed up compaction? What is a good approach when we have greater number of reads and writes, so compaction catch up with the write speed? Thank you in advance. Jay On Sun, Apr 21, 2013 at 1:43 PM, aaron morton aa...@thelastpickle.comwrote: You are suggesting to go back to STCS and increase the compaction_throughput step by step to see if compaction catch up with write traffic? As a personal approach, when so many config settings are changed it becomes impossible to understand cause and effect. So I try to return to a known base line and then make changes. As I watched Disk latency on DSE Opscenter as well as on iostat the await is always 35 to 40 ms for longer period of time during the test. You previously said this was the await on the commit log. What is the queue size ? The problem sounds like IO is not keeping up, moving to STS will reduce the IO. Levelled Compaction is designed to reduce the number os SSTables in a read, not to do compaction faster. At some point you may be writing too fast for the nodes. I'm not sure if you have discussed the level of writes going through the system. Get something that works and then make one change at a time until it does not. You should then be able to say The system can handle X writes of Y size per second, but after that compaction cannot keep up. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 19/04/2013, at 7:16 AM, Jay Svc jaytechg...@gmail.com wrote: Thanks Aaron, Please find answers to your questions. 1. I started test with default parameters the compaction is backing up. So went for various options. 2. The data is on RAID10. 3. As I watched Disk latency on DSE Opscenter as well as on iostat the await is always 35 to 40 ms for longer period of time during the test. (which probably gives me high write latency on client side) Do you think this could contribute to slowing down the compaction? probably not..! So Aaron, I am trying to understand - You are suggesting to go back to STCS and increase the compaction_throughput step by step to see if compaction catch up with write traffic? Thank you for your inputs. Regards, Jay On Thu, Apr 18, 2013 at 1:52 PM, aaron morton aa...@thelastpickle.comwrote: Parameters used: • SSTable size: 500MB (tried various sizes from 20MB to 1GB) • Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) • Concurrent write: 196 (tried from 32 to 296) • Concurrent compactors: 72 (tried disabling to making it 172) • Multithreaded compaction: true (tried both true and false) • Compaction strategy: LCS (tried STCS as well) • Memtable total space in mb: 4096 MB (tried default and some other params too) I would restore to default settings before I did anything else. Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog disks. Please find the iostat of some of 3 different boxes in my cluster. What is the data on ? It's important to call iostat with a period and watch the await / queue size of time. Not just view a snapshot. I would go back to STS with default settings, and ramp up the throughput until compaction cannot keep up. Then increase the throughout and see how that works. Then increase throughput again and see what happens. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 19/04/2013, at 5:05 AM, Jay Svc jaytechg...@gmail.com wrote: Hi Aaron, Alexis, Thanks for reply, Please find some more details below. Core problems: Compaction is taking longer time to finish. So it will affect my reads. I have more CPU and memory, want to utilize that to speed up the compaction process. Parameters used: • SSTable size: 500MB (tried various sizes from 20MB to 1GB) • Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) • Concurrent write: 196 (tried from 32 to 296) • Concurrent compactors: 72 (tried disabling to making it 172) • Multithreaded compaction: true (tried both true and false) • Compaction strategy: LCS (tried STCS as well) • Memtable total space in mb: 4096 MB (tried default and some other params too) Note: I have tried almost all permutation combination
Re: How to make compaction run faster?
Hi Alexis, Yes compaction happens on data files - 1. What my disk latency is high for SSDs which been only used for commit log 2. Why my compaction is not catching up with my write traffic in spite of low CPU, low memory and low JVM usage. I am adding more details to this thread. Thanks, Jayant K Kenjale On Wed, Apr 17, 2013 at 6:58 PM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: Jay, I believe that compaction occurs on the data directories and not in the commitlog. http://wiki.apache.org/cassandra/MemtableSSTable On Wed, Apr 17, 2013 at 7:58 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Alexis, Thank you for your response. My commit log is on SSD. which shows me 30 to 40 ms of disk latency. When I ran iostat; I see await 26ms to 30 ms for my commit log disk. My CPU is less than 18% used. How I reduce the disk latency for my commit log disk. They are SSDs. Thank you in advance, Jay On Wed, Apr 17, 2013 at 3:58 PM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: :D Jay, check if your disk(s) utilization allows you to change the configuration the way Edward suggest. iostat -xkcd 1 will show you how much of your disk(s) are in use. On Wed, Apr 17, 2013 at 5:26 PM, Edward Capriolo edlinuxg...@gmail.comwrote: three things: 1) compaction throughput is fairly low (yaml nodetool) 2) concurrent compactions is fairly low (yaml) 3) multithreaded compaction might be off in your version Try raising these things. Otherwise consider option 4. 4)$$$ RAID,RAMCPU$$ On Wed, Apr 17, 2013 at 4:01 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Team, I have a high write traffic to my Cassandra cluster. I experience a very high number of pending compactions. As I expect higher writes, The pending compactions keep increasing. Even when I stop my writes it takes several hours to finishing pending compactions. My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. How can I increase the compaction rate so it will run bit faster to match my write speed? Your inputs are appreciated. Thanks, Jay
Re: How to make compaction run faster?
Hi Aaron, Alexis, Thanks for reply, Please find some more details below. *Core problems:* Compaction is taking longer time to finish. So it will affect my reads. I have more CPU and memory, want to utilize that to speed up the compaction process. *Parameters used:* 1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) 2. Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) 3. Concurrent write: 196 (tried from 32 to 296) 4. Concurrent compactors: 72 (tried disabling to making it 172) 5. Multithreaded compaction: true (tried both true and false) 6. Compaction strategy: LCS (tried STCS as well) 7. Memtable total space in mb: 4096 MB (tried default and some other params too) Note: I have tried almost all permutation combination of these parameters. *Observations: * I ran test for 1.15 hrs with writes at the rate of 21000 records/sec(total 60GB data during 1.15 hrs). After I stopped the test compaction took additional 1.30 hrs to finish compaction, that reduced the SSTable count from 170 to 17. CPU(24 cores): almost 80% idle during the run JVM: 48G RAM, 8G Heap, (3G to 5G heap used) Pending Writes: sometimes high spikes for small amount of time otherwise pretty flat Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog disks. Please find the iostat of some of 3 different boxes in my cluster. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.11 0.59 0.01 0.00 97.09 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 0.3224.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32 dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.12 0.52 0.01 0.00 97.14 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. dm-2 0.00 0.00 0.08 175.93 10.52 703.74 8.12 3.13 17.78 0. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 1.14 0. dm-4 0.00 0.00 9.19 427.55 1166.91 1710.21 13.18 4.67 10.65 0. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.15 1.13 0.52 0.01 0.00 97.19 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.02 429.97 9.28 7.29 1176.81 1749.00 353.12 0.07 4.10 0.55 0.91 sdb 0.00 173.65 0.08 3.09 10.50 706.96 452.25 0.09 27.23 0.99 0.31 dm-0 0.00 0.00 0.04 0.79 0.82 3.16 9.61 0.00 1.54 0.27 0.02 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.68 0.63 0.00 dm-2 0.00 0.00 0.08 176.74 10.50 706.96 8.12 3.46 19.53 0.02 0.31 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.85 0.83 0.00 dm-4 0.00 0.00 9.26 436.46 1175.98 1745.84 13.11 0.03 0.03 0.02 0.89 Thanks, Jay On Thu, Apr 18, 2013 at 2:50 AM, aaron morton aa...@thelastpickle.comwrote: I believe that compaction occurs on the data directories and not in the commitlog. Yes, compaction only works on the data files. When I ran iostat; I see await 26ms to 30 ms for my commit log disk. My CPU is less than 18% used. How I reduce the disk latency for my commit log disk. They are SSDs. That does not sound right. Can you include the output from iostat for the commit log and data volumes. Also some information on how many writes you are processing the the size of rows as well. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 18/04/2013, at 11:58 AM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: Jay, I believe that compaction occurs on the data directories and not in the commitlog. http://wiki.apache.org/cassandra/MemtableSSTable On Wed, Apr 17, 2013 at 7:58 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Alexis, Thank you for your response. My commit log is on SSD. which shows me 30 to 40 ms of disk latency. When I ran iostat; I see await 26ms to 30 ms for my commit log disk. My CPU is less than 18% used. How I reduce the disk latency for my commit log disk. They are SSDs. Thank you in advance, Jay On Wed, Apr 17, 2013 at 3:58 PM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: :D Jay, check
Re: How to make compaction run faster?
Looks like formatting is bit messed up. Please let me know if you want the same in clean format. Thanks, Jay On Thu, Apr 18, 2013 at 12:05 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Aaron, Alexis, Thanks for reply, Please find some more details below. *Core problems:* Compaction is taking longer time to finish. So it will affect my reads. I have more CPU and memory, want to utilize that to speed up the compaction process. *Parameters used:* 1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) 2. Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) 3. Concurrent write: 196 (tried from 32 to 296) 4. Concurrent compactors: 72 (tried disabling to making it 172) 5. Multithreaded compaction: true (tried both true and false) 6. Compaction strategy: LCS (tried STCS as well) 7. Memtable total space in mb: 4096 MB (tried default and some other params too) Note: I have tried almost all permutation combination of these parameters. *Observations: * I ran test for 1.15 hrs with writes at the rate of 21000 records/sec(total 60GB data during 1.15 hrs). After I stopped the test compaction took additional 1.30 hrs to finish compaction, that reduced the SSTable count from 170 to 17. CPU(24 cores): almost 80% idle during the run JVM: 48G RAM, 8G Heap, (3G to 5G heap used) Pending Writes: sometimes high spikes for small amount of time otherwise pretty flat Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog disks. Please find the iostat of some of 3 different boxes in my cluster. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.11 0.59 0.01 0.00 97.09 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 0.3224.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32 dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.12 0.52 0.01 0.00 97.14 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. dm-2 0.00 0.00 0.08 175.93 10.52 703.74 8.12 3.13 17.78 0. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 1.14 0. dm-4 0.00 0.00 9.19 427.55 1166.91 1710.21 13.18 4.67 10.65 0. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.15 1.13 0.52 0.01 0.00 97.19 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.02 429.97 9.28 7.29 1176.81 1749.00 353.12 0.07 4.10 0.55 0.91 sdb 0.00 173.65 0.08 3.09 10.50 706.96 452.25 0.09 27.23 0.99 0.31 dm-0 0.00 0.00 0.04 0.79 0.82 3.16 9.61 0.00 1.54 0.27 0.02 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.68 0.63 0.00 dm-2 0.00 0.00 0.08 176.74 10.50 706.96 8.12 3.46 19.53 0.02 0.31 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.85 0.83 0.00 dm-4 0.00 0.00 9.26 436.46 1175.98 1745.84 13.11 0.03 0.03 0.02 0.89 Thanks, Jay On Thu, Apr 18, 2013 at 2:50 AM, aaron morton aa...@thelastpickle.comwrote: I believe that compaction occurs on the data directories and not in the commitlog. Yes, compaction only works on the data files. When I ran iostat; I see await 26ms to 30 ms for my commit log disk. My CPU is less than 18% used. How I reduce the disk latency for my commit log disk. They are SSDs. That does not sound right. Can you include the output from iostat for the commit log and data volumes. Also some information on how many writes you are processing the the size of rows as well. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 18/04/2013, at 11:58 AM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: Jay, I believe that compaction occurs on the data directories and not in the commitlog. http://wiki.apache.org/cassandra/MemtableSSTable On Wed, Apr 17, 2013 at 7:58 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Alexis, Thank you for your response. My commit log is on SSD. which shows me 30 to 40 ms of disk latency. When I ran iostat; I see await 26ms to 30 ms for my
Re: How to make compaction run faster?
Hi Alexis, Thank you so much for inputs. Let me try those suggested options. I will also look forward to see if you have any more suggestions on our Compaction. Thanks, Jay On Thu, Apr 18, 2013 at 1:03 PM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: Jay, await, according to iostat's man page it is the time of a request to the disk to get served. You may try changing the io scheduler. I've read that noop it's recommended for SSDs, you can check here http://goo.gl/XMiIA Regarding compaction, a week ago we had serious problems with compaction in a test machine, solved by changing from openjdk 1.6 to sun-jdk 1.6. On Thu, Apr 18, 2013 at 2:08 PM, Jay Svc jaytechg...@gmail.com wrote: By the way the compaction and commit log disk latency, these are two seperate problems I see. The important one is compaction problem, How I can speed that up? Thanks, Jay On Thu, Apr 18, 2013 at 12:07 PM, Jay Svc jaytechg...@gmail.com wrote: Looks like formatting is bit messed up. Please let me know if you want the same in clean format. Thanks, Jay On Thu, Apr 18, 2013 at 12:05 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Aaron, Alexis, Thanks for reply, Please find some more details below. *Core problems:* Compaction is taking longer time to finish. So it will affect my reads. I have more CPU and memory, want to utilize that to speed up the compaction process. *Parameters used:* 1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) 2. Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) 3. Concurrent write: 196 (tried from 32 to 296) 4. Concurrent compactors: 72 (tried disabling to making it 172) 5. Multithreaded compaction: true (tried both true and false) 6. Compaction strategy: LCS (tried STCS as well) 7. Memtable total space in mb: 4096 MB (tried default and some other params too) Note: I have tried almost all permutation combination of these parameters. *Observations: * I ran test for 1.15 hrs with writes at the rate of 21000 records/sec(total 60GB data during 1.15 hrs). After I stopped the test compaction took additional 1.30 hrs to finish compaction, that reduced the SSTable count from 170 to 17. CPU(24 cores): almost 80% idle during the run JVM: 48G RAM, 8G Heap, (3G to 5G heap used) Pending Writes: sometimes high spikes for small amount of time otherwise pretty flat Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog disks. Please find the iostat of some of 3 different boxes in my cluster. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.11 0.59 0.01 0.00 97.09 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz awaitsvctm %util sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 0.3224.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32 dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.12 0.52 0.01 0.00 97.14 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. dm-2 0.00 0.00 0.08 175.93 10.52 703.74 8.12 3.13 17.78 0. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 1.14 0. dm-4 0.00 0.00 9.19 427.55 1166.91 1710.21 13.18 4.67 10.65 0. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.15 1.13 0.52 0.01 0.00 97.19 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz awaitsvctm %util sda 0.02 429.97 9.28 7.29 1176.81 1749.00 353.12 0.07 4.10 0.55 0.91 sdb 0.00 173.65 0.08 3.09 10.50 706.96 452.25 0.09 27.23 0.99 0.31 dm-0 0.00 0.00 0.04 0.79 0.82 3.16 9.61 0.00 1.54 0.27 0.02 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.68 0.63 0.00 dm-2 0.00 0.00 0.08 176.74 10.50 706.96 8.12 3.46 19.53 0.02 0.31 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.85 0.83 0.00 dm-4 0.00 0.00 9.26 436.46 1175.98 1745.84 13.11 0.03 0.03 0.02 0.89 Thanks, Jay On Thu, Apr 18, 2013 at 2:50 AM, aaron morton aa...@thelastpickle.comwrote: I believe that compaction occurs on the data directories and not in the commitlog. Yes, compaction only works on the data files. When I ran
Re: How to make compaction run faster?
Thanks Aaron, Please find answers to your questions. 1. I started test with default parameters the compaction is backing up. So went for various options. 2. The data is on RAID10. 3. As I watched Disk latency on DSE Opscenter as well as on iostat the await is always 35 to 40 ms for longer period of time during the test. (which probably gives me high write latency on client side) Do you think this could contribute to slowing down the compaction? probably not..! So Aaron, I am trying to understand - You are suggesting to go back to STCS and increase the compaction_throughput step by step to see if compaction catch up with write traffic? Thank you for your inputs. Regards, Jay On Thu, Apr 18, 2013 at 1:52 PM, aaron morton aa...@thelastpickle.comwrote: Parameters used: • SSTable size: 500MB (tried various sizes from 20MB to 1GB) • Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) • Concurrent write: 196 (tried from 32 to 296) • Concurrent compactors: 72 (tried disabling to making it 172) • Multithreaded compaction: true (tried both true and false) • Compaction strategy: LCS (tried STCS as well) • Memtable total space in mb: 4096 MB (tried default and some other params too) I would restore to default settings before I did anything else. Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog disks. Please find the iostat of some of 3 different boxes in my cluster. What is the data on ? It's important to call iostat with a period and watch the await / queue size of time. Not just view a snapshot. I would go back to STS with default settings, and ramp up the throughput until compaction cannot keep up. Then increase the throughout and see how that works. Then increase throughput again and see what happens. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 19/04/2013, at 5:05 AM, Jay Svc jaytechg...@gmail.com wrote: Hi Aaron, Alexis, Thanks for reply, Please find some more details below. Core problems: Compaction is taking longer time to finish. So it will affect my reads. I have more CPU and memory, want to utilize that to speed up the compaction process. Parameters used: • SSTable size: 500MB (tried various sizes from 20MB to 1GB) • Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) • Concurrent write: 196 (tried from 32 to 296) • Concurrent compactors: 72 (tried disabling to making it 172) • Multithreaded compaction: true (tried both true and false) • Compaction strategy: LCS (tried STCS as well) • Memtable total space in mb: 4096 MB (tried default and some other params too) Note: I have tried almost all permutation combination of these parameters. Observations: I ran test for 1.15 hrs with writes at the rate of 21000 records/sec(total 60GB data during 1.15 hrs). After I stopped the test compaction took additional 1.30 hrs to finish compaction, that reduced the SSTable count from 170 to 17. CPU(24 cores): almost 80% idle during the run JVM: 48G RAM, 8G Heap, (3G to 5G heap used) Pending Writes: sometimes high spikes for small amount of time otherwise pretty flat Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog disks. Please find the iostat of some of 3 different boxes in my cluster. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.11 0.59 0.01 0.00 97.09 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 0.32 dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.12 0.52 0.01 0.00 97.14 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. dm-2 0.00 0.00 0.08 175.93 10.52 703.74 8.12 3.13 17.78 0. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 1.14 0. dm-4 0.00 0.00 9.19 427.55 1166.91 1710.21 13.18 4.67 10.65 0. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-1) 04/18/2013
How to make compaction run faster?
Hi Team, I have a high write traffic to my Cassandra cluster. I experience a very high number of pending compactions. As I expect higher writes, The pending compactions keep increasing. Even when I stop my writes it takes several hours to finishing pending compactions. My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. How can I increase the compaction rate so it will run bit faster to match my write speed? Your inputs are appreciated. Thanks, Jay
Re: Does Memtable resides in Heap?
Thanks Edward.! On Fri, Apr 12, 2013 at 9:46 AM, Edward Capriolo edlinuxg...@gmail.comwrote: This issue describes the design of the arena allocation of memtabes. https://issues.apache.org/jira/browse/CASSANDRA-2252 On Fri, Apr 12, 2013 at 1:35 AM, Viktor Jevdokimov viktor.jevdoki...@adform.com wrote: Memtables resides in heap, write rate impacts GC, more writes - more frequent and longer ParNew GC pauses. From: Jay Svc [mailto:jaytechg...@gmail.com] Sent: Friday, April 12, 2013 01:03 To: user@cassandra.apache.org Subject: Does Memtable resides in Heap? Hi Team, I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My question is the size of Memtable does it contribute to heap size? or they are part of off-heap? Does bigger Memtable would have impact on GC and overall memory management? I am using DSE 3.0 / Cassandra 1.1.9. Thanks, Jay Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063 Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Re: Does Memtable resides in Heap?
Thanks Vitor, So as per recommendation its only efficient when heap size is below 8GB. How about when we have more RAM, does that rest of the RAM can be left for OS to make use? How about the bloom filter and index samples, are they part of off-heap? Thank you for your response. Regards, Jay On Thu, Apr 11, 2013 at 10:35 PM, Viktor Jevdokimov viktor.jevdoki...@adform.com wrote: Memtables resides in heap, write rate impacts GC, more writes - more frequent and longer ParNew GC pauses. From: Jay Svc [mailto:jaytechg...@gmail.com] Sent: Friday, April 12, 2013 01:03 To: user@cassandra.apache.org Subject: Does Memtable resides in Heap? Hi Team, I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My question is the size of Memtable does it contribute to heap size? or they are part of off-heap? Does bigger Memtable would have impact on GC and overall memory management? I am using DSE 3.0 / Cassandra 1.1.9. Thanks, Jay Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063 Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Does Memtable resides in Heap?
Hi Team, I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My question is the size of Memtable does it contribute to heap size? or they are part of off-heap? Does bigger Memtable would have impact on GC and overall memory management? I am using DSE 3.0 / Cassandra 1.1.9. Thanks, Jay
Re: Insert v/s Update performance
Hi Aaron, I will try to slow down the compaction. Since this is our DEV environment the core problem remains as we just have 2 cores. I will get back in case I see any different situation. Thank you for your inputs. Thanks, Jay On Sun, Mar 31, 2013 at 5:35 AM, aaron morton aa...@thelastpickle.comwrote: How this parameter works? I have 3 nodes and 2 core each CPU and I have higher writes. It slows down the rate that compaction reads from disk. It reads at bit then has to take a break and wait until it can read again. With only 2 cores you will be running into issues when compaction or repair do their work. So usually for high update and high read situation what parameter we should consider for tuning? In this case I think the issue is only having 2 cores. There are background processing like compaction and repair that have to run when you system is running. Slowing down compaction will reduce it's impact. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 30/03/2013, at 12:58 AM, Jay Svc jaytechg...@gmail.com wrote: Hi Aaron, Thank you for your input. I have been monitoring my GC activities and looking at my Heap, it shows pretty linear activities, without any spikes. When I look at CPU it shows higher utilization while during writes alone. I also expect hevy read traffic. When I tried compaction_throughput_* parameter, I obsered that higher number here in my case gets better CPU utilization and keeps pending compactions pretty low. How this parameter works? I have 3 nodes and 2 core each CPU and I have higher writes. So usually for high update and high read situation what parameter we should consider for tuning? Thanks, Jay On Wed, Mar 27, 2013 at 9:55 PM, aaron morton aa...@thelastpickle.com wrote: * Check for GC activity in the logs * check the volume the commit log is on to see it it's over utilised. * check if the dropped messages correlate to compaction, look at the compaction_* settings in yaml and consider reducing the throughput. Like Dean says if you have existing data it will result in more compaction. You may be able to get a lot of writes through in a clean new cluster, but it also has to work when compaction and repair are running. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 27/03/2013, at 1:43 PM, Jay Svc jaytechg...@gmail.com wrote: Thanks Dean again! My use case is high number of reads and writes out of that I am just focusing on write now. I thought LCS is a suitable for my situation. I tried simillar on STCS and results are same. I ran nodetool for tpstats and MutationStage pending are very high. At the same time the SSTable count and Pending Compaction are high too during my updates. Please find the snapshot of my syslog. INFO [ScheduledTasks:1] 2013-03-26 15:05:48,560 StatusLogger.java (line 116) OpsCenter.rollups864000,0 INFO [FlushWriter:55] 2013-03-26 15:05:48,608 Memtable.java (line 264) Writing Memtable-InventoryPrice@1051586614(11438914/129587272 serialized/live bytes, 404320 ops) INFO [ScheduledTasks:1] 2013-03-26 15:05:53,561 MessagingService.java (line 658) 2701 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2013-03-26 15:05:53,562 StatusLogger.java (line 57) Pool NameActive Pending Blocked INFO [ScheduledTasks:1] 2013-03-26 15:05:53,563 StatusLogger.java (line 72) ReadStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,568 StatusLogger.java (line 72) RequestResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) MutationStage32 19967 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MigrationStage0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MemtablePostFlusher 1 1 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) FlushWriter
Re: Insert v/s Update performance
Hi Aaron, Thank you for your input. I have been monitoring my GC activities and looking at my Heap, it shows pretty linear activities, without any spikes. When I look at CPU it shows higher utilization while during writes alone. I also expect hevy read traffic. When I tried compaction_throughput_* parameter, I obsered that higher number here in my case gets better CPU utilization and keeps pending compactions pretty low. How this parameter works? I have 3 nodes and 2 core each CPU and I have higher writes. So usually for high *update* and high read situation what parameter we should consider for tuning? Thanks, Jay On Wed, Mar 27, 2013 at 9:55 PM, aaron morton aa...@thelastpickle.comwrote: * Check for GC activity in the logs * check the volume the commit log is on to see it it's over utilised. * check if the dropped messages correlate to compaction, look at the compaction_* settings in yaml and consider reducing the throughput. Like Dean says if you have existing data it will result in more compaction. You may be able to get a lot of writes through in a clean new cluster, but it also has to work when compaction and repair are running. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 27/03/2013, at 1:43 PM, Jay Svc jaytechg...@gmail.com wrote: Thanks Dean again! My use case is high number of reads and writes out of that I am just focusing on write now. I thought LCS is a suitable for my situation. I tried simillar on STCS and results are same. I ran nodetool for tpstats and MutationStage pending are very high. At the same time the SSTable count and Pending Compaction are high too during my updates. Please find the snapshot of my syslog. INFO [ScheduledTasks:1] 2013-03-26 15:05:48,560 StatusLogger.java (line 116) OpsCenter.rollups864000,0 INFO [FlushWriter:55] 2013-03-26 15:05:48,608 Memtable.java (line 264) Writing Memtable-InventoryPrice@1051586614(11438914/129587272 serialized/live bytes, 404320 ops) INFO [ScheduledTasks:1] 2013-03-26 15:05:53,561 MessagingService.java (line 658) 2701 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2013-03-26 15:05:53,562 StatusLogger.java (line 57) Pool NameActive Pending Blocked INFO [ScheduledTasks:1] 2013-03-26 15:05:53,563 StatusLogger.java (line 72) ReadStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,568 StatusLogger.java (line 72) RequestResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) MutationStage32 19967 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MigrationStage0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MemtablePostFlusher 1 1 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) FlushWriter 1 1 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) commitlog_archiver0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 72) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 72) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 77) CompactionManager 127 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,675 StatusLogger.java (line 89) MessagingServicen/a 0,22 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,724 StatusLogger.java (line 99) Cache Type Size Capacity KeysToSave Provider INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line 100) KeyCache 142315 2118997 all INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725
Insert v/s Update performance
Hi Team, I have this 3 node cluster. I am writing data to these node at the rate of 2,000 records/second. What I observed that if I do inserts. (Means records for those keys does not exist, my column family has 0 records to start with) then I have better write performacne, low SSTable count, low pending compaction and write latency is acceptable and CPU utilization on each node between 35% to 85%. When I ran same test but for update this time (means records already exists in Column family with same key), I observed that my SSTable count gone high 3 times. Pending compactions gone high more than 2 times and write latency has gone high too and CPU utilization was almost 92% to 100%. What is a reason of deteriorating Update performance v/s Insert performance. Since this is critical you help is highly appriciated. P.S. I also observed that high number of pending Mutation Stage on my nodetool tpstats. Thanks, Jay
Re: Insert v/s Update performance
Thanks Dean, I have used LCS with sstable_size_in_mb of 15. I have also tried bigger sstable_size_in_mb and observed simillar behavior. Does compaction works differently for update v/s Insert? I belive all keys goes to single SST. What other options I should look into? Thanks, Jay On Tue, Mar 26, 2013 at 6:18 PM, Hiller, Dean dean.hil...@nrel.gov wrote: Most likely compaction kicks in as updates cause duplicated rows in STCS and compaction causes load that may not have been there before(check your logs). Also, you can increase the number of nodes in your cluster as well to better handle the load. Later, Dean From: Jay Svc jaytechg...@gmail.commailto:jaytechg...@gmail.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Tuesday, March 26, 2013 5:05 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Insert v/s Update performance Hi Team, I have this 3 node cluster. I am writing data to these node at the rate of 2,000 records/second. What I observed that if I do inserts. (Means records for those keys does not exist, my column family has 0 records to start with) then I have better write performacne, low SSTable count, low pending compaction and write latency is acceptable and CPU utilization on each node between 35% to 85%. When I ran same test but for update this time (means records already exists in Column family with same key), I observed that my SSTable count gone high 3 times. Pending compactions gone high more than 2 times and write latency has gone high too and CPU utilization was almost 92% to 100%. What is a reason of deteriorating Update performance v/s Insert performance. Since this is critical you help is highly appriciated. P.S. I also observed that high number of pending Mutation Stage on my nodetool tpstats. Thanks, Jay
Re: Insert v/s Update performance
Thanks Dean again! My use case is high number of reads and writes out of that I am just focusing on write now. I thought LCS is a suitable for my situation. I tried simillar on STCS and results are same. I ran nodetool for tpstats and MutationStage pending are very high. At the same time the SSTable count and Pending Compaction are high too during my updates. Please find the snapshot of my syslog. INFO [ScheduledTasks:1] 2013-03-26 15:05:48,560 StatusLogger.java (line 116) OpsCenter.rollups864000,0 INFO [FlushWriter:55] 2013-03-26 15:05:48,608 Memtable.java (line 264) Writing Memtable-InventoryPrice@1051586614(11438914/129587272 serialized/live bytes, 404320 ops) INFO [ScheduledTasks:1] 2013-03-26 15:05:53,561 MessagingService.java (line 658) 2701 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2013-03-26 15:05:53,562 StatusLogger.java (line 57) Pool NameActive Pending Blocked INFO [ScheduledTasks:1] 2013-03-26 15:05:53,563 StatusLogger.java (line 72) ReadStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,568 StatusLogger.java (line 72) RequestResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) MutationStage 32 19967 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MigrationStage0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MemtablePostFlusher 1 1 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) FlushWriter 1 1 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) commitlog_archiver0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 72) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 72) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 77) CompactionManager 127 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,675 StatusLogger.java (line 89) MessagingServicen/a 0,22 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,724 StatusLogger.java (line 99) Cache Type Size Capacity KeysToSave Provider INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line 100) KeyCache 142315 2118997 all INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line 106) RowCache 0 0 all org.apache.cassandra.cache.SerializingCacheProvider INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line 113) ColumnFamilyMemtable ops,data INFO [ScheduledTasks:1] 2013-03-26 15:0 Thanks, Jay On Tue, Mar 26, 2013 at 7:15 PM, Hiller, Dean dean.hil...@nrel.gov wrote: LCS is generally used for high read vs. write ratio though it sounds like you may be doing a heavy write load instead. LCS will involve more compactions as you write to the system compared to STCS because LCS is always trying to keep a 1 to 10 ratio between levels. While LCS will involve more compaction in general(more I/o, more cpu), I am not sure on update vs. insert though From what I understand STCS will happily duplicate rows across SS tables while LCS does not like to do this so as you update you will constantly compact….well, that is my understanding. Have you tried STCS out at all? (ps. This is just from what I understand so take with a grain of salt). Also, there are some great tools in the nodetool tool as well so you can get nodetool compactionstats, etc. etc. and see how backlogged you are in pending tasks….how many pending? Later, Dean From: Jay Svc jaytechg...@gmail.commailto:jaytechg...@gmail.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user
conditional update or insert
Hi All, On each row I have a column which maintains the timestamp like lastUpdated etc. While inserting such row I want to make sure that the row should be only updated if the lastUpdated is older than the new one I am inserting. One way to do this is - Read the record first check the timestamp if newer is latest then update. Since I have higher volume of read and writes load. This additional read will add to it. Any alternative to achieve this? Thanks, Jay
Concurrent write performance
Folks, I would like to write(insert or update) to a single row in a column family. I have concurrent requests which will write to a single row. Do we see any performance implications because of concurrent writes to a single row where comparator has to sort the columns at the same time? Please share your thoughts. Thanks, Jay
High Read and write through put
Folks, For given situation I am expecting multiple read and write request to a same cluster. What are primary design or configuration consideration we should make? Any thoughts or links to such documentation is appreciated. Thanks, Jay
