[jira] [Comment Edited] (CASSANDRA-8192) Better error logging on corrupt compressed SSTables: currently AssertionError in Memory.java
[ https://issues.apache.org/jira/browse/CASSANDRA-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256721#comment-14256721 ] Marcus Eriksson edited comment on CASSANDRA-8192 at 12/23/14 7:57 AM: -- I need to create a file like this: {code} DataOutputStream dos = new DataOutputStream(new FileOutputStream("/tmp/comp_info")); dos.writeUTF("LZ4Compressor"); // compressor name dos.writeInt(0);// option count dos.writeInt(0);//chunk length dos.writeLong(0); // data length dos.writeInt(0); // chunk count dos.close(); {code} to reproduce (dunno how common it is for file corruption to create files like that (ie, being zeroed out after a number of bytes)?) Anyway, could we add an assert where we serialize the CompressionInfo to disk to make sure we don't have a bug there? And, if it is corrupt data, we should probably check if chunkCount <= 0? was (Author: krummas): I need to create a file like this: {code} DataOutputStream dos = new DataOutputStream(new FileOutputStream("/tmp/comp_info")); dos.writeUTF("LZ4Compressor"); // compressor name dos.writeInt(0);// option count dos.writeInt(0);//chunk length dos.writeLong(0); // data length dos.writeInt(0); // chunk count dos.close(); {code} to reproduce (dunno how common it is for file corruption to create files like that?) Anyway, could we add an assert where we serialize the CompressionInfo to disk to make sure we don't have a bug there? And, if it is corrupt data, we should probably check if chunkCount <= 0? > Better error logging on corrupt compressed SSTables: currently AssertionError > in Memory.java > > > Key: CASSANDRA-8192 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8192 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: Windows-7-32 bit, 3GB RAM, Java 1.7.0_67 >Reporter: Andreas Schnitzerling >Assignee: Joshua McKenzie >Priority: Minor > Fix For: 2.1.3 > > Attachments: 8192_v1.txt, cassandra.bat, cassandra.yaml, > logdata-onlinedata-ka-196504-CompressionInfo.zip, printChunkOffsetErrors.txt, > system-compactions_in_progress-ka-47594-CompressionInfo.zip, > system-sstable_activity-jb-25-Filter.zip, system.log, system_AssertionTest.log > > > Since update of 1 of 12 nodes from 2.1.0-rel to 2.1.1-rel Exception during > start up. > {panel:title=system.log} > ERROR [SSTableBatchOpen:1] 2014-10-27 09:44:00,079 CassandraDaemon.java:153 - > Exception in thread Thread[SSTableBatchOpen:1,5,main] > java.lang.AssertionError: null > at org.apache.cassandra.io.util.Memory.size(Memory.java:307) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:135) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > ~[na:1.7.0_55] > at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > [na:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > [na:1.7.0_55] > at java.lang.Thread.run(Unknown Source) [na:1.7.0_55] > {panel} > In the attached log you can still see as well CASSANDRA-8069 and > CASSANDRA-6283. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8192) Better error logging on corrupt compressed SSTables: currently AssertionError in Memory.java
[ https://issues.apache.org/jira/browse/CASSANDRA-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-8192: --- Reviewer: Marcus Eriksson > Better error logging on corrupt compressed SSTables: currently AssertionError > in Memory.java > > > Key: CASSANDRA-8192 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8192 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: Windows-7-32 bit, 3GB RAM, Java 1.7.0_67 >Reporter: Andreas Schnitzerling >Assignee: Joshua McKenzie >Priority: Minor > Fix For: 2.1.3 > > Attachments: 8192_v1.txt, cassandra.bat, cassandra.yaml, > logdata-onlinedata-ka-196504-CompressionInfo.zip, printChunkOffsetErrors.txt, > system-compactions_in_progress-ka-47594-CompressionInfo.zip, > system-sstable_activity-jb-25-Filter.zip, system.log, system_AssertionTest.log > > > Since update of 1 of 12 nodes from 2.1.0-rel to 2.1.1-rel Exception during > start up. > {panel:title=system.log} > ERROR [SSTableBatchOpen:1] 2014-10-27 09:44:00,079 CassandraDaemon.java:153 - > Exception in thread Thread[SSTableBatchOpen:1,5,main] > java.lang.AssertionError: null > at org.apache.cassandra.io.util.Memory.size(Memory.java:307) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:135) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > ~[na:1.7.0_55] > at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > [na:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > [na:1.7.0_55] > at java.lang.Thread.run(Unknown Source) [na:1.7.0_55] > {panel} > In the attached log you can still see as well CASSANDRA-8069 and > CASSANDRA-6283. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8192) Better error logging on corrupt compressed SSTables: currently AssertionError in Memory.java
[ https://issues.apache.org/jira/browse/CASSANDRA-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256721#comment-14256721 ] Marcus Eriksson commented on CASSANDRA-8192: I need to create a file like this: {code} DataOutputStream dos = new DataOutputStream(new FileOutputStream("/tmp/comp_info")); dos.writeUTF("LZ4Compressor"); // compressor name dos.writeInt(0);// option count dos.writeInt(0);//chunk length dos.writeLong(0); // data length dos.writeInt(0); // chunk count dos.close(); {code} to reproduce (dunno how common it is for file corruption to create files like that?) Anyway, could we add an assert where we serialize the CompressionInfo to disk to make sure we don't have a bug there? And, if it is corrupt data, we should probably check if chunkCount <= 0? > Better error logging on corrupt compressed SSTables: currently AssertionError > in Memory.java > > > Key: CASSANDRA-8192 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8192 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: Windows-7-32 bit, 3GB RAM, Java 1.7.0_67 >Reporter: Andreas Schnitzerling >Assignee: Joshua McKenzie >Priority: Minor > Fix For: 2.1.3 > > Attachments: 8192_v1.txt, cassandra.bat, cassandra.yaml, > logdata-onlinedata-ka-196504-CompressionInfo.zip, printChunkOffsetErrors.txt, > system-compactions_in_progress-ka-47594-CompressionInfo.zip, > system-sstable_activity-jb-25-Filter.zip, system.log, system_AssertionTest.log > > > Since update of 1 of 12 nodes from 2.1.0-rel to 2.1.1-rel Exception during > start up. > {panel:title=system.log} > ERROR [SSTableBatchOpen:1] 2014-10-27 09:44:00,079 CassandraDaemon.java:153 - > Exception in thread Thread[SSTableBatchOpen:1,5,main] > java.lang.AssertionError: null > at org.apache.cassandra.io.util.Memory.size(Memory.java:307) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:135) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > ~[na:1.7.0_55] > at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > [na:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > [na:1.7.0_55] > at java.lang.Thread.run(Unknown Source) [na:1.7.0_55] > {panel} > In the attached log you can still see as well CASSANDRA-8069 and > CASSANDRA-6283. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Ren updated CASSANDRA-8518: - Description: We have been suffering from cassandra node crash due to out of memory for a long time. The heap dump from the recent crash shows there are 22 native transport request threads each of which consumes 3.3% of heap size, taking more than 70% in total. Heap dump: !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! Expanded view of one thread: !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor as the request executor and provided a default request size estimator which constantly returns 1, meaning it limits only the number of requests being pushed to the pool. To have more fine-grained control on handling requests and better protect our node from OOM issue, we propose implementing a more precise estimator. Here is our two cents: For update/delete/insert request: Size could be estimated by adding size of all class members together. For scan query, the major part of the request is response, which can be estimated from the history data. For example if we receive a scan query on a column family for a certain token range, we keep track of its response size used as the estimated response size for later scan query on the same cf. For future requests on the same cf, response size could be calculated by token range*recorded size/ recorded token range. The request size should be estimated as (query size + estimated response size). We believe what we're proposing here can be useful for other people in the Cassandra community as well. Would you mind providing us feedbacks? Please let us know if you have any concerns or suggestions regarding this proposal. Thanks, Cheng was: We have been suffering from cassandra node crash due to out of memory for a long time. The heap dump from the recent crash shows there are 22 native transport request threads of queries each of which consumes 3.3% of heap size, taking more than 70% in total. Heap dump: !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! Expanded view of one thread: !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor as the request executor and provided a default request size estimator which constantly returns 1, meaning it limits only the number of requests being pushed to the pool. To have more fine-grained control on handling requests and better protect our node from OOM issue, we propose implementing a more precise estimator. Here is our two cents: For update/delete/insert request: Size could be estimated by adding size of all class members together. For scan query, the major part of the request is response, which can be estimated from the history data. For example if we receive a scan query on a column family for a certain token range, we keep track of its response size used as the estimated response size for later scan query on the same cf. For future requests on the same cf, response size could be calculated by token range*recorded size/ recorded token range. The request size should be estimated as (query size + estimated response size). We believe what we're proposing here can be useful for other people in the Cassandra community as well. Would you mind providing us feedbacks? Please let us know if you have any concerns or suggestions regarding this proposal. Thanks, Cheng > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads each of which consumes 3.3% of heap size, taking > more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we
[jira] [Updated] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Ren updated CASSANDRA-8518: - Priority: Major (was: Minor) > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads of queries each of which consumes 3.3% of heap > size, taking more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor > as the request executor and provided a default request size estimator which > constantly returns 1, meaning it limits only the number of requests being > pushed to the pool. To have more fine-grained control on handling requests > and better protect our node from OOM issue, we propose implementing a more > precise estimator. > Here is our two cents: > For update/delete/insert request: Size could be estimated by adding size of > all class members together. > For scan query, the major part of the request is response, which can be > estimated from the history data. For example if we receive a scan query on a > column family for a certain token range, we keep track of its response size > used as the estimated response size for later scan query on the same cf. > For future requests on the same cf, response size could be calculated by > token range*recorded size/ recorded token range. The request size should be > estimated as (query size + estimated response size). > We believe what we're proposing here can be useful for other people in the > Cassandra community as well. Would you mind providing us feedbacks? Please > let us know if you have any concerns or suggestions regarding this proposal. > Thanks, > Cheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256326#comment-14256326 ] Ariel Weisberg edited comment on CASSANDRA-8457 at 12/22/14 11:48 PM: -- Repeating these benchmarks, trunk can be slower as or more often than it is faster. It's one of those peak performance vs. average and worst case sort of decisions. This is odd. Some measurements running trunk. Soft interrupt hotspot on one CPU. Don't see why that should happen. Interrupt hotspots are usually related to not enough TCP streams and there should be several in this workload. Kind of looks like someone somewhere pinned interrupts to core 0. {noformat} 11:20:04 PM CPU%usr %nice%sys %iowait%irq %soft %steal %guest %idle 11:20:05 PM all 11.190.009.820.000.000.981.27 0.00 76.74 11:20:05 PM0 18.970.00 12.070.000.00 32.761.72 0.00 34.48 11:20:05 PM16.120.00 10.200.000.000.004.08 0.00 79.59 11:20:05 PM27.550.00 11.320.000.000.003.77 0.00 77.36 11:20:05 PM3 11.110.009.260.000.000.001.85 0.00 77.78 11:20:05 PM4 17.860.005.360.000.000.000.00 0.00 76.79 11:20:05 PM5 10.710.008.930.000.000.000.00 0.00 80.36 11:20:05 PM6 10.170.00 10.170.000.000.001.69 0.00 77.97 11:20:05 PM7 10.000.00 11.670.000.000.001.67 0.00 76.67 11:20:05 PM8 13.560.006.780.000.000.001.69 0.00 77.97 11:20:05 PM9 12.500.00 10.940.000.000.001.56 0.00 75.00 11:20:05 PM 10 15.150.009.090.000.000.001.52 0.00 74.24 11:20:05 PM 117.940.00 12.700.000.000.001.59 0.00 77.78 11:20:05 PM 129.520.009.520.000.000.001.59 0.00 79.37 11:20:05 PM 139.090.00 12.120.000.000.000.00 0.00 78.79 11:20:05 PM 14 11.760.00 10.290.000.000.001.47 0.00 76.47 11:20:05 PM 159.230.009.230.000.000.001.54 0.00 80.00 11:20:05 PM 16 10.290.00 14.710.000.000.001.47 0.00 73.53 11:20:05 PM 17 10.450.00 11.940.000.000.000.00 0.00 77.61 11:20:05 PM 18 11.940.008.960.000.000.001.49 0.00 77.61 11:20:05 PM 19 12.680.00 12.680.000.000.001.41 0.00 73.24 11:20:05 PM 20 13.240.008.820.000.000.000.00 0.00 77.94 11:20:05 PM 215.800.00 15.940.000.000.000.00 0.00 78.26 11:20:05 PM 22 13.890.009.720.000.000.001.39 0.00 75.00 11:20:05 PM 239.380.009.380.000.000.000.00 0.00 81.25 11:20:05 PM 247.810.009.380.000.000.001.56 0.00 81.25 11:20:05 PM 258.960.008.960.000.000.001.49 0.00 80.60 11:20:05 PM 26 11.590.00 10.140.000.000.000.00 0.00 78.26 11:20:05 PM 27 13.040.007.250.000.000.001.45 0.00 78.26 11:20:05 PM 287.810.007.810.000.000.000.00 0.00 84.38 11:20:05 PM 29 11.760.008.820.000.000.000.00 0.00 79.41 11:20:05 PM 30 11.430.00 10.000.000.000.001.43 0.00 77.14 11:20:05 PM 31 12.500.004.690.000.000.000.00 0.00 82.81 {noformat} Modified mpstat output is similar. perf stat doesn't have access to performance counters in these instances. I'll have to see if I can get instances that do that. I have a flight recording of each, but not a flight recording of great performance on trunk. was (Author: aweisberg): Repeating these benchmarks, trunk can be slower as or more often then it is faster. It's one of those peak performance vs. average and worst case sort of decisions. This is odd. Some measurements running trunk. Soft interrupt hotspot on one CPU. Don't see why that should happen. Interrupt hotspots are usually related to not enough TCP streams and there should be several in this workload. Kind of looks like someone somewhere pinned interrupts to core 0. {noformat} 11:20:04 PM CPU%usr %nice%sys %iowait%irq %soft %steal %guest %idle 11:20:05 PM all 11.190.009.820.000.000.981.27 0.00 76.74 11:20:05 PM0 18.970.00 12.070.000.00 32.761.72 0.00 34.48 11:20:05 PM16.120.00 10.200.000.000.004.08 0.00 79.59 11:20:05
[jira] [Commented] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256326#comment-14256326 ] Ariel Weisberg commented on CASSANDRA-8457: --- Repeating these benchmarks, trunk can be slower as or more often then it is faster. It's one of those peak performance vs. average and worst case sort of decisions. This is odd. Some measurements running trunk. Soft interrupt hotspot on one CPU. Don't see why that should happen. Interrupt hotspots are usually related to not enough TCP streams and there should be several in this workload. Kind of looks like someone somewhere pinned interrupts to core 0. {noformat} 11:20:04 PM CPU%usr %nice%sys %iowait%irq %soft %steal %guest %idle 11:20:05 PM all 11.190.009.820.000.000.981.27 0.00 76.74 11:20:05 PM0 18.970.00 12.070.000.00 32.761.72 0.00 34.48 11:20:05 PM16.120.00 10.200.000.000.004.08 0.00 79.59 11:20:05 PM27.550.00 11.320.000.000.003.77 0.00 77.36 11:20:05 PM3 11.110.009.260.000.000.001.85 0.00 77.78 11:20:05 PM4 17.860.005.360.000.000.000.00 0.00 76.79 11:20:05 PM5 10.710.008.930.000.000.000.00 0.00 80.36 11:20:05 PM6 10.170.00 10.170.000.000.001.69 0.00 77.97 11:20:05 PM7 10.000.00 11.670.000.000.001.67 0.00 76.67 11:20:05 PM8 13.560.006.780.000.000.001.69 0.00 77.97 11:20:05 PM9 12.500.00 10.940.000.000.001.56 0.00 75.00 11:20:05 PM 10 15.150.009.090.000.000.001.52 0.00 74.24 11:20:05 PM 117.940.00 12.700.000.000.001.59 0.00 77.78 11:20:05 PM 129.520.009.520.000.000.001.59 0.00 79.37 11:20:05 PM 139.090.00 12.120.000.000.000.00 0.00 78.79 11:20:05 PM 14 11.760.00 10.290.000.000.001.47 0.00 76.47 11:20:05 PM 159.230.009.230.000.000.001.54 0.00 80.00 11:20:05 PM 16 10.290.00 14.710.000.000.001.47 0.00 73.53 11:20:05 PM 17 10.450.00 11.940.000.000.000.00 0.00 77.61 11:20:05 PM 18 11.940.008.960.000.000.001.49 0.00 77.61 11:20:05 PM 19 12.680.00 12.680.000.000.001.41 0.00 73.24 11:20:05 PM 20 13.240.008.820.000.000.000.00 0.00 77.94 11:20:05 PM 215.800.00 15.940.000.000.000.00 0.00 78.26 11:20:05 PM 22 13.890.009.720.000.000.001.39 0.00 75.00 11:20:05 PM 239.380.009.380.000.000.000.00 0.00 81.25 11:20:05 PM 247.810.009.380.000.000.001.56 0.00 81.25 11:20:05 PM 258.960.008.960.000.000.001.49 0.00 80.60 11:20:05 PM 26 11.590.00 10.140.000.000.000.00 0.00 78.26 11:20:05 PM 27 13.040.007.250.000.000.001.45 0.00 78.26 11:20:05 PM 287.810.007.810.000.000.000.00 0.00 84.38 11:20:05 PM 29 11.760.008.820.000.000.000.00 0.00 79.41 11:20:05 PM 30 11.430.00 10.000.000.000.001.43 0.00 77.14 11:20:05 PM 31 12.500.004.690.000.000.000.00 0.00 82.81 {noformat} Modified mpstat output is similar. perf stat doesn't have access to performance counters in these instances. I'll have to see if I can get instances that do that. I have a flight recording of each, but not a flight recording of great performance on trunk. > nio MessagingService > > > Key: CASSANDRA-8457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8457 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Ariel Weisberg > Labels: performance > Fix For: 3.0 > > > Thread-per-peer (actually two each incoming and outbound) is a big > contributor to context switching, especially for larger clusters. Let's look > at switching to nio, possibly via Netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256293#comment-14256293 ] Ariel Weisberg commented on CASSANDRA-8457: --- Testing on AWS with 6 client and 9 servers. c3.8xlarge running Ubuntu 12.04 and no placement groups. I changed the workload to run RF=5 with the same data set size and increased the number of operations so the test runs longer. All tests were run on the same set of instances without restarting the instances. Numbers for trunk are all over the map and range from fantastically faster to slower. The modified version was more consistent but slower. When trunk was faster it had better CPU utilization. There is a correlation between low CPU utilization (not 1600% on a 16 core 32 thread server) and lower throughput. It kind of suggests there is some kind of starvation or contention preventing tasks from flowing. ||Run|Trunk|Modified|| |#1| 147925|119058| |#2|101740| 139155| |#3| 197265|147973| |#4|105774|#| Trunk performance is not ideal if only from a consistency perspective. I am going to try and get the performance counters for cache misses and such from perf as well as a profile tomorrow. > nio MessagingService > > > Key: CASSANDRA-8457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8457 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Ariel Weisberg > Labels: performance > Fix For: 3.0 > > > Thread-per-peer (actually two each incoming and outbound) is a big > contributor to context switching, especially for larger clusters. Let's look > at switching to nio, possibly via Netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8399) Reference Counter exception when dropping user type
[ https://issues.apache.org/jira/browse/CASSANDRA-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256225#comment-14256225 ] Benedict commented on CASSANDRA-8399: - The real question is: why are we even trying to obtain a reference here? Creating and using a scanner is a protected operation; it can be protected by either reference counting or the OpOrder. But this should be ensured by the operation that creates the scanner, not the scanner itself. We are leaking resource management into the scanner code that shouldn't be there. We should remove the reference counting all together from this area of code. The fact that we were blindly acquiring before means that we knew it had to succeed, meaning that it also was not necessary. Having two mechanisms for resource management is fine, so long as we don't mix them. A macro operation needs to be protected by one or the other, and sub-operations should rely on these macro operations to ensure their protection. > Reference Counter exception when dropping user type > --- > > Key: CASSANDRA-8399 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8399 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Joshua McKenzie > Fix For: 2.1.3 > > Attachments: 8399_fix_empty_results.txt, node2.log, ubuntu-8399.log > > > When running the dtest > {{user_types_test.py:TestUserTypes.test_type_keyspace_permission_isolation}} > with the current 2.1-HEAD code, very frequently, but not always, when > dropping a type, the following exception is seen:{code} > ERROR [MigrationStage:1] 2014-12-01 13:54:54,824 CassandraDaemon.java:170 - > Exception in thread Thread[MigrationStage:1,5,main] > java.lang.AssertionError: Reference counter -1 for > /var/folders/v3/z4wf_34n1q506_xjdy49gb78gn/T/dtest-eW2RXj/test/node2/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-sche > ma_keyspaces-ka-14-Data.db > at > org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1662) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableScanner.close(SSTableScanner.java:164) > ~[main/:na] > at > org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore$8.close(ColumnFamilyStore.java:1943) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:2116) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:2029) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1963) > ~[main/:na] > at > org.apache.cassandra.db.SystemKeyspace.serializedSchema(SystemKeyspace.java:744) > ~[main/:na] > at > org.apache.cassandra.db.SystemKeyspace.serializedSchema(SystemKeyspace.java:731) > ~[main/:na] > at org.apache.cassandra.config.Schema.updateVersion(Schema.java:374) > ~[main/:na] > at > org.apache.cassandra.config.Schema.updateVersionAndAnnounce(Schema.java:399) > ~[main/:na] > at > org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:167) > ~[main/:na] > at > org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) > ~[main/:na] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[main/:na] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_67] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_67] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_67] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_67] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]{code} > Log of the node with the error is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8399) Reference Counter exception when dropping user type
[ https://issues.apache.org/jira/browse/CASSANDRA-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joshua McKenzie updated CASSANDRA-8399: --- Attachment: 8399_fix_empty_results.txt Good catch; the role of the readOrdering in CFS w/regards to deletion of sstables wasn't clear to me prior and the possibility of a race between OpOrder start and sstable ref count decrement was subtle enough that I missed it. Attached a patch that replicates CASSANDRA-8513 for current 2.1. Ultimately, it feels like the right solution is to untangle the various getScanner calls in SSTR and have a separate getReadScanner and getCompactionScanner path, one without reference counting and one with. This current workaround loses the return of an empty SSTableScanner on the compaction path and also enforces ref counting on range queries which is not ideal. I'll create another ticket for these changes. I don't like us having two separate paradigms for resource tracking / protection on SSTableReaders (OpOrder and ref counting) as mentally modelling state and potential races becomes considerably more difficult with different primitives used on different paths. > Reference Counter exception when dropping user type > --- > > Key: CASSANDRA-8399 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8399 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Joshua McKenzie > Fix For: 2.1.3 > > Attachments: 8399_fix_empty_results.txt, node2.log, ubuntu-8399.log > > > When running the dtest > {{user_types_test.py:TestUserTypes.test_type_keyspace_permission_isolation}} > with the current 2.1-HEAD code, very frequently, but not always, when > dropping a type, the following exception is seen:{code} > ERROR [MigrationStage:1] 2014-12-01 13:54:54,824 CassandraDaemon.java:170 - > Exception in thread Thread[MigrationStage:1,5,main] > java.lang.AssertionError: Reference counter -1 for > /var/folders/v3/z4wf_34n1q506_xjdy49gb78gn/T/dtest-eW2RXj/test/node2/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-sche > ma_keyspaces-ka-14-Data.db > at > org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1662) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableScanner.close(SSTableScanner.java:164) > ~[main/:na] > at > org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore$8.close(ColumnFamilyStore.java:1943) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:2116) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:2029) > ~[main/:na] > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1963) > ~[main/:na] > at > org.apache.cassandra.db.SystemKeyspace.serializedSchema(SystemKeyspace.java:744) > ~[main/:na] > at > org.apache.cassandra.db.SystemKeyspace.serializedSchema(SystemKeyspace.java:731) > ~[main/:na] > at org.apache.cassandra.config.Schema.updateVersion(Schema.java:374) > ~[main/:na] > at > org.apache.cassandra.config.Schema.updateVersionAndAnnounce(Schema.java:399) > ~[main/:na] > at > org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:167) > ~[main/:na] > at > org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) > ~[main/:na] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[main/:na] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_67] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_67] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_67] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_67] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]{code} > Log of the node with the error is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8464) Support direct buffer decompression for reads
[ https://issues.apache.org/jira/browse/CASSANDRA-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-8464: -- Attachment: compression_direct.png Updated stress performance chart !compression_direct.png! > Support direct buffer decompression for reads > - > > Key: CASSANDRA-8464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8464 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Labels: performance > Fix For: 3.0 > > Attachments: compression_direct.png > > > Currently when we read a compressed sstable we copy the data on heap then > send it to be de-compressed to another on heap buffer (albeit pooled). > But now both snappy and lz4 (with CASSANDRA-7039) allow decompression of > direct byte buffers. This lets us mmap the data and decompress completely > off heap (and avoids moving bytes over JNI). > One issue is performing the checksum offheap but the Adler32 does support in > java 8 (it's also in java 7 but marked private?!) > This change yields a > 10% boost in read performance on cstar. Locally I see > upto 30% improvement. > http://cstar.datastax.com/graph?stats=5ebcdd70-816b-11e4-aed6-42010af0688f&metric=op_rate&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=200.09&ymin=0&ymax=135908.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8530) Query on a secondary index creates huge CPU spike + unable to trace
[ https://issues.apache.org/jira/browse/CASSANDRA-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255898#comment-14255898 ] Pavel Baranov commented on CASSANDRA-8530: -- "What I get from this is that the problem was not triggered by the upgrade, but the upgrade did not fix it" - correct As far as "performance over time" goes - we are running this script on monthly bases so I cannot speak for day by day, but month by month all those queries were running in subsecond range. And, one thing to note is that the script is multithreaded, once the issue appeared I cannot even run it with 1 thread. > Query on a secondary index creates huge CPU spike + unable to trace > --- > > Key: CASSANDRA-8530 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8530 > Project: Cassandra > Issue Type: Bug > Components: API, Core > Environment: CentOs 6.5 / Cassandra 2.1.2 >Reporter: Pavel Baranov > > After upgrading cassandra from 2.0.10 to 2.1.2 we are having all kinds of > issues, especially with performance. > java version "1.7.0_65" > Table creation: > {noformat} > tweets> desc table tweets; > CREATE TABLE tweets.tweets ( > uname text, > tweet_id bigint, > tweet text, > tweet_date timestamp, > tweet_date_only text, > uid bigint, > PRIMARY KEY (uname, tweet_id) > ) WITH CLUSTERING ORDER BY (tweet_id ASC) > AND bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '10', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.0 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.1 > AND speculative_retry = '99.0PERCENTILE'; > CREATE INDEX tweets_tweet_date_only_idx ON tweets.tweets (tweet_date_only); > CREATE INDEX tweets_uid ON tweets.tweets (uid); > {noformat} > With Cassandra 2.0.10 this query: > {noformat} > select uname from tweets where uid = 636732672 limit 1; > {noformat} > did not have any issues. After upgrade, I can see the cpu spikes and load avg > goes from ~1 to ~13, especially if I execute the query over and over again. > Doing "tracing on" does not work and just returns: > "Statement trace did not complete within 10 seconds" > I've done: > nodetool upgradesstables > recreated indexes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8530) Query on a secondary index creates huge CPU spike + unable to trace
[ https://issues.apache.org/jira/browse/CASSANDRA-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255892#comment-14255892 ] Sylvain Lebresne commented on CASSANDRA-8530: - bq. 3. until recently we were using a python script that does very simple selects \[...\] All of a sudden, same queries and same process started giving the following message \[...\] and accompanied by huge cpu spikes (load avg went from ~2 to about ~15) bq. 4. We figured maybe it's time to upgrade, so we did to 2.1.2 - did not help. What I get from this is that the problem was *not* triggered by the upgrade, but the upgrade did not fix it. Which infortunately makes it a little less easy to narrow down. Are you sure this really happened "All of a sudden"? Namely, would it be possible that the query performance degraded over time, but you only started noticing once it started throwing you the error message (i.e. do you actually have monitoring that shows how fast those queries were before they started timing out?). > Query on a secondary index creates huge CPU spike + unable to trace > --- > > Key: CASSANDRA-8530 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8530 > Project: Cassandra > Issue Type: Bug > Components: API, Core > Environment: CentOs 6.5 / Cassandra 2.1.2 >Reporter: Pavel Baranov > > After upgrading cassandra from 2.0.10 to 2.1.2 we are having all kinds of > issues, especially with performance. > java version "1.7.0_65" > Table creation: > {noformat} > tweets> desc table tweets; > CREATE TABLE tweets.tweets ( > uname text, > tweet_id bigint, > tweet text, > tweet_date timestamp, > tweet_date_only text, > uid bigint, > PRIMARY KEY (uname, tweet_id) > ) WITH CLUSTERING ORDER BY (tweet_id ASC) > AND bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '10', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.0 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.1 > AND speculative_retry = '99.0PERCENTILE'; > CREATE INDEX tweets_tweet_date_only_idx ON tweets.tweets (tweet_date_only); > CREATE INDEX tweets_uid ON tweets.tweets (uid); > {noformat} > With Cassandra 2.0.10 this query: > {noformat} > select uname from tweets where uid = 636732672 limit 1; > {noformat} > did not have any issues. After upgrade, I can see the cpu spikes and load avg > goes from ~1 to ~13, especially if I execute the query over and over again. > Doing "tracing on" does not work and just returns: > "Statement trace did not complete within 10 seconds" > I've done: > nodetool upgradesstables > recreated indexes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8530) Query on a secondary index creates huge CPU spike + unable to trace
[ https://issues.apache.org/jira/browse/CASSANDRA-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255884#comment-14255884 ] Pavel Baranov commented on CASSANDRA-8530: -- Providing a self-contained script would be tough right now. I can describe what we've done so far: 1. tweets table was create under Cassandra 2.0.10 2. currently there are 6 nodes in the cluster, total storage usage is 3.3TB with an avg. data size of 401GB per node. 3. until recently we were using a python script that does very simple selects (on secondary index, like I've posted in the original ticket) and also deletes based on primary key (this was done once a month, and on a small volume, nothing intense). All of a sudden, same queries and same process started giving the following message: {noformat} code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} {noformat} and accompanied by huge cpu spikes (load avg went from ~2 to about ~15) 4. We figured maybe it's time to upgrade, so we did to 2.1.2 - did not help. 5. after upgrading, the secondary indexes didn't work at all, so I ran "nodetool sstableupgrade" which fixed it but the issue remained. 6. tried removing and reCreating indexes - didn't help 7. did couple of rolling cluster restarts - didn't help 8. unable to see the trace info with "tracing on", however, tracing info appears on queries on primary index or without WHERE clause at all. 9. insertions are still coming in and do not trigger any cpu spikes. The only this that shows up in the system.log is: {noformat} WARN [SharedPool-Worker-1] 2014-12-22 08:07:54,953 BatchStatement.java:255 - Batch of prepared statements for [tweets.tweets] is of size 35434, exceeding specified threshold of 5120 by 30314. {noformat} and that's not even after running the "bad" queries, so I don't even know where else to look for abnormalities... Please let me know if there is anything I can do/provide. Thank you! > Query on a secondary index creates huge CPU spike + unable to trace > --- > > Key: CASSANDRA-8530 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8530 > Project: Cassandra > Issue Type: Bug > Components: API, Core > Environment: CentOs 6.5 / Cassandra 2.1.2 >Reporter: Pavel Baranov > > After upgrading cassandra from 2.0.10 to 2.1.2 we are having all kinds of > issues, especially with performance. > java version "1.7.0_65" > Table creation: > {noformat} > tweets> desc table tweets; > CREATE TABLE tweets.tweets ( > uname text, > tweet_id bigint, > tweet text, > tweet_date timestamp, > tweet_date_only text, > uid bigint, > PRIMARY KEY (uname, tweet_id) > ) WITH CLUSTERING ORDER BY (tweet_id ASC) > AND bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '10', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.0 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.1 > AND speculative_retry = '99.0PERCENTILE'; > CREATE INDEX tweets_tweet_date_only_idx ON tweets.tweets (tweet_date_only); > CREATE INDEX tweets_uid ON tweets.tweets (uid); > {noformat} > With Cassandra 2.0.10 this query: > {noformat} > select uname from tweets where uid = 636732672 limit 1; > {noformat} > did not have any issues. After upgrade, I can see the cpu spikes and load avg > goes from ~1 to ~13, especially if I execute the query over and over again. > Doing "tracing on" does not work and just returns: > "Statement trace did not complete within 10 seconds" > I've done: > nodetool upgradesstables > recreated indexes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/3] cassandra git commit: ninja remove synchronized from MappedFileDataInput.readBytes
ninja remove synchronized from MappedFileDataInput.readBytes Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5cff788e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5cff788e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5cff788e Branch: refs/heads/trunk Commit: 5cff788efcc63ac738e16ca8cd912e1108d5c1be Parents: 29ff2d2 Author: Benedict Elliott Smith Authored: Mon Dec 22 15:39:29 2014 + Committer: Benedict Elliott Smith Committed: Mon Dec 22 15:39:29 2014 + -- src/java/org/apache/cassandra/io/util/MappedFileDataInput.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5cff788e/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java -- diff --git a/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java b/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java index f397ddc..fb5b62e 100644 --- a/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java +++ b/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java @@ -128,7 +128,7 @@ public class MappedFileDataInput extends AbstractDataInput implements FileDataIn * @return buffer with portion of file content * @throws IOException on any fail of I/O operation */ -public synchronized ByteBuffer readBytes(int length) throws IOException +public ByteBuffer readBytes(int length) throws IOException { int remaining = buffer.remaining() - position; if (length > remaining)
[1/3] cassandra git commit: ninja remove synchronized from MappedFileDataInput.readBytes
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 29ff2d26f -> 5cff788ef refs/heads/trunk 977cbddc4 -> 20ccf46f1 ninja remove synchronized from MappedFileDataInput.readBytes Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5cff788e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5cff788e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5cff788e Branch: refs/heads/cassandra-2.1 Commit: 5cff788efcc63ac738e16ca8cd912e1108d5c1be Parents: 29ff2d2 Author: Benedict Elliott Smith Authored: Mon Dec 22 15:39:29 2014 + Committer: Benedict Elliott Smith Committed: Mon Dec 22 15:39:29 2014 + -- src/java/org/apache/cassandra/io/util/MappedFileDataInput.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5cff788e/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java -- diff --git a/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java b/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java index f397ddc..fb5b62e 100644 --- a/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java +++ b/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java @@ -128,7 +128,7 @@ public class MappedFileDataInput extends AbstractDataInput implements FileDataIn * @return buffer with portion of file content * @throws IOException on any fail of I/O operation */ -public synchronized ByteBuffer readBytes(int length) throws IOException +public ByteBuffer readBytes(int length) throws IOException { int remaining = buffer.remaining() - position; if (length > remaining)
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/20ccf46f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/20ccf46f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/20ccf46f Branch: refs/heads/trunk Commit: 20ccf46f1830859f0e2fbacf0b9dc8ac69907f35 Parents: 977cbdd 5cff788 Author: Benedict Elliott Smith Authored: Mon Dec 22 15:39:38 2014 + Committer: Benedict Elliott Smith Committed: Mon Dec 22 15:39:38 2014 + -- src/java/org/apache/cassandra/io/util/MappedFileDataInput.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/20ccf46f/src/java/org/apache/cassandra/io/util/MappedFileDataInput.java --
[jira] [Commented] (CASSANDRA-8099) Refactor and modernize the storage engine
[ https://issues.apache.org/jira/browse/CASSANDRA-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255824#comment-14255824 ] Sylvain Lebresne commented on CASSANDRA-8099: - As an update on this I've (force) pushed to [the branch|https://github.com/pcmanus/cassandra/tree/8099_engine_refactor]. This is still not done but it compiles, run and work for basic queries. That said, at least reverse, distinct and 2ndary index queries (and a bunch of other smaller details) are known not to work (but that should change relatively soon). Also, I'll still need to work on thrift support and general backward compatibility for rolling upgrades. Nonetheless, the general design is here and shouldn't change drastically. Plus it's barely tested at this point, I need to add more comments. This is a huge patch. And while this doesn't change how Cassandra works in any fudamental way, this can't really be called a small incremental change. I'm the first to admit that those are not good things in theory, but to be honest, as this tries to clean up the abstractions that are at the core of the storage engine, I don't see how this could be done a whole lot more incrementally. I do am convinced that this will make implementing tons of outstanding features a lot more easy and is worth it just for that. Anyway, that's my (lame) excuse for this. Also, it's in a single big commit because I had a lot of back and forth while implementing so that my wip commits were more misleading than anything. At at latter point, if that's deemed better for review, I could try to split it in smaller commits of somewhat-related code (though I'm not entirely sure it will help tremendously). > Refactor and modernize the storage engine > - > > Key: CASSANDRA-8099 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8099 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Fix For: 3.0 > > > The current storage engine (which for this ticket I'll loosely define as "the > code implementing the read/write path") is suffering from old age. One of the > main problem is that the only structure it deals with is the cell, which > completely ignores the more high level CQL structure that groups cell into > (CQL) rows. > This leads to many inefficiencies, like the fact that during a reads we have > to group cells multiple times (to count on replica, then to count on the > coordinator, then to produce the CQL resultset) because we forget about the > grouping right away each time (so lots of useless cell names comparisons in > particular). But outside inefficiencies, having to manually recreate the CQL > structure every time we need it for something is hindering new features and > makes the code more complex that it should be. > Said storage engine also has tons of technical debt. To pick an example, the > fact that during range queries we update {{SliceQueryFilter.count}} is pretty > hacky and error prone. Or the overly complex ways {{AbstractQueryPager}} has > to go into to simply "remove the last query result". > So I want to bite the bullet and modernize this storage engine. I propose to > do 2 main things: > # Make the storage engine more aware of the CQL structure. In practice, > instead of having partitions be a simple iterable map of cells, it should be > an iterable list of row (each being itself composed of per-column cells, > though obviously not exactly the same kind of cell we have today). > # Make the engine more iterative. What I mean here is that in the read path, > we end up reading all cells in memory (we put them in a ColumnFamily object), > but there is really no reason to. If instead we were working with iterators > all the way through, we could get to a point where we're basically > transferring data from disk to the network, and we should be able to reduce > GC substantially. > Please note that such refactor should provide some performance improvements > right off the bat but it's not it's primary goal either. It's primary goal is > to simplify the storage engine and adds abstraction that are better suited to > further optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8464) Support direct buffer decompression for reads
[ https://issues.apache.org/jira/browse/CASSANDRA-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255789#comment-14255789 ] T Jake Luciani commented on CASSANDRA-8464: --- I've updated the branch with fixes based on your comments: * tested the case when the Adler32.update(ByteBuffer) call is not found and there is a slight perf hit but it's still faster than current. * added boolean ICompressor.useDirectOutbutByteBuffers() method so the caller knows what to pass into the uncompress call. This fixes the DeflateCompressor and SnappyCompressor issue you raised. * changed the tests to test all compressor implementations. * I left the treemap as is, since I'm not sure there would be any measurable impact to optimize this for a single segment vs a treemap of one. > Support direct buffer decompression for reads > - > > Key: CASSANDRA-8464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8464 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Labels: performance > Fix For: 3.0 > > > Currently when we read a compressed sstable we copy the data on heap then > send it to be de-compressed to another on heap buffer (albeit pooled). > But now both snappy and lz4 (with CASSANDRA-7039) allow decompression of > direct byte buffers. This lets us mmap the data and decompress completely > off heap (and avoids moving bytes over JNI). > One issue is performing the checksum offheap but the Adler32 does support in > java 8 (it's also in java 7 but marked private?!) > This change yields a > 10% boost in read performance on cstar. Locally I see > upto 30% improvement. > http://cstar.datastax.com/graph?stats=5ebcdd70-816b-11e4-aed6-42010af0688f&metric=op_rate&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=200.09&ymin=0&ymax=135908.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] cassandra git commit: Cassandra stress user defined writes should populate sequentially
Repository: cassandra Updated Branches: refs/heads/trunk bf599fb5b -> 977cbddc4 Cassandra stress user defined writes should populate sequentially Patch by tjake; reviewed by Benedict for CASSANDRA-8524 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/29ff2d26 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/29ff2d26 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/29ff2d26 Branch: refs/heads/trunk Commit: 29ff2d26f87f5977aa919d037ca3bbb67e40bdb0 Parents: 9775a9c Author: T Jake Luciani Authored: Mon Dec 22 08:59:12 2014 -0500 Committer: T Jake Luciani Committed: Mon Dec 22 08:59:12 2014 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/stress/settings/SettingsCommandUser.java | 5 + .../apache/cassandra/stress/settings/SettingsPopulation.java| 5 + 3 files changed, 11 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/29ff2d26/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index f7b47e8..64bdb62 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.1.3 + * Stress - user defined writes should populate sequentally (CASSANDRA-8524) * Fix regression in SSTableRewriter causing some rows to become unreadable during compaction (CASSANDRA-8429) * Run major compactions for repaired/unrepaired in parallel (CASSANDRA-8510) http://git-wip-us.apache.org/repos/asf/cassandra/blob/29ff2d26/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java -- diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java index 9841433..d5b221c 100644 --- a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java +++ b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java @@ -65,6 +65,11 @@ public class SettingsCommandUser extends SettingsCommand throw new IllegalArgumentException("Must specify at least one command with a non-zero ratio"); } +public boolean hasInsertOnly() +{ +return ratios.size() == 1 && ratios.containsKey("insert"); +} + public OpDistributionFactory getFactory(final StressSettings settings) { final SeedManager seeds = new SeedManager(settings); http://git-wip-us.apache.org/repos/asf/cassandra/blob/29ff2d26/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java -- diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java index da4c282..e57faea 100644 --- a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java +++ b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java @@ -134,6 +134,11 @@ public class SettingsPopulation implements Serializable String[] params = clArgs.remove("-pop"); if (params == null) { +if (command instanceof SettingsCommandUser && ((SettingsCommandUser)command).hasInsertOnly()) +{ +return new SettingsPopulation(new SequentialOptions(defaultLimit)); +} + // return defaults: switch(command.type) {
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/977cbddc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/977cbddc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/977cbddc Branch: refs/heads/trunk Commit: 977cbddc44da75a39813a84c582c7a9e08099d8c Parents: bf599fb 29ff2d2 Author: T Jake Luciani Authored: Mon Dec 22 09:01:01 2014 -0500 Committer: T Jake Luciani Committed: Mon Dec 22 09:01:01 2014 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/stress/settings/SettingsCommandUser.java | 5 + .../apache/cassandra/stress/settings/SettingsPopulation.java| 5 + 3 files changed, 11 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/977cbddc/CHANGES.txt -- diff --cc CHANGES.txt index 9e9a254,64bdb62..03d7e7f --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,47 -1,5 +1,48 @@@ +3.0 + * Support index key/value entries on map collections (CASSANDRA-8473) + * Modernize schema tables (CASSANDRA-8261) + * Support for user-defined aggregation functions (CASSANDRA-8053) + * Fix NPE in SelectStatement with empty IN values (CASSANDRA-8419) + * Refactor SelectStatement, return IN results in natural order instead + of IN value list order (CASSANDRA-7981) + * Support UDTs, tuples, and collections in user-defined + functions (CASSANDRA-7563) + * Fix aggregate fn results on empty selection, result column name, + and cqlsh parsing (CASSANDRA-8229) + * Mark sstables as repaired after full repair (CASSANDRA-7586) + * Extend Descriptor to include a format value and refactor reader/writer apis (CASSANDRA-7443) + * Integrate JMH for microbenchmarks (CASSANDRA-8151) + * Keep sstable levels when bootstrapping (CASSANDRA-7460) + * Add Sigar library and perform basic OS settings check on startup (CASSANDRA-7838) + * Support for aggregation functions (CASSANDRA-4914) + * Remove cassandra-cli (CASSANDRA-7920) + * Accept dollar quoted strings in CQL (CASSANDRA-7769) + * Make assassinate a first class command (CASSANDRA-7935) + * Support IN clause on any clustering column (CASSANDRA-4762) + * Improve compaction logging (CASSANDRA-7818) + * Remove YamlFileNetworkTopologySnitch (CASSANDRA-7917) + * Do anticompaction in groups (CASSANDRA-6851) + * Support pure user-defined functions (CASSANDRA-7395, 7526, 7562, 7740, 7781, 7929, + 7924, 7812, 8063, 7813) + * Permit configurable timestamps with cassandra-stress (CASSANDRA-7416) + * Move sstable RandomAccessReader to nio2, which allows using the + FILE_SHARE_DELETE flag on Windows (CASSANDRA-4050) + * Remove CQL2 (CASSANDRA-5918) + * Add Thrift get_multi_slice call (CASSANDRA-6757) + * Optimize fetching multiple cells by name (CASSANDRA-6933) + * Allow compilation in java 8 (CASSANDRA-7028) + * Make incremental repair default (CASSANDRA-7250) + * Enable code coverage thru JaCoCo (CASSANDRA-7226) + * Switch external naming of 'column families' to 'tables' (CASSANDRA-4369) + * Shorten SSTable path (CASSANDRA-6962) + * Use unsafe mutations for most unit tests (CASSANDRA-6969) + * Fix race condition during calculation of pending ranges (CASSANDRA-7390) + * Fail on very large batch sizes (CASSANDRA-8011) + * Improve concurrency of repair (CASSANDRA-6455, 8208) + + 2.1.3 + * Stress - user defined writes should populate sequentally (CASSANDRA-8524) * Fix regression in SSTableRewriter causing some rows to become unreadable during compaction (CASSANDRA-8429) * Run major compactions for repaired/unrepaired in parallel (CASSANDRA-8510)
cassandra git commit: Cassandra stress user defined writes should populate sequentially
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 9775a9c68 -> 29ff2d26f Cassandra stress user defined writes should populate sequentially Patch by tjake; reviewed by Benedict for CASSANDRA-8524 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/29ff2d26 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/29ff2d26 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/29ff2d26 Branch: refs/heads/cassandra-2.1 Commit: 29ff2d26f87f5977aa919d037ca3bbb67e40bdb0 Parents: 9775a9c Author: T Jake Luciani Authored: Mon Dec 22 08:59:12 2014 -0500 Committer: T Jake Luciani Committed: Mon Dec 22 08:59:12 2014 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/stress/settings/SettingsCommandUser.java | 5 + .../apache/cassandra/stress/settings/SettingsPopulation.java| 5 + 3 files changed, 11 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/29ff2d26/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index f7b47e8..64bdb62 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.1.3 + * Stress - user defined writes should populate sequentally (CASSANDRA-8524) * Fix regression in SSTableRewriter causing some rows to become unreadable during compaction (CASSANDRA-8429) * Run major compactions for repaired/unrepaired in parallel (CASSANDRA-8510) http://git-wip-us.apache.org/repos/asf/cassandra/blob/29ff2d26/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java -- diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java index 9841433..d5b221c 100644 --- a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java +++ b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommandUser.java @@ -65,6 +65,11 @@ public class SettingsCommandUser extends SettingsCommand throw new IllegalArgumentException("Must specify at least one command with a non-zero ratio"); } +public boolean hasInsertOnly() +{ +return ratios.size() == 1 && ratios.containsKey("insert"); +} + public OpDistributionFactory getFactory(final StressSettings settings) { final SeedManager seeds = new SeedManager(settings); http://git-wip-us.apache.org/repos/asf/cassandra/blob/29ff2d26/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java -- diff --git a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java index da4c282..e57faea 100644 --- a/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java +++ b/tools/stress/src/org/apache/cassandra/stress/settings/SettingsPopulation.java @@ -134,6 +134,11 @@ public class SettingsPopulation implements Serializable String[] params = clArgs.remove("-pop"); if (params == null) { +if (command instanceof SettingsCommandUser && ((SettingsCommandUser)command).hasInsertOnly()) +{ +return new SettingsPopulation(new SequentialOptions(defaultLimit)); +} + // return defaults: switch(command.type) {
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255743#comment-14255743 ] T Jake Luciani commented on CASSANDRA-7168: --- The CollationController class is what merges the rows from the iterators > Add repair aware consistency levels > --- > > Key: CASSANDRA-7168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: T Jake Luciani > Labels: performance > Fix For: 3.0 > > > With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to > avoid a lot of extra disk I/O when running queries with higher consistency > levels. > Since repaired data is by definition consistent and we know which sstables > are repaired, we can optimize the read path by having a REPAIRED_QUORUM which > breaks reads into two phases: > > 1) Read from one replica the result from the repaired sstables. > 2) Read from a quorum only the un-repaired data. > For the node performing 1) we can pipeline the call so it's a single hop. > In the long run (assuming data is repaired regularly) we will end up with > much closer to CL.ONE performance while maintaining consistency. > Some things to figure out: > - If repairs fail on some nodes we can have a situation where we don't have > a consistent repaired state across the replicas. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8531) metric CommitLog.PendingTasks is always growing
Anton Lebedevich created CASSANDRA-8531: --- Summary: metric CommitLog.PendingTasks is always growing Key: CASSANDRA-8531 URL: https://issues.apache.org/jira/browse/CASSANDRA-8531 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra21-2.1.1-1.noarch Linux 2.6.32-431.el6.x86_64 RHEL 6.5 java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) Reporter: Anton Lebedevich org.apache.cassandra.db.commitlog.AbstractCommitLogService increments pending task counter each time it is being read (see method getPendingTasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-4851) CQL3: improve support for paginating over composites
[ https://issues.apache.org/jira/browse/CASSANDRA-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255679#comment-14255679 ] Aleksey Yeschenko commented on CASSANDRA-4851: -- Try 2.1-HEAD (cassandra-2.1 branch), and if you still have a bug there, open a new issue. > CQL3: improve support for paginating over composites > > > Key: CASSANDRA-4851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4851 > Project: Cassandra > Issue Type: Improvement > Components: API >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 2.0.6 > > Attachments: 4851.txt > > > Consider the following table: > {noformat} > CREATE TABLE test ( > k int, > c1 int, > c2 int, > PRIMARY KEY (k, c1, c2) > ) > {noformat} > with the following data: > {noformat} > k | c1 | c2 > > 0 | 0 | 0 > 0 | 0 | 1 > 0 | 1 | 0 > 0 | 1 | 1 > {noformat} > Currently, CQL3 allows to slice over either c1 or c2: > {noformat} > SELECT * FROM test WHERE k = 0 AND c1 > 0 AND c1 < 2 > SELECT * FROM test WHERE k = 0 AND c1 = 1 AND c2 > 0 AND c2 < 2 > {noformat} > but you cannot express a query that return the 3 last records. Indeed, for > that you would need to do a query like say: > {noformat} > SELECT * FROM test WHERE k = 0 AND ((c1 = 0 AND c2 > 0) OR c2 > 0) > {noformat} > but we don't support that. > This can make it hard to paginate over say all records for {{k = 0}} (I'm > saying "can" because if the value for c2 cannot be very large, an easy > workaround could be to paginate by entire value of c1, which you can do). > For the case where you only paginate to avoid OOMing on a query, > CASSANDRA-4415 will that and is probably the best solution. However, there > may be case where the pagination is say user (as in, the user of your > application) triggered. > I note that one solution would be to add the OR support at least in case like > the one above. That's definitively doable but on the other side, we won't be > able to support full-blown OR, so it may not be very natural that we support > seemingly random combination of OR and not others. > Another solution would be to allow the following syntax: > {noformat} > SELECT * FROM test WHERE k = 0 AND (c1, c2) > (0, 0) > {noformat} > which would literally mean that you want records where the values of c1 and > c2 taken as a tuple is lexicographically greater than the tuple (0, 0). This > is less SQL-like (though maybe some SQL store have that, it's a fairly thing > to have imo?), but would be much simpler to implement and probably to use too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-4851) CQL3: improve support for paginating over composites
[ https://issues.apache.org/jira/browse/CASSANDRA-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255676#comment-14255676 ] Pierre edited comment on CASSANDRA-4851 at 12/22/14 12:01 PM: -- Hi Aleksey, I edited my posts, indeed it works on 2.1.1 when I use driver, but I have a bug when I do the exact same query the driver sent in cqlsh was (Author: pierz): Hi Aleksey, I edited my posts, indeed it works on 2.1.1 when I use driver, but I have a bug when i do the exact same query the driver send via cqlsh. > CQL3: improve support for paginating over composites > > > Key: CASSANDRA-4851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4851 > Project: Cassandra > Issue Type: Improvement > Components: API >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 2.0.6 > > Attachments: 4851.txt > > > Consider the following table: > {noformat} > CREATE TABLE test ( > k int, > c1 int, > c2 int, > PRIMARY KEY (k, c1, c2) > ) > {noformat} > with the following data: > {noformat} > k | c1 | c2 > > 0 | 0 | 0 > 0 | 0 | 1 > 0 | 1 | 0 > 0 | 1 | 1 > {noformat} > Currently, CQL3 allows to slice over either c1 or c2: > {noformat} > SELECT * FROM test WHERE k = 0 AND c1 > 0 AND c1 < 2 > SELECT * FROM test WHERE k = 0 AND c1 = 1 AND c2 > 0 AND c2 < 2 > {noformat} > but you cannot express a query that return the 3 last records. Indeed, for > that you would need to do a query like say: > {noformat} > SELECT * FROM test WHERE k = 0 AND ((c1 = 0 AND c2 > 0) OR c2 > 0) > {noformat} > but we don't support that. > This can make it hard to paginate over say all records for {{k = 0}} (I'm > saying "can" because if the value for c2 cannot be very large, an easy > workaround could be to paginate by entire value of c1, which you can do). > For the case where you only paginate to avoid OOMing on a query, > CASSANDRA-4415 will that and is probably the best solution. However, there > may be case where the pagination is say user (as in, the user of your > application) triggered. > I note that one solution would be to add the OR support at least in case like > the one above. That's definitively doable but on the other side, we won't be > able to support full-blown OR, so it may not be very natural that we support > seemingly random combination of OR and not others. > Another solution would be to allow the following syntax: > {noformat} > SELECT * FROM test WHERE k = 0 AND (c1, c2) > (0, 0) > {noformat} > which would literally mean that you want records where the values of c1 and > c2 taken as a tuple is lexicographically greater than the tuple (0, 0). This > is less SQL-like (though maybe some SQL store have that, it's a fairly thing > to have imo?), but would be much simpler to implement and probably to use too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-4851) CQL3: improve support for paginating over composites
[ https://issues.apache.org/jira/browse/CASSANDRA-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255676#comment-14255676 ] Pierre commented on CASSANDRA-4851: --- Hi Aleksey, I edited my posts, indeed it works on 2.1.1 when I use driver, but I have a bug when i do the exact same query the driver send via cqlsh. > CQL3: improve support for paginating over composites > > > Key: CASSANDRA-4851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4851 > Project: Cassandra > Issue Type: Improvement > Components: API >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 2.0.6 > > Attachments: 4851.txt > > > Consider the following table: > {noformat} > CREATE TABLE test ( > k int, > c1 int, > c2 int, > PRIMARY KEY (k, c1, c2) > ) > {noformat} > with the following data: > {noformat} > k | c1 | c2 > > 0 | 0 | 0 > 0 | 0 | 1 > 0 | 1 | 0 > 0 | 1 | 1 > {noformat} > Currently, CQL3 allows to slice over either c1 or c2: > {noformat} > SELECT * FROM test WHERE k = 0 AND c1 > 0 AND c1 < 2 > SELECT * FROM test WHERE k = 0 AND c1 = 1 AND c2 > 0 AND c2 < 2 > {noformat} > but you cannot express a query that return the 3 last records. Indeed, for > that you would need to do a query like say: > {noformat} > SELECT * FROM test WHERE k = 0 AND ((c1 = 0 AND c2 > 0) OR c2 > 0) > {noformat} > but we don't support that. > This can make it hard to paginate over say all records for {{k = 0}} (I'm > saying "can" because if the value for c2 cannot be very large, an easy > workaround could be to paginate by entire value of c1, which you can do). > For the case where you only paginate to avoid OOMing on a query, > CASSANDRA-4415 will that and is probably the best solution. However, there > may be case where the pagination is say user (as in, the user of your > application) triggered. > I note that one solution would be to add the OR support at least in case like > the one above. That's definitively doable but on the other side, we won't be > able to support full-blown OR, so it may not be very natural that we support > seemingly random combination of OR and not others. > Another solution would be to allow the following syntax: > {noformat} > SELECT * FROM test WHERE k = 0 AND (c1, c2) > (0, 0) > {noformat} > which would literally mean that you want records where the values of c1 and > c2 taken as a tuple is lexicographically greater than the tuple (0, 0). This > is less SQL-like (though maybe some SQL store have that, it's a fairly thing > to have imo?), but would be much simpler to implement and probably to use too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-4851) CQL3: improve support for paginating over composites
[ https://issues.apache.org/jira/browse/CASSANDRA-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255669#comment-14255669 ] Aleksey Yeschenko commented on CASSANDRA-4851: -- Yes. > CQL3: improve support for paginating over composites > > > Key: CASSANDRA-4851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4851 > Project: Cassandra > Issue Type: Improvement > Components: API >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 2.0.6 > > Attachments: 4851.txt > > > Consider the following table: > {noformat} > CREATE TABLE test ( > k int, > c1 int, > c2 int, > PRIMARY KEY (k, c1, c2) > ) > {noformat} > with the following data: > {noformat} > k | c1 | c2 > > 0 | 0 | 0 > 0 | 0 | 1 > 0 | 1 | 0 > 0 | 1 | 1 > {noformat} > Currently, CQL3 allows to slice over either c1 or c2: > {noformat} > SELECT * FROM test WHERE k = 0 AND c1 > 0 AND c1 < 2 > SELECT * FROM test WHERE k = 0 AND c1 = 1 AND c2 > 0 AND c2 < 2 > {noformat} > but you cannot express a query that return the 3 last records. Indeed, for > that you would need to do a query like say: > {noformat} > SELECT * FROM test WHERE k = 0 AND ((c1 = 0 AND c2 > 0) OR c2 > 0) > {noformat} > but we don't support that. > This can make it hard to paginate over say all records for {{k = 0}} (I'm > saying "can" because if the value for c2 cannot be very large, an easy > workaround could be to paginate by entire value of c1, which you can do). > For the case where you only paginate to avoid OOMing on a query, > CASSANDRA-4415 will that and is probably the best solution. However, there > may be case where the pagination is say user (as in, the user of your > application) triggered. > I note that one solution would be to add the OR support at least in case like > the one above. That's definitively doable but on the other side, we won't be > able to support full-blown OR, so it may not be very natural that we support > seemingly random combination of OR and not others. > Another solution would be to allow the following syntax: > {noformat} > SELECT * FROM test WHERE k = 0 AND (c1, c2) > (0, 0) > {noformat} > which would literally mean that you want records where the values of c1 and > c2 taken as a tuple is lexicographically greater than the tuple (0, 0). This > is less SQL-like (though maybe some SQL store have that, it's a fairly thing > to have imo?), but would be much simpler to implement and probably to use too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-4851) CQL3: improve support for paginating over composites
[ https://issues.apache.org/jira/browse/CASSANDRA-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255660#comment-14255660 ] Pierre edited comment on CASSANDRA-4851 at 12/22/14 11:55 AM: -- When I run such query on cqlsh it trigger an error : "list index out of range" however it looks like working via session.execute(query) ([cqlsh 5.0.1 | Cassandra 2.1.1 | CQL spec 3.2.0 | Native protocol v3]) was (Author: pierz): Is this patch included in cassandra 2.1.1 ? > CQL3: improve support for paginating over composites > > > Key: CASSANDRA-4851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4851 > Project: Cassandra > Issue Type: Improvement > Components: API >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 2.0.6 > > Attachments: 4851.txt > > > Consider the following table: > {noformat} > CREATE TABLE test ( > k int, > c1 int, > c2 int, > PRIMARY KEY (k, c1, c2) > ) > {noformat} > with the following data: > {noformat} > k | c1 | c2 > > 0 | 0 | 0 > 0 | 0 | 1 > 0 | 1 | 0 > 0 | 1 | 1 > {noformat} > Currently, CQL3 allows to slice over either c1 or c2: > {noformat} > SELECT * FROM test WHERE k = 0 AND c1 > 0 AND c1 < 2 > SELECT * FROM test WHERE k = 0 AND c1 = 1 AND c2 > 0 AND c2 < 2 > {noformat} > but you cannot express a query that return the 3 last records. Indeed, for > that you would need to do a query like say: > {noformat} > SELECT * FROM test WHERE k = 0 AND ((c1 = 0 AND c2 > 0) OR c2 > 0) > {noformat} > but we don't support that. > This can make it hard to paginate over say all records for {{k = 0}} (I'm > saying "can" because if the value for c2 cannot be very large, an easy > workaround could be to paginate by entire value of c1, which you can do). > For the case where you only paginate to avoid OOMing on a query, > CASSANDRA-4415 will that and is probably the best solution. However, there > may be case where the pagination is say user (as in, the user of your > application) triggered. > I note that one solution would be to add the OR support at least in case like > the one above. That's definitively doable but on the other side, we won't be > able to support full-blown OR, so it may not be very natural that we support > seemingly random combination of OR and not others. > Another solution would be to allow the following syntax: > {noformat} > SELECT * FROM test WHERE k = 0 AND (c1, c2) > (0, 0) > {noformat} > which would literally mean that you want records where the values of c1 and > c2 taken as a tuple is lexicographically greater than the tuple (0, 0). This > is less SQL-like (though maybe some SQL store have that, it's a fairly thing > to have imo?), but would be much simpler to implement and probably to use too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-4851) CQL3: improve support for paginating over composites
[ https://issues.apache.org/jira/browse/CASSANDRA-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255660#comment-14255660 ] Pierre commented on CASSANDRA-4851: --- Is this patch included in cassandra 2.1.1 ? > CQL3: improve support for paginating over composites > > > Key: CASSANDRA-4851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4851 > Project: Cassandra > Issue Type: Improvement > Components: API >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 2.0.6 > > Attachments: 4851.txt > > > Consider the following table: > {noformat} > CREATE TABLE test ( > k int, > c1 int, > c2 int, > PRIMARY KEY (k, c1, c2) > ) > {noformat} > with the following data: > {noformat} > k | c1 | c2 > > 0 | 0 | 0 > 0 | 0 | 1 > 0 | 1 | 0 > 0 | 1 | 1 > {noformat} > Currently, CQL3 allows to slice over either c1 or c2: > {noformat} > SELECT * FROM test WHERE k = 0 AND c1 > 0 AND c1 < 2 > SELECT * FROM test WHERE k = 0 AND c1 = 1 AND c2 > 0 AND c2 < 2 > {noformat} > but you cannot express a query that return the 3 last records. Indeed, for > that you would need to do a query like say: > {noformat} > SELECT * FROM test WHERE k = 0 AND ((c1 = 0 AND c2 > 0) OR c2 > 0) > {noformat} > but we don't support that. > This can make it hard to paginate over say all records for {{k = 0}} (I'm > saying "can" because if the value for c2 cannot be very large, an easy > workaround could be to paginate by entire value of c1, which you can do). > For the case where you only paginate to avoid OOMing on a query, > CASSANDRA-4415 will that and is probably the best solution. However, there > may be case where the pagination is say user (as in, the user of your > application) triggered. > I note that one solution would be to add the OR support at least in case like > the one above. That's definitively doable but on the other side, we won't be > able to support full-blown OR, so it may not be very natural that we support > seemingly random combination of OR and not others. > Another solution would be to allow the following syntax: > {noformat} > SELECT * FROM test WHERE k = 0 AND (c1, c2) > (0, 0) > {noformat} > which would literally mean that you want records where the values of c1 and > c2 taken as a tuple is lexicographically greater than the tuple (0, 0). This > is less SQL-like (though maybe some SQL store have that, it's a fairly thing > to have imo?), but would be much simpler to implement and probably to use too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8524) Cassandra stress user defined writes should populate sequentially
[ https://issues.apache.org/jira/browse/CASSANDRA-8524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255600#comment-14255600 ] Benedict commented on CASSANDRA-8524: - I think it should only switch to sequential if it's _only_ inserts being performed, since otherwise you're guaranteed to insert/query a disjoint set of keys. Though I'm not _too_ concerned about the default, so I'll leave it to you. Otherwise +1. > Cassandra stress user defined writes should populate sequentially > - > > Key: CASSANDRA-8524 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8524 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Priority: Trivial > Fix For: 2.1.3 > > Attachments: 8524.txt, 8524v2.txt > > > This matches the old stress way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8499) Ensure SSTableWriter cleans up properly after failure
[ https://issues.apache.org/jira/browse/CASSANDRA-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255596#comment-14255596 ] Marcus Eriksson commented on CASSANDRA-8499: In general LGTM, but at least for the 2.0 patch I think we should mimic current behavior as much as possible (ie, in SSTableWriter.abort(), we used closeQuietly which only logs an error message if we fail closing, now we throw an FSWriteError). Since we always* propagate the exception that caused abort() to be called, maybe it is better to always just log the exceptions in abort() and let the cause of abort() be thrown all the way out? (*we should propagate the cause in doAntiCompaction as well) > Ensure SSTableWriter cleans up properly after failure > - > > Key: CASSANDRA-8499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8499 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 2.0.12 > > Attachments: 8499-20.txt, 8499-21.txt > > > In 2.0 we do not free a bloom filter, in 2.1 we do not free a small piece of > offheap memory for writing compression metadata. In both we attempt to flush > the BF despite having encountered an exception, making the exception slow to > propagate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8530) Query on a secondary index creates huge CPU spike + unable to trace
[ https://issues.apache.org/jira/browse/CASSANDRA-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255578#comment-14255578 ] Sylvain Lebresne commented on CASSANDRA-8530: - We'll have a look, but know that this is a lot more likely to be looked at/fix quickly if you provide full reproduction steps. Typically, a self-contained script that reproduce the problem on a branch new cluster would be really good (and of course, if you can't reproduce on a test cluster, then that's useful information anyway). > Query on a secondary index creates huge CPU spike + unable to trace > --- > > Key: CASSANDRA-8530 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8530 > Project: Cassandra > Issue Type: Bug > Components: API, Core > Environment: CentOs 6.5 / Cassandra 2.1.2 >Reporter: Pavel Baranov > > After upgrading cassandra from 2.0.10 to 2.1.2 we are having all kinds of > issues, especially with performance. > java version "1.7.0_65" > Table creation: > {noformat} > tweets> desc table tweets; > CREATE TABLE tweets.tweets ( > uname text, > tweet_id bigint, > tweet text, > tweet_date timestamp, > tweet_date_only text, > uid bigint, > PRIMARY KEY (uname, tweet_id) > ) WITH CLUSTERING ORDER BY (tweet_id ASC) > AND bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '10', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.0 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.1 > AND speculative_retry = '99.0PERCENTILE'; > CREATE INDEX tweets_tweet_date_only_idx ON tweets.tweets (tweet_date_only); > CREATE INDEX tweets_uid ON tweets.tweets (uid); > {noformat} > With Cassandra 2.0.10 this query: > {noformat} > select uname from tweets where uid = 636732672 limit 1; > {noformat} > did not have any issues. After upgrade, I can see the cpu spikes and load avg > goes from ~1 to ~13, especially if I execute the query over and over again. > Doing "tracing on" does not work and just returns: > "Statement trace did not complete within 10 seconds" > I've done: > nodetool upgradesstables > recreated indexes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8513) SSTableScanner may not acquire reference, but will still release it when closed
[ https://issues.apache.org/jira/browse/CASSANDRA-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255545#comment-14255545 ] Marcus Eriksson edited comment on CASSANDRA-8513 at 12/22/14 8:37 AM: -- [~benedict] could you explain when this happens? (ie, references == 0, but we still expect it to return data?) edit: ah, saw the comment in CASSANDRA-8399 also, a unit test showing this would be nice was (Author: krummas): [~benedict] could you explain when this happens? (ie, references == 0, but we still expect it to return data?) also, a unit test showing this would be nice > SSTableScanner may not acquire reference, but will still release it when > closed > --- > > Key: CASSANDRA-8513 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8513 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 2.1.3 > > Attachments: 8513.txt > > > getRangeSlice and related methods do not require a reference to perform a > safe read, so during one it is possible for SSTableScanner to fail to acquire > a reference to the table it's being created on. In this case it doesn't fail, > but it also doesn't log that no reference was acquired; on close, it still > releases the reference that it never held. > We can fix this one of two ways: we can tell the scanner if we require a > reference to be acquired; or we can track if one was obtained. The former is > a bit more invasive but probably better, since we probably care if we fail to > grab one in cases where we need it, and probably don't want to grab one > otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8513) SSTableScanner may not acquire reference, but will still release it when closed
[ https://issues.apache.org/jira/browse/CASSANDRA-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255545#comment-14255545 ] Marcus Eriksson commented on CASSANDRA-8513: [~benedict] could you explain when this happens? (ie, references == 0, but we still expect it to return data?) also, a unit test showing this would be nice > SSTableScanner may not acquire reference, but will still release it when > closed > --- > > Key: CASSANDRA-8513 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8513 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 2.1.3 > > Attachments: 8513.txt > > > getRangeSlice and related methods do not require a reference to perform a > safe read, so during one it is possible for SSTableScanner to fail to acquire > a reference to the table it's being created on. In this case it doesn't fail, > but it also doesn't log that no reference was acquired; on close, it still > releases the reference that it never held. > We can fix this one of two ways: we can tell the scanner if we require a > reference to be acquired; or we can track if one was obtained. The former is > a bit more invasive but probably better, since we probably care if we fail to > grab one in cases where we need it, and probably don't want to grab one > otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8525) Bloom Filter truePositive counter not updated on key cache hit
[ https://issues.apache.org/jira/browse/CASSANDRA-8525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-8525: --- Attachment: 0001-inc-truepositive-for-keycache-hits.patch > Bloom Filter truePositive counter not updated on key cache hit > -- > > Key: CASSANDRA-8525 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8525 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Tyler Hobbs >Assignee: Marcus Eriksson > Fix For: 2.0.12, 2.1.3 > > Attachments: 0001-inc-truepositive-for-keycache-hits.patch > > > In {{SSTableReader.getPosition()}}, we're not incrementing the bloom filter > truePositive count when we get a key cache hit. This can lead to misleading > FP ratios in metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)