Re: Seed nodes in cassandra.yaml can not be hostnames
I just discovered that using host names for seed nodes in cassandra.yaml do not work. This is done on purpose? I believe so yes, to avoid relying on DNS to map correctly given that everything else is based on IP address. (IIRC, someone chime in if there is a different reason.) -- / Peter Schuller (@scode on twitter)
Re: messages stopped for 3 minutes?
h. never mind, possibly the first 24 seconds delay was caused by GC, the GC logging was not printed in system.log, I found one line on stdout that possibly corresponds to that. I found I left out the enable parallel remark param, let me add that and retry. Thanks Yang On Mon, Sep 26, 2011 at 12:25 AM, Yang tedd...@gmail.com wrote: ok, I got more detailed logging that seems to give some clue out of this myth I enabled TRACE level logging for Gossiper, also added a debug to MessageDeliveryTask so that every arrival msg shows up in logging. I have 2 nodes, all the requests come from the coordinator node, basically I see 2 possible issues. 1) in the front of bad_coord.txt, we see that the coordinator was receiving messages correctly until DEBUG [RequestResponseStage:22] 2011-09-26 05:43:35,521 MessageDeliveryTask.java (line 60) delivering msg then the line right after it is DEBUG [RequestResponseStage:24] 2011-09-26 05:43:59,516 MessageDeliveryTask.java (line 60) delivering msg so there is a 24 seconds delay , in this period, no messages were delivered on the coordinator. this is enough to trigger a Timeout and Unavailable. indeed we see time out and UnavailableException later in the log. during this period, on the replica box, in bad_replica.txt we see correct message delivery up to DEBUG [ReadStage:107] 2011-09-26 05:43:35,214 MessageDeliveryTask.java (line 60) delivering msg then on the next line, it starts doing something different: TRACE [GossipTasks:1] 2011-09-26 05:43:35,570 Gossiper.java (line 125) My heartbeat is now 5721 TRACE [GossipTasks:1] 2011-09-26 05:43:35,570 Gossiper.java (line 357) Gossip Digests are : /10.71.13.5:1317009996:5765 ec2-50-19-172-99.compute-1.amazonaws.com/10.71.21.46:1317010027:5721 I did not read the code in detail to understand what this means, but it seems to correspond with the silent period on the coord in terms of time. it keeps on doing this till INFO [GossipTasks:1] 2011-09-26 05:43:53,638 Gossiper.java (line 721) InetAddress /10.71.13.5 is now dead. and DEBUG [MutationStage:55] 2011-09-26 05:43:59,208 MessageDeliveryTask.java (line 60) delivering msg INFO [GossipStage:1] 2011-09-26 05:43:59,211 Gossiper.java (line 707) InetAddress /10.71.13.5 is now UP = so, here the problem is that the messages stopped showing up from 05:43:35 to 05:43:59 on the coord, though the replica is trying to send them . not sure if this is due to network. 2) the other problem actually causes a longer delay on application side in the end of bad_coord.txt we see that the TimeoutException and AvailableException were printed out actually much later, after 06:06:33 TRACE [GossipStage:1] 2011-09-26 06:06:33,107 Gossiper.java (line 802) /10.71.21.46local generation 1317010027, remote generation 1317010027 TRACE [GossipStage:1] 2011-09-26 06:06:33,107 Gossiper.java (line 847) Updating heartbeat state version to 7119 from 7118 for /10.71.21.46 ... ERROR [New I/O server worker #1-15] 2011-09-26 05:43:59,500 MY_APPLICATION_CODE.java (line 234) : Caused by: TimedOutException() ** the main issue here is that the timestamp of the logging line itself says 05:43:59, which is correctly when the TimeOutException was generated**, but the line was actually not printed out till 06:06:33, 23minutes later. this causes a 23minutes pause visible to the clients. this 23minutes delay is much more damaging than the 24 seconds message drop period in 1). I wonder what could have caused this 23minutes delay. could be lock contention on grabbing the log4j appender ?? but it shouldn't be, since I only added the extra logging AFTER I saw the problem appear before. Thanks for your patience for reading thus far :) Yang On Sun, Sep 25, 2011 at 10:32 AM, Yang tedd...@gmail.com wrote: thanks Jonathan, I really don't know, I just did further tests to catch the jstack on the receiving side over the last night. going through these stacks now. if I can't find anything suspicious, I'll add these debugging to the sending side too. another useful piece of info: when I did a single-node setup, I also found a lot of TimedOutException, similar to what I see with the 2-node setup. I think I didn't see the UnavailableException, probably because it's just a single node, and the node always believes itself to be available. I think GC issue is not the culprit here, since I don't see any length GC logging around when the delay is happening. no compaction/flushing either On Sun, Sep 25, 2011 at 6:33 AM, Jonathan Ellis jbel...@gmail.com wrote: What makes you think the problem is on the receiving node, rather than the sending node? On Sun, Sep 25, 2011 at 1:19 AM, Yang tedd...@gmail.com wrote: I constantly see TimedOutException , then followed by UnavailableException in my logs, so I added some extra debugging to Gossiper. notifyFailureDetector() void notifyFailureDetector(InetAddress
[ANN] Mojo's Cassandra Maven Plugin 0.8.6-1 released
The Mojo team is pleased to announce the release of Mojo's Cassandra Maven Plugin version 0.8.6-1. Mojo's Cassandra Plugin is used when you want to install and control a test instance of Apache Cassandra from within your Apache Maven build. The Cassandra Plugin has the following goals. * cassandra:start Starts up a test instance of Cassandra in the background. * cassandra:stop Stops the test instance of Cassandra that was started using cassandra:start. * cassandra:start-cluster Starts up a test cluster of Cassandra in the background bound to the local loopback IP addresses 127.0.0.1, 127.0.0.2, etc. * cassandra:stop Stops the test cluster of Cassandra that was started using cassandra:start. * cassandra:run Starts up a test instance of Cassandra in the foreground. * cassandra:load Runs a cassandra-cli script against the test instance of Cassandra. * cassandra:repair Runs nodetool repair against the test instance of Cassandra. * cassandra:flush Runs nodetool flush against the test instance of Cassandra. * cassandra:compact Runs nodetool compact against the test instance of Cassandra. * cassandra:cleanup Runs nodetool cleanup against the test instance of Cassandra. * cassandra:delete Deletes the the test instance of Cassandra. * cassandra:cql-exec Execute a CQL statement (directly or from a file) against the test instance of Cassandra. http://mojo.codehaus.org/cassandra-maven-plugin/ To use this version, simply specify the version in your project's plugin configuration: plugin groupIdorg.codehaus.mojo/groupId artifactIdcassandra-maven-plugin/artifactId version0.8.6-1/version /plugin Release Notes - Mojo's Cassandra Maven Plugin - Version 0.8.6-1 ** Bug * [MCASSANDRA-12] - cassandra:start fails on Cywin ** Improvement * [MCASSANDRA-13] - Upgrade to Cassandra 0.8.6 Enjoy, The Mojo team. Apache, Apache Maven, Apache Cassandra, Maven and Cassandra are trademarks of The Apache Software Foundation.
how does compaction_throughput_kb_per_sec affect disk io?
I am using the default 16MB when running repair. but the disk io is still quite high: Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 136.00 0.00 506.00 26.00 63430.00 5880.00 260.56 101.73 224.386.60 4462.62 1.88 100.00 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 58.50 21978.00 131.00 122.50 16226.00 52596.00 542.97 122.98 870.28 10.02 1790.24 3.94 100.00 the rkB/s and wKB/s are almost 60MB, did I misunderstand the meaning of compaction throttle? cassandra version is 0.8.6
Thrift CPU Usage
Hello - I have been running read tests on Cassandra using stress tool. I have been noticing that thrift seems to be taking lot of CPU over 70% when I look at the CPU samples report. Is this normal? CPU usage seems to go down by 5 to 10% when I change the RPC from sync to async. Is this normal? I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle JVM. - Thanks Baskar Duraikannu
Quick advice needed - CF backup and restore
Hi, What is the easiest way to save/backup a single column family across the cluster and later reload it? Thank you very much, Oleg
Re: Seed nodes in cassandra.yaml can not be hostnames
The seed names should match what the seeds advertise as listen_address. I can't think of a reason host names shouldn't work, but as Peter said, using host names is a bad idea anyway. 2011/9/25 Radim Kolar h...@sendmail.cz: I just discovered that using host names for seed nodes in cassandra.yaml do not work. This is done on purpose? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: how does compaction_throughput_kb_per_sec affect disk io?
compaction throughput doesn't affect flushing or reads On Mon, Sep 26, 2011 at 7:40 AM, Yan Chunlu springri...@gmail.com wrote: I am using the default 16MB when running repair. but the disk io is still quite high: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 136.00 0.00 506.00 26.00 63430.00 5880.00 260.56 101.73 224.38 6.60 4462.62 1.88 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 58.50 21978.00 131.00 122.50 16226.00 52596.00 542.97 122.98 870.28 10.02 1790.24 3.94 100.00 the rkB/s and wKB/s are almost 60MB, did I misunderstand the meaning of compaction throttle? cassandra version is 0.8.6 -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Quick advice needed - CF backup and restore
dsh -g mycluster -c nodetool -h localhost snapshot see http://www.netfort.gr.jp/~dancer/software/dsh.html.en On Mon, Sep 26, 2011 at 8:36 AM, Oleg Proudnikov ol...@cloudorange.com wrote: Hi, What is the easiest way to save/backup a single column family across the cluster and later reload it? Thank you very much, Oleg -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
SCF column comparator
Hi all, Im trying to create a Threads SCF that will store message thread id's in date order and i want to store the threadID = subject as the supercolumns. Please correct me if im incorrect but my understanding of a super column family is as follows: Category: //row key Timestamp: //Column ThreadID = Thread Subject //Supercolumn My question is how do I ensure the timestamp column is ordered chronologically as the documentation for the cassandra-cli specify's that the comparator type is applied to the supercolumns. Also im new to Cassandra so any advice on alternate schemas is welcome, not sure if this would be better to split up into multiple CF's or somehow use an index. Thanks in advance Sam
Re: how does compaction_throughput_kb_per_sec affect disk io?
I would think that compaction_throughput_kb_per_sec does have indirect impact on disk IO. High number means or setting it to 0 means there is no throttling on how much IO is being performed. Wouldn't it impact normal reads from disk during the time when disk IO or util is high which compaction is taking place? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-does-compaction-throughput-kb-per-sec-affect-disk-io-tp6831711p6832308.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Seed nodes in cassandra.yaml can not be hostnames
Dne 26.9.2011 16:37, Jonathan Ellis napsal(a): The seed names should match what the seeds advertise as listen_address. I can't think of a reason host names shouldn't work, I used DNS alias, that was probably reason why it didn't worked.
Update of column sometimes takes 10 seconds
Hi All: We have a simple junit test that inserts a column, immediately updates that column and then validates that the data updated. Cassandra is run embedded in the unit test. Sometimes the test will pass, i.e. the updated data is correct, and sometimes the test will fail. The configuration is set to: CommitLogSyncperiodic/CommitLogSync and CommitLogSyncPeriodInMS1/CommitLogSyncPeriodInMS We are running version 0.6.9. We plan to update to the latest version but cannot until after the release we are wrapping up. We are using the client batch mutate to create and update the data. From what I understand, the commit log write will return immediately and the data will be store in memory. If that is the case, then why would our test sometimes fail? -Rick Whitesel Simplify, Standardize and Conserve Rick Whitesel Technical Lead Customer Contact Business Unit rwhit...@cisco.com mailto:nordb...@cisco.com Phone :978-936-0479 500 Beaver Brook Road Boxboro, MA 01719 Mailing address: 1414 Massachusetts Avenue Boxboro, MA 01719 United States www.cisco.com http://www.cisco.com/ image001.gif
RE: Update of column sometimes takes 10 seconds
Waiting 10 seconds between the update and reading the updated data seems to always work. Not waiting the 10 seconds will cause the test to randomly pass or fail. -Rick From: Jim Ancona [mailto:j...@anconafamily.com] Sent: Monday, September 26, 2011 3:04 PM To: user@cassandra.apache.org Subject: Re: Update of column sometimes takes 10 seconds Do you actually see the update occur if you wait for 10 seconds (as your subject implies), or do you just see intermittent failures when running the unit test? If it's the latter, are you sure that the update has a greater timestamp than the insert? I've seen similar unit tests fail because because the timestamp values were the same. Jim On Mon, Sep 26, 2011 at 2:55 PM, Rick Whitesel (rwhitese) rwhit...@cisco.com wrote: Hi All: We have a simple junit test that inserts a column, immediately updates that column and then validates that the data updated. Cassandra is run embedded in the unit test. Sometimes the test will pass, i.e. the updated data is correct, and sometimes the test will fail. The configuration is set to: CommitLogSyncperiodic/CommitLogSync and CommitLogSyncPeriodInMS1/CommitLogSyncPeriodInMS We are running version 0.6.9. We plan to update to the latest version but cannot until after the release we are wrapping up. We are using the client batch mutate to create and update the data. From what I understand, the commit log write will return immediately and the data will be store in memory. If that is the case, then why would our test sometimes fail? -Rick Whitesel Simplify, Standardize and Conserve Rick Whitesel Technical Lead Customer Contact Business Unit rwhit...@cisco.com mailto:nordb...@cisco.com Phone :978-936-0479 500 Beaver Brook Road Boxboro, MA 01719 Mailing address: 1414 Massachusetts Avenue Boxboro, MA 01719 United States www.cisco.com http://www.cisco.com/ image001.gif
Re: Token != DecoratedKey assertion
Looks like a mismatch between the key the index says should be at a certain position in the date file and the key that is actually there. I've not checked but scrub *may* fix this this. Try it and see. (repair is for repairing consistency between nodes, scrub fixes local issues with data. ) Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 26/09/2011, at 12:53 PM, Philippe wrote: Juste did Could there be data corruption or will repairs do this? Thanks Le 25 sept. 2011 15:30, Jonathan Ellis jbel...@gmail.com a écrit : Assertion errors are bugs, so that should worry you. However, I'd upgrade before filing a ticket. There were a lot of fixes in 0.8.5. On Sun, Sep 25, 2011 at 2:27 AM, Philippe watche...@gmail.com wrote: Hello, I've seen a couple of these in my logs, running 0.8.4. This is a RF=3, 3-node cluster. 2 nodes including this one are on 0.8.4 and one is on 0.8.5 The node is still functionning hours later. Should I be worried ? Thanks ERROR [ReadStage:94911] 2011-09-24 22:40:30,043 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[ReadStage:94911,5,main] java.lang.AssertionError: DecoratedKey(Token(bytes[224ceb80b5fb11e0848783ceb9bf0002ff33]), 224ceb80b5fb11e0848783ceb9bf0002ff33) != DecoratedKey(Token(bytes[038453154cb0005f14]), 038453154cb0005f14) in /var/lib/cassandra/data/X/PUBLIC_MONTHLY_20-g-10634-Data.db at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:59) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315) at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [ReadStage:94936] 2011-09-24 22:40:30,042 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[ReadStage:94936,5,main] java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) != DecoratedKey(Token(bytes[038453154c90005f14]), 038453154c90005f14) in /var/lib/cassandra/data/X/PUBLIC_MONTHLY_20-g-10634-Data.db at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:59) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315) at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [ReadStage:94713] 2011-09-24 22:40:30,041 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[ReadStage:94713,5,main] java.lang.AssertionError:
Re: Thrift CPU Usage
How are you deciding what is thrift ? Thrift is used to handle connections and serialize / de-serialize off the wire. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27/09/2011, at 2:32 AM, Baskar Duraikannu wrote: Hello - I have been running read tests on Cassandra using stress tool. I have been noticing that thrift seems to be taking lot of CPU over 70% when I look at the CPU samples report. Is this normal? CPU usage seems to go down by 5 to 10% when I change the RPC from sync to async. Is this normal? I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle JVM. - Thanks Baskar Duraikannu
Re: Surgecon Meetup?
I'll be at Surge on Thursday, would love to meet up. Anyone else planning to be there? On Sun, Sep 25, 2011 at 7:27 PM, Chris Burroughs chris.burrou...@gmail.comwrote: Surge [1] is scalability focused conference in late September hosted in Baltimore. It's a pretty cool conference with a good mix of operationally minded people interested in scalability, distributed systems, systems level performance and good stuff like that. You should go! [2] Anyway, I'll be there if there, and are if any other Cassandra users are coming I'm happy to help herd us towards meeting up, lunch, hacking, etc. I *think* there might be some time for structured BoF type sessions as well. [1] http://omniti.com/surge/2011 [2] Actually tickets recenlty sold out, you should go in 2012!
Re: Surgecon Meetup?
I'll be there from Tuesday night thru the weekend. - wilson...@gmail.com, 310.320-7878 On Mon, Sep 26, 2011 at 4:55 PM, Dan Kuebrich dan.kuebr...@gmail.com wrote: I'll be at Surge on Thursday, would love to meet up. Anyone else planning to be there? On Sun, Sep 25, 2011 at 7:27 PM, Chris Burroughs chris.burrou...@gmail.com wrote: Surge [1] is scalability focused conference in late September hosted in Baltimore. It's a pretty cool conference with a good mix of operationally minded people interested in scalability, distributed systems, systems level performance and good stuff like that. You should go! [2] Anyway, I'll be there if there, and are if any other Cassandra users are coming I'm happy to help herd us towards meeting up, lunch, hacking, etc. I *think* there might be some time for structured BoF type sessions as well. [1] http://omniti.com/surge/2011 [2] Actually tickets recenlty sold out, you should go in 2012!
Re: SCF column comparator
it's the other way around… row-key: super-column: (sub)column: When using Create Column Family in the CLI: key_validation_class applies to the row key comparator applies to the super column (when using a Super Column Family) subcomparator applied to the sub column (when using a Super Column Family) In your current model consider: - using an Integer or TimeUUID comparator - setting the super column name to milliseconds / seconds past epoch or using a Time UUID. I would suggest trying to model your data using a standard CF to start with, it will be easier on your brain and there are some limitations to a super CF. Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27/09/2011, at 3:46 AM, Sam Hodgson wrote: Hi all, Im trying to create a Threads SCF that will store message thread id's in date order and i want to store the threadID = subject as the supercolumns. Please correct me if im incorrect but my understanding of a super column family is as follows: Category: //row key Timestamp: //Column ThreadID = Thread Subject //Supercolumn My question is how do I ensure the timestamp column is ordered chronologically as the documentation for the cassandra-cli specify's that the comparator type is applied to the supercolumns. Also im new to Cassandra so any advice on alternate schemas is welcome, not sure if this would be better to split up into multiple CF's or somehow use an index. Thanks in advance Sam
Re: Thrift CPU Usage
Aaron From the CPU samples report. Here is the parts of the CPU samples report (-Xrunhprof:cpu=samples, depth=4). TRACE 300668: java.net.SocketInputStream.socketRead0(SocketInputStream.java:Unknown line) java.net.SocketInputStream.read(SocketInputStream.java:129) org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) TRACE 300310: java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line) java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) java.net.ServerSocket.implAccept(ServerSocket.java:462) java.net.ServerSocket.accept(ServerSocket.java:430) TRACE 300639: sun.nio.ch.ServerSocketChannelImpl.accept0(ServerSocketChannelImpl.java:Unknown line) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:617) TRACE 300670: java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java:Unknown line) java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) java.net.SocketOutputStream.write(SocketOutputStream.java:136) org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) TRACE 301122: java.lang.Object.notify(Object.java:Unknown line) org.apache.cassandra.utils.SimpleCondition.signal(SimpleCondition.java:62) org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:169) org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java rank self accum count trace method 1 74.00% 74.00% 160934 300668 java.net.SocketInputStream.socketRead0 2 14.85% 88.85% 32302 300310 java.net.PlainSocketImpl.socketAccept 3 3.67% 92.52%7990 300639 sun.nio.ch.ServerSocketChannelImpl.accept0 4 1.90% 94.43%4142 300670 java.net.SocketOutputStream.socketWrite0 5 0.79% 95.22%1716 301122 java.lang.Object.notify -- Thanks Baskar Duraikannu On Sep 26, 2011, at 4:55 PM, aaron morton wrote: How are you deciding what is thrift ? Thrift is used to handle connections and serialize / de-serialize off the wire. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27/09/2011, at 2:32 AM, Baskar Duraikannu wrote: Hello - I have been running read tests on Cassandra using stress tool. I have been noticing that thrift seems to be taking lot of CPU over 70% when I look at the CPU samples report. Is this normal? CPU usage seems to go down by 5 to 10% when I change the RPC from sync to async. Is this normal? I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle JVM. - Thanks Baskar Duraikannu
Re: Thrift CPU Usage
Yes. All the stress tool does is flood data through the API, no real processing or anything happens. So thrift reading/writing data should be the majority of the CPU time... On 09/26/2011 08:32 AM, Baskar Duraikannu wrote: Hello - I have been running read tests on Cassandra using stress tool. I have been noticing that thrift seems to be taking lot of CPU over 70% when I look at the CPU samples report. Is this normal? CPU usage seems to go down by 5 to 10% when I change the RPC from sync to async. Is this normal? I am running Cassandra 0.8.4 on Cent OS 5.6 ( Kernel 2.6.18.238) and Oracle JVM. - Thanks Baskar Duraikannu
Re: reverse range query performance
Does not matter to much but are you looking to get all the columns for some know keys (get_slice, multiget_slice) ? Or are you getting the columns for keys within a range (get_range_slices)? If you provide do a reversed query the server will skip to the end of the column range. Here is some info I wrote about how the the different slice predicates work http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/ Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27/09/2011, at 5:51 AM, Ramesh Natarajan wrote: Hi, I am trying to use the range query to retrieve a bunch of columns in reverse order. The API documentation has a parameter bool reversed which should return the results when queried using keys in a reverse order. Lets say my row has about 1500 columns with column names 1 to 1500, and I query asking for columns 1500 (start ) - 1400 (end ) with reverse set to true. Does cassandra read the entire row 1 - 1500 columns and then return the result 1400 - 1500 or it is optimized to look directly into the 1400 - 1500 columns? thanks Ramesh SliceRange A SliceRange is a structure that stores basic range, ordering and limit information for a query that will return multiple columns. It could be thought of as Cassandra's version of LIMIT and ORDER BY. Attribute Type Default Required Description start binary n/a Y The column name to start the slice with. This attribute is not required, though there is no default value, and can be safely set to '', i.e., an empty byte array, to start with the first column name. Otherwise, it must be a valid value under the rules of the Comparator defined for the given ColumnFamily. finish binary n/a Y The column name to stop the slice at. This attribute is not required, though there is no default value, and can be safely set to an empty byte array to not stop until count results are seen. Otherwise, it must also be a valid value to the ColumnFamily Comparator. reversed bool false Y Whether the results should be ordered in reversed order. Similar to ORDER BY blah DESC in SQL. count integer 100 Y How many columns to return. Similar to LIMIT 100 in SQL. May be arbitrarily large, but Thrift will materialize the whole result into memory before returning it to the client, so be aware that you may be better served by iterating through slices by passing the last value of one call in as the start of the next instead of increasing count arbitrarily large.
Re: Update of column sometimes takes 10 seconds
Try turning up the logging to DEBUG and watch the requests come through. Check that the two inserts do indeed have different time stamps. In cases of lost updates, timestamps are most often the cause of the kerfuffle. btw, in this situation the commit log is a red hearing riding a scapegoat. Data is dumped into the commit log for durability at the start of the write request, but it is never read from there. Data is read from memtables and sstables. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27/09/2011, at 9:27 AM, Rick Whitesel (rwhitese) wrote: Thank you to all for the quick response. The test that fails is doing a insert, another insert (to update data) and then a get to validate. If I make multiple copies of the same test and execute them in succession, different copies will fail on successive runs. Each test only has a single get, so one would assume the get in testN must be causing testN+1 to fail. It is going to take me a little bit to dig down deep enough into our code to see if that is the issue. -Rick From: Jonathan Ellis [mailto:jbel...@gmail.com] Sent: Monday, September 26, 2011 3:34 PM To: user@cassandra.apache.org Subject: Re: Update of column sometimes takes 10 seconds Sounds a lot like this to me: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Updates-lost-td6739612.html On Mon, Sep 26, 2011 at 1:55 PM, Rick Whitesel (rwhitese) rwhit...@cisco.com wrote: Hi All: We have a simple junit test that inserts a column, immediately updates that column and then validates that the data updated. Cassandra is run embedded in the unit test. Sometimes the test will pass, i.e. the updated data is correct, and sometimes the test will fail. The configuration is set to: CommitLogSyncperiodic/CommitLogSync and CommitLogSyncPeriodInMS1/CommitLogSyncPeriodInMS We are running version 0.6.9. We plan to update to the latest version but cannot until after the release we are wrapping up. We are using the client batch mutate to create and update the data. From what I understand, the commit log write will return immediately and the data will be store in memory. If that is the case, then why would our test sometimes fail? -Rick Whitesel Simplify, Standardize and Conserve image001.gif Rick Whitesel Technical Lead Customer Contact Business Unit rwhit...@cisco.com Phone :978-936-0479 500 Beaver Brook Road Boxboro, MA 01719 Mailing address: 1414 Massachusetts Avenue Boxboro, MA 01719 United States www.cisco.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: how does compaction_throughput_kb_per_sec affect disk io?
okay, thanks! On Mon, Sep 26, 2011 at 10:38 PM, Jonathan Ellis jbel...@gmail.com wrote: compaction throughput doesn't affect flushing or reads On Mon, Sep 26, 2011 at 7:40 AM, Yan Chunlu springri...@gmail.com wrote: I am using the default 16MB when running repair. but the disk io is still quite high: Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 136.00 0.00 506.00 26.00 63430.00 5880.00 260.56 101.73 224.386.60 4462.62 1.88 100.00 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 58.50 21978.00 131.00 122.50 16226.00 52596.00 542.97 122.98 870.28 10.02 1790.24 3.94 100.00 the rkB/s and wKB/s are almost 60MB, did I misunderstand the meaning of compaction throttle? cassandra version is 0.8.6 -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
