Re: Commitlog Disk Full
Hi Peter, Thanks for the response. I haven't explictly set a value for the memtable_flush_after_mins parameter. Looks like the default is 60minutes. I will try to play around this value to see if that fixes things. Thanks again! On Thu, May 12, 2011 at 11:41 AM, Peter Schuller peter.schul...@infidyne.com wrote: I understand that cassandra periodically cleans up the commitlog directories by generating sstables in datadir. Is there any way to speed up this movement from commitog to datadir? commitlog_rotation_threshold_in_mb could cause problems if it was set very very high, but with the default of 128mb it should not be an issue. I suspect the most likely reason is that you have a column family whose memtable flush settings are extreme. A commit log segment cannot be removed until the corresponding data has been flushed to an sstable. For high-throughput memtables where you flush regularly this should happen often. For idle or almost idle memtables you may be waiting on the timeout criteria to trigger. So in general, having a memtable with a long expiry time will have the potential to generate commit logs of whatever size is implied by the write traffic during that periods. The memtable setting in question is the memtable_flush_after setting. Do you have that set to something very high on one of your column families? You can use describe keyspace name_of_keyspace in cassandra-cli to check current settings. -- / Peter Schuller
Cassandra node is not blanced Rf=2 Random Partitioner
My cluster is unbalanced.One have 99 GB Data and other have 87 GB can any one explain why this is happening. [root@cassandra2 conf]# /root/cassandra/bin/nodetool -h 10.0.0.4 ring Address Status Load Range Ring 133487003891811234514792661331209113407 10.0.0.4 Up 99.63 GB 89592293417191096473651402050455 |--| 10.0.0.3 Up 87.02 GB 133487003891811234514792661331209113407|--| -- S.Ali Ahsan Senior System Engineer e-Business (Pvt) Ltd 49-C Jail Road, Lahore, P.O. Box 676 Lahore 54000, Pakistan Tel: +92 (0)42 3758 7140 Ext. 128 Mobile: +92 (0)345 831 8769 Fax: +92 (0)42 3758 0027 Email: ali.ah...@panasiangroup.com www.ebusiness-pg.com www.panasiangroup.com Confidentiality: This e-mail and any attachments may be confidential and/or privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person use it for any purpose or store or copy the information in any medium. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. We do not accept liability for any errors or omissions.
Re: Cassandra node is not blanced Rf=2 Random Partitioner
On 05/13/2011 11:46 AM, Ali Ahsan wrote: My cluster is unbalanced.One have 99 GB Data and other have 87 GB can any one explain why this is happening. [root@cassandra2 conf]# /root/cassandra/bin/nodetool -h 10.0.0.4 ring Address Status Load Range Ring 133487003891811234514792661331209113407 10.0.0.4 Up 99.63 GB 89592293417191096473651402050455 |--| 10.0.0.3 Up 87.02 GB 133487003891811234514792661331209113407|--| We are using 0.6.3 -- S.Ali Ahsan
Re: Commitlog Disk Full
I haven't explictly set a value for the memtable_flush_after_mins parameter. Looks like the default is 60minutes. I will try to play around this value to see if that fixes things. Is the amount of data in the commit log consistent with what you might have been writing during 60 minutes? Including overwrites. If not, I'm not sure what's going on. Since you said it took about a day of traffic it feels fishy. -- / Peter Schuller
Re: Monitoring bytes read per cf
It seems to work fine but maybe someone here knows that this is not a good idea It depends. If you're trying to figure out what generates actual iops to disk, it won't really tell you since you're counting all I/O including that which is cached. Since whatever goes down to disk tends to be, due to the LRU:ishness of caches, the less frequently accessed data that tends to make it difficult to judge by numbers that include all I/O. -- / Peter Schuller
Inter node communication over UDP
we are deploying cassandra over a huge cluster size and we think that inter node communication can become overload on nodes as the protocol used is TCP. Similarly gossip was earlier using udp but has been changed to use Tcp. Is there added advantage for using tcp over udp in cassandra both in gossip and data storage(internal storage and retrival, etc)? And how much development time i should expect to code udp? -pankaj -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Inter-node-communication-over-UDP-tp6358459p6358459.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Monitoring bytes read per cf
Thanks - yes I agree. Didn't want to judge solely based on this figure. It should just add to the picture. But since we know access patterns and other stats like key and row cache hit ratios we hope to be able to make a more educated guess whats going on. On May 13, 2011, at 9:08 AM, Peter Schuller wrote: It seems to work fine but maybe someone here knows that this is not a good idea It depends. If you're trying to figure out what generates actual iops to disk, it won't really tell you since you're counting all I/O including that which is cached. Since whatever goes down to disk tends to be, due to the LRU:ishness of caches, the less frequently accessed data that tends to make it difficult to judge by numbers that include all I/O. -- / Peter Schuller
Re: Import/Export of Schema Migrations
What sort of schema changes are you making? can you manage them as a CLI script under source control ? You may also be interested in CASSANDRA-2221. Cheers Aaron - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 12 May 2011, at 20:45, David Boxenhorn wrote: My use case is like this: I have a development cluster, a staging cluster and a production cluster. When I finish a set of migrations (i.e. changes) on the development cluster, I want to apply them to the staging cluster, and eventually the production cluster. I don't want to do it by hand, because it's a painful and error-prone process. What I would like to do is export the last N migrations from the development cluster as a text file, with exactly the same format as the original text commands, and import them to the staging and production clusters. I think the best place to do this might be the CLI, since you would probably want to view your migrations before exporting them. Something like this: show migrations N;Shows the last N migrations. export migrations N fileName; Exports the last N migrations to file fileName. import migrations fileName; Imports migrations from fileName. The import process would apply the migrations one at a time giving you feedback like, applying migration: update column family If a migration fails, the process should give an appropriate message and stop. Is anyone else interested in this? I have created a Jira ticket for it here: https://issues.apache.org/jira/browse/CASSANDRA-2636
Feedback on ACM SOCC paper about elasticity and scalability
Hello, I have written with a few other people a paper for the ACM Symposium On Cloud Computing. This paper describes the methodology, infrastructure and configuration used as well as the results obtained for elasticity and scalability of three noSQL databases, of wich Cassandra. The paper can be downloaded here : http://www.nosqlbenchmarking.com/wp-content/uploads/2011/05/paper.pdfhttp://www.google.com/url?sa=Dq=http://www.nosqlbenchmarking.com/wp-content/uploads/2011/05/paper.pdf Any feedback on the methodology used would be appreciated, we would like to know if Cassandra is used in a fair way in those tests. If the paper is accepted, we will have additional time to correct eventual mistakes or to include developers feedback. Regards, Thibault Dory
Re: Excessive allocation during hinted handoff
The number of Completed HH tasks is interesting. AFAIK a task is started when the node detects another in the cluster has returned. Were you doing some other restarts around the cluster ? I don't want to divert from the GC issue, just wondering if something else is going on as well. Like the node is been asked to record a lot of hints. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 13 May 2011, at 03:51, Gabriel Tataranu wrote: What does the TPStats look like on the nodes under pressure ? And how many nodes are delivering hints to the nodes when they restart? $nodetool -h 127.0.0.1 tpstats Pool NameActive Pending Completed ReadStage 1 11992475 RequestResponseStage 0 02247486 MutationStage 0 01631349 ReadRepairStage 0 0 583432 GossipStage 0 0 241324 AntiEntropyStage 0 0 0 MigrationStage0 0 0 MemtablePostFlusher 0 0 46 StreamStage 0 0 0 FlushWriter 0 0 46 MiscStage 0 0 0 FlushSorter 0 0 0 InternalResponseStage 0 0 0 HintedHandoff 1 5152 dstat -cmdln during the event: total-cpu-usage --memory-usage- ---load-avg--- -dsk/total- -net/total- usr sys idl wai hiq siq| used buff cach free| 1m 5m 15m | read writ| recv send 87 6 6 0 0 1|6890M 32.1M 1001M 42.8M|2.36 2.87 1.73| 0 0 | 75k 144k 88 10 2 0 0 0|6889M 32.2M 1002M 41.6M|3.05 3.00 1.78| 0 0 | 60k 102k 89 9 2 0 0 0|6890M 32.2M 1003M 41.0M|3.05 3.00 1.78| 0 0 | 38k 70k 89 10 1 0 0 0|6890M 32.2M 1003M 40.7M|3.05 3.00 1.78| 0 0 | 26k 24k 93 6 2 0 0 0|6890M 32.2M 1003M 40.9M|3.05 3.00 1.78| 0 0 | 37k 31k 90 8 2 0 0 0|6890M 32.2M 1003M 39.9M|3.05 3.00 1.78| 0 0 | 67k 69k 87 8 4 0 0 1|6890M 32.2M 1004M 38.7M|4.09 3.22 1.85| 0 0 | 123k 262k 83 13 2 0 0 2|6890M 32.2M 1004M 38.3M|4.09 3.22 1.85| 0 0 | 445k 18M 90 6 3 0 0 0|6890M 32.2M 1005M 38.2M|4.09 3.22 1.85| 0 0 | 72k 91k 40 7 25 27 0 0|6890M 32.2M 1005M 37.8M|4.09 3.22 1.85| 0 0 | 246k 8034k 0 0 59 41 0 0|6890M 32.2M 1005M 37.7M|4.09 3.22 1.85| 0 0 | 19k 6490B 1 2 45 52 0 0|6891M 32.2M 999M 43.1M|4.00 3.21 1.86| 0 0 | 29k 18k 72 8 15 3 0 1|6892M 32.2M 999M 41.6M|4.00 3.21 1.86| 0 0 | 431k 11M 88 9 2 0 0 1|6907M 32.0M 985M 41.1M|4.00 3.21 1.86| 0 0 | 99k 77k 88 10 1 0 0 1|6913M 31.9M 977M 44.1M|4.00 3.21 1.86| 0 0 | 112k 619k 89 9 1 0 0 1|6892M 31.9M 977M 64.4M|4.00 3.21 1.86| 0 0 | 109k 369k 90 8 1 0 0 0|6892M 31.9M 979M 62.5M|4.80 3.39 1.92| 0 0 | 130k 97k 83 13 1 0 0 3|6893M 32.0M 981M 59.8M|4.80 3.39 1.92| 0 0 | 503k 18M 78 11 10 0 0 0|6893M 32.0M 981M 59.5M|4.80 3.39 1.92| 0 0 | 102k 110k The low cpu periods are due to major GC (JVM frozen). TPStats do show activity on the HH. I'll have some examples latter if the nodes decide to do this again. Finally hinted_handoff_throttle_delay_in_ms in conf/cassandra.yaml will let you slow down the delivery rate if HH is indeed the problem. Best, Gabriel
Re: nodetool move in cassandra
nodetool move decommissions the node then bootstraps it back . I've not checked the code but decommission will only send the data it needs to the other nodes, and I believe bootstrap can do that same but I am not sure. http://wiki.apache.org/cassandra/Operations#Moving_nodes Aaron - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 13 May 2011, at 12:22, Anurag Gujral wrote: Hi All, I want to change the token for my cassandra node from T to T+1. It appears that cassandra in this case brings all the data from the other nodes(which basically mean i can only do move token if RF is 1 for my cluster) . Does cassandra node whose token I am changing looks at the data stored locally also since it can get that data locally. Thanks Anurag
Re: Cassandra node is not blanced Rf=2 Random Partitioner
On Fri, May 13, 2011 at 2:46 AM, Ali Ahsan ali.ah...@panasiangroup.com wrote: My cluster is unbalanced.One have 99 GB Data and other have 87 GB can any one explain why this is happening. They are pretty close ... since a row key is pinned to a node - it is possible that you have a really large row(s) on your .4 node. Sorry I dont have more info, but that is my best guess. -Eric
Re: assertion error in cassandra when doing nodetool move
I think you may have a data centre defined in the strategy_options for the keyspace that does not have any nodes in it. Check the dc names in the create keyspace statement match the names in your cassandra-topology.propeties file (assuming you are using the PropertyFileSnitch), or that you have nodes assigned to all dc's. Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 13 May 2011, at 13:45, Anurag Gujral wrote: Hi All, I run following command on one of my nodes to move the token from 0 to 2. /usr/cassandra/cassandra/bin/nodetool -h 10.170.195.204 -p 8080 move 2. I dont understand why is this happening? I am getting the following assertion error: Exception in thread main java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:389) at org.apache.cassandra.locator.TokenMetadata.ringIterator(TokenMetadata.java:414) at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:94) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895) at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1595) at org.apache.cassandra.service.StorageService.move(StorageService.java:1733) at org.apache.cassandra.service.StorageService.move(StorageService.java:1708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Thanks Anurag
Re: Cassandra node is not blanced Rf=2 Random Partitioner
You need to manually assign the tokens to the nodes http://wiki.apache.org/cassandra/Operations#Load_balancing One should be 0 and the other 85070591730234615865843651857942052864 Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 13 May 2011, at 22:50, Eric tamme wrote: On Fri, May 13, 2011 at 2:46 AM, Ali Ahsan ali.ah...@panasiangroup.com wrote: My cluster is unbalanced.One have 99 GB Data and other have 87 GB can any one explain why this is happening. They are pretty close ... since a row key is pinned to a node - it is possible that you have a really large row(s) on your .4 node. Sorry I dont have more info, but that is my best guess. -Eric
Re: Hinted Handoff
HH will be stored into one of live replica node. It is just a hint, rather than data to be replicated. maki 2011/5/12 Anurag Gujral anurag.guj...@gmail.com: Hi All, I have two questions: a) Is there a way to turn on and off hinted handoff per keyspace rather than for multiple keyspaces. b)It looks like cassandra stores hinted handoff data in one row.Is it true? .Does having one row for hinted handoff implies if nodes are down for longer period of time not all the data which needs to be replicated will be on the node which is alive. Thanks Anurag
Re: Import/Export of Schema Migrations
Actually, I want a way to propagate *any* changes from development to staging to production, but schema changes are the most important. Could I use 2221 to propagate schema changes by deleting the schema in the target cluster, doing show schema in the source cluster, redirecting to a file, and running the file as a script in the target cluster? Of course, I would have to delete the files of dropped CFs by hand (something I *hate* to do, because I'm afraid of making a mistake), but it would be a big improvement. I am open to any other ideas of how to propagate changes from one cluster to another in an efficient non-error-prone fashion. Our development environment (i.e. development, staging, production) is pretty standard, so I'm sure that I'm not the only one with this problem! On Fri, May 13, 2011 at 12:51 PM, aaron morton aa...@thelastpickle.comwrote: What sort of schema changes are you making? can you manage them as a CLI script under source control ? You may also be interested in CASSANDRA-2221. Cheers Aaron - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 12 May 2011, at 20:45, David Boxenhorn wrote: My use case is like this: I have a development cluster, a staging cluster and a production cluster. When I finish a set of migrations (i.e. changes) on the development cluster, I want to apply them to the staging cluster, and eventually the production cluster. I don't want to do it by hand, because it's a painful and error-prone process. What I would like to do is export the last N migrations from the development cluster as a text file, with exactly the same format as the original text commands, and import them to the staging and production clusters. I think the best place to do this might be the CLI, since you would probably want to view your migrations before exporting them. Something like this: show migrations N;Shows the last N migrations. export migrations N fileName; Exports the last N migrations to file fileName. import migrations fileName; Imports migrations from fileName. The import process would apply the migrations one at a time giving you feedback like, applying migration: update column family If a migration fails, the process should give an appropriate message and stop. Is anyone else interested in this? I have created a Jira ticket for it here: https://issues.apache.org/jira/browse/CASSANDRA-2636
Re: Inter node communication over UDP
UDP is a bad fit for Cassandra; you'd have to implement your own ordering and robustness on top of it. Better to just use TCP. On Fri, May 13, 2011 at 2:55 AM, pankajsoni0126 pankajsoni0...@gmail.com wrote: we are deploying cassandra over a huge cluster size and we think that inter node communication can become overload on nodes as the protocol used is TCP. Similarly gossip was earlier using udp but has been changed to use Tcp. Is there added advantage for using tcp over udp in cassandra both in gossip and data storage(internal storage and retrival, etc)? And how much development time i should expect to code udp? -pankaj -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Inter-node-communication-over-UDP-tp6358459p6358459.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: How to invoke getNaturalEndpoints with jconsole?
I wrote a small JMX client to invoke getNaturalEndpoints. It works fine at my test environment, but throws NPE for keyspace we will use for our application (both 0.7.5). Does anyone know quick resolution of that before I setting up cassandra on eclipse to inspect what happens :) thanks Exception in thread main javax.management.RuntimeMBeanException: java.lang.NullPointerException at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:877) [snip] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993) at my.test.getNaturalEndpoints.main(getNaturalEndpoints.java:32) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:266) at org.apache.cassandra.db.Table.init(Table.java:212) at org.apache.cassandra.db.Table.open(Table.java:106) at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1497) [snip] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 2011/5/11 Jonathan Ellis jbel...@gmail.com: Thanks! On Wed, May 11, 2011 at 10:20 AM, Maki Watanabe watanabe.m...@gmail.com wrote: Add a new faq: http://wiki.apache.org/cassandra/FAQ#jconsole_array_arg 2011/5/11 Nick Bailey n...@datastax.com: Yes. On Wed, May 11, 2011 at 8:25 AM, Maki Watanabe watanabe.m...@gmail.com wrote: Thanks, So my options are: 1. Write a thrift client code to call describe_ring with hashed key or 2. Write a JMX client code to call getNaturalEndpoints right? 2011/5/11 Nick Bailey n...@datastax.com: As far as I know you can not call getNaturalEndpoints from jconsole because it takes a byte array as a parameter and jconsole doesn't provide a way for inputting a byte array. You might be able to use the thrift call 'describe_ring' to do what you want though. You will have to manually hash your key to see what range it falls in however. On Wed, May 11, 2011 at 6:14 AM, Maki Watanabe watanabe.m...@gmail.com wrote: Hello, It's a question on jconsole rather than cassandra, how can I invoke getNaturalEndpoints with jconsole? org.apache.cassandra.service.StorageService.Operations.getNaturalEndpoints I want to run this method to find nodes which are responsible to store data for specific row key. I can find this method on jconsole but I can't invoke it because the button is gray out and doesn't accept click. Thanks, -- maki -- w3m -- w3m -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- w3m Exception in thread main javax.management.RuntimeMBeanException: java.lang.NullPointerException at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:877) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:890) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:859) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255) at
[howto measure disk usage]
cassandra-people,
I'm trying to measure disk usage by cassandra after inserting some columns
in order to plan disk sizes and configurations for future deploys.
My approach is very straightforward:
clean_data (stop_cassandra rm -rf
/var/lib/cassandra/{dara,commitlog,saved_caches}/*)
perform_inserts
measure_disk_usage (nodetool -flush du -ch /var/lib/cassandra)
There are two types of inserts:
- In a simple column with key, name and value a random string of size 100
- In a super-column with key, super-column-name, name and value a random
string of size 100
But surprisingly when I'm inserting 100 million columns on a simple column
it uses more disk than the same amount in a super-column. How can that be
possible?
For simple column 41984 MB and for super-column 29696, the difference is
more than noticeable!
Somebody told me yesterday that super-columns don't have a per-column
timestamp, but... it in my case, even if every data was in the same
super-column-key it will not explain the difference!
ps: sorry, English is not my first language
results.eps
Description: PostScript document
Re: assertion error in cassandra when doing nodetool move
I checked file cassandra-topology.properties everything looks good to me. Please Advise on next steps I can do. Thanks Anurag On Fri, May 13, 2011 at 3:52 AM, aaron morton aa...@thelastpickle.comwrote: I think you may have a data centre defined in the strategy_options for the keyspace that does not have any nodes in it. Check the dc names in the create keyspace statement match the names in your cassandra-topology.propeties file (assuming you are using the PropertyFileSnitch), or that you have nodes assigned to all dc's. Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 13 May 2011, at 13:45, Anurag Gujral wrote: Hi All, I run following command on one of my nodes to move the token from 0 to 2. /usr/cassandra/cassandra/bin/nodetool -h 10.170.195.204 -p 8080 move 2. I dont understand why is this happening? I am getting the following assertion error: Exception in thread main java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:389) at org.apache.cassandra.locator.TokenMetadata.ringIterator(TokenMetadata.java:414) at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:94) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895) at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1595) at org.apache.cassandra.service.StorageService.move(StorageService.java:1733) at org.apache.cassandra.service.StorageService.move(StorageService.java:1708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Thanks Anurag
Re: How to invoke getNaturalEndpoints with jconsole?
On 5/13/11 10:08 AM, Maki Watanabe wrote: I wrote a small JMX client to invoke getNaturalEndpoints. It works fine at my test environment, but throws NPE for keyspace we will use for our application (both 0.7.5). Does anyone know quick resolution of that before I setting up cassandra on eclipse to inspect what happens :) thanks Exception in thread main javax.management.RuntimeMBeanException: java.lang.NullPointerException at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:877) [snip] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993) at my.test.getNaturalEndpoints.main(getNaturalEndpoints.java:32) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:266) at org.apache.cassandra.db.Table.init(Table.java:212) at org.apache.cassandra.db.Table.open(Table.java:106) at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1497) [snip] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Did you by chance see this after dropping the keyspace? I believe I've seen this as well. If so (and if I'm interpreting the stack trace and code correctly) it might be related to queuing an op for a keyspace that's been dropped without checking if its metadata is null rather than your code.
Re: Excessive allocation during hinted handoff
The number of Completed HH tasks is interesting. AFAIK a task is started when the node detects another in the cluster has returned. Were you doing some other restarts around the cluster ? Not at all. The restarts seem to happen as normal operation. I don't want to divert from the GC issue, just wondering if something else is going on as well. Like the node is been asked to record a lot of hints. NP, this is another issue that a bit weird. Nodes goes MIA and then return few seconds latter. Not sure why. In other news, I've discovered than one of the nodes had some corruption in one of the SSTables - some 1TB record. I'm looking into cleaning up the data and monitoring the nodes. Gabriel
Re: Commitlog Disk Full
Is there a way to look at the actual size of memtable? Would that help? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Commitlog-Disk-Full-tp6356797p6360001.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Crash when uploading large data sets
It is a 64 bit VM. I didn't notice the hs_err_pid.log files since I'm staring over ssh, so they're in my home directory instead of my working directory. I've attached one of those below. I don't know much about Java, so I'm not sure how to interpret this file. # # An unexpected error has been detected by Java Runtime Environment: # # Internal Error (nmethod.cpp:1707), pid=10068, tid=1085823312 # Error: guarantee(cont_offset != 0,unhandled implicit exception in compiled code) # # Java VM: OpenJDK 64-Bit Server VM (1.6.0_0-b11 mixed mode linux-amd64) # If you would like to submit a bug report, please visit: # http://icedtea.classpath.org/bugzilla # --- T H R E A D --- Current thread (0x7f7e441a9c00): JavaThread EXPIRING-MAP-TIMER-1 daemon [_thread_in_Java, id=10095, stack(0x40b65000,0x40b86000)] Stack: [0x40b65000,0x40b86000], sp=0x40b83eb0, free space=123k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x5d198f] V [libjvm.so+0x5d1cf1] V [libjvm.so+0x27e506] V [libjvm.so+0x4970d6] V [libjvm.so+0x514cd9] V [libjvm.so+0x4b0b64] --- P R O C E S S --- Java Threads: ( = current thread ) 0x02913c00 JavaThread NonPeriodicTasks:1 [_thread_blocked, id=10233, stack(0x401fc000,0x4021d000)] 0x026eec00 JavaThread FILEUTILS-DELETE-POOL:1 [_thread_blocked, id=10222, stack(0x42533000,0x42554000)] 0x7f7e3c01c000 JavaThread Thread-22 [_thread_in_native, id=10212, stack(0x42512000,0x42533000)] 0x0253c000 JavaThread Thread-21 [_thread_in_native, id=10211, stack(0x40dfc000,0x40e1d000)] 0x0253a800 JavaThread Thread-20 [_thread_in_native, id=10210, stack(0x40ddb000,0x40dfc000)] 0x02537400 JavaThread Thread-19 [_thread_in_native, id=10209, stack(0x424f1000,0x42512000)] 0x02725c00 JavaThread pool-1-thread-1 [_thread_in_native, id=10208, stack(0x4007d000,0x4009e000)] 0x026f7400 JavaThread Thread-18 [_thread_in_native, id=10207, stack(0x4034f000,0x4037)] 0x02902800 JavaThread Thread-17 [_thread_in_native, id=10206, stack(0x40d6b000,0x40d8c000)] 0x02901400 JavaThread Thread-16 [_thread_in_native, id=10205, stack(0x424d,0x424f1000)] 0x02613c00 JavaThread Thread-15 [_thread_in_native, id=10204, stack(0x4024e000,0x4026f000)] 0x026ad800 JavaThread Thread-14 [_thread_in_native, id=10203, stack(0x40d11000,0x40d32000)] 0x0276f000 JavaThread Thread-13 [_thread_in_native, id=10202, stack(0x424af000,0x424d)] 0x026b2c00 JavaThread Thread-12 [_thread_in_native, id=10201, stack(0x4049,0x404b1000)] 0x026aec00 JavaThread Thread-11 [_thread_in_native, id=10200, stack(0x4248e000,0x424af000)] 0x0254ec00 JavaThread Thread-10 [_thread_in_native, id=10199, stack(0x4246d000,0x4248e000)] 0x0254d000 JavaThread Thread-9 [_thread_in_native, id=10198, stack(0x4244c000,0x4246d000)] 0x02505000 JavaThread Thread-8 [_thread_in_native, id=10197, stack(0x4242b000,0x4244c000)] 0x02502400 JavaThread Thread-7 [_thread_in_native, id=10196, stack(0x4240a000,0x4242b000)] 0x02500400 JavaThread WRITE-/172.19.149.80 [_thread_blocked, id=10195, stack(0x407dd000,0x407fe000)] 0x024ff000 JavaThread WRITE-/172.19.149.80 [_thread_blocked, id=10194, stack(0x40c8e000,0x40caf000)] 0x024f4400 JavaThread WRITE-/172.19.149.64 [_thread_blocked, id=10193, stack(0x423e9000,0x4240a000)] 0x024f3000 JavaThread WRITE-/172.19.149.64 [_thread_blocked, id=10192, stack(0x423c8000,0x423e9000)] 0x024f1400 JavaThread WRITE-/172.19.149.71 [_thread_blocked, id=10191, stack(0x40a2d000,0x40a4e000)] 0x024f JavaThread WRITE-/172.19.149.71 [_thread_blocked, id=10190, stack(0x423a7000,0x423c8000)] 0x024ee400 JavaThread WRITE-/172.19.149.62 [_thread_blocked, id=10189, stack(0x42386000,0x423a7000)] 0x026bf800 JavaThread WRITE-/172.19.149.62 [_thread_blocked, id=10188, stack(0x42365000,0x42386000)] 0x026bdc00 JavaThread WRITE-/172.19.149.72 [_thread_blocked, id=10187, stack(0x40dba000,0x40ddb000)] 0x026bc400 JavaThread WRITE-/172.19.149.72 [_thread_blocked, id=10186, stack(0x42344000,0x42365000)] 0x026bac00 JavaThread WRITE-/172.19.149.63 [_thread_blocked, id=10185, stack(0x40431000,0x40452000)] 0x026b9400 JavaThread
Converting separate clusters in mutliple dc to one cluster across multiple dcs
Hi All, I have 3 separate cassandra clusters running in multiple data centers which I want to convert to one cassandra cluster across multiple data centers Does anyone tried this? If so what are the steps: I think I need to do the following: a)Change cluster name: What is the procedure to do this? b)Change tokens so that each node have a unique token c)Change seed nodes so that machines in one data center know about machines in another data center. Thanks Anurag
Ghost token
Has anyone seen this and know if it is causing an issue or how to fix it? Anytime I run nodetool ring (on any node) I get this extra token show up on the top of the list. Address Status State LoadOwnsToken 113427455640312821154458202477256070484 10.1.2.171 Up Normal 106.72 KB 33.33% 0 10.1.2.172 Up Normal 101.7 KB33.33% 56713727820156410577229101238628035242 10.1.2.173 Up Normal 104.63 KB 33.33% 113427455640312821154458202477256070484
Re: Ghost token
That's the same as the last one. The token space is a circle so the last one at the list is repeated at the top. -ryan On Fri, May 13, 2011 at 9:59 AM, Scott McPheeters smcpheet...@healthx.com wrote: Has anyone seen this and know if it is causing an issue or how to fix it? Anytime I run nodetool ring (on any node) I get this extra token show up on the top of the list. Address Status State Load Owns Token 113427455640312821154458202477256070484 10.1.2.171 Up Normal 106.72 KB 33.33% 0 10.1.2.172 Up Normal 101.7 KB 33.33% 56713727820156410577229101238628035242 10.1.2.173 Up Normal 104.63 KB 33.33% 113427455640312821154458202477256070484
RE: Ghost token
Ahh, thanks. I got questioned on that as soon as our test environment came up and I had no idea. Thanks for the reply. Scott -Original Message- From: Ryan King [mailto:r...@twitter.com] Sent: Friday, May 13, 2011 1:03 PM To: user@cassandra.apache.org Subject: Re: Ghost token That's the same as the last one. The token space is a circle so the last one at the list is repeated at the top. -ryan On Fri, May 13, 2011 at 9:59 AM, Scott McPheeters smcpheet...@healthx.com wrote: Has anyone seen this and know if it is causing an issue or how to fix it? Anytime I run nodetool ring (on any node) I get this extra token show up on the top of the list. Address Status State Load Owns Token 113427455640312821154458202477256070484 10.1.2.171 Up Normal 106.72 KB 33.33% 0 10.1.2.172 Up Normal 101.7 KB 33.33% 56713727820156410577229101238628035242 10.1.2.173 Up Normal 104.63 KB 33.33% 113427455640312821154458202477256070484
Re: assertion error in cassandra when doing nodetool move
I am using network Topology strategy for one of my keyspaces. Thanks Anurag On Thu, May 12, 2011 at 6:45 PM, Anurag Gujral anurag.guj...@gmail.comwrote: Hi All, I run following command on one of my nodes to move the token from 0 to 2. /usr/cassandra/cassandra/bin/nodetool -h 10.170.195.204 -p 8080 move 2. I dont understand why is this happening? I am getting the following assertion error: Exception in thread main java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:389) at org.apache.cassandra.locator.TokenMetadata.ringIterator(TokenMetadata.java:414) at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:94) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895) at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1595) at org.apache.cassandra.service.StorageService.move(StorageService.java:1733) at org.apache.cassandra.service.StorageService.move(StorageService.java:1708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Thanks Anurag
Re: How to invoke getNaturalEndpoints with jconsole?
sounds like https://issues.apache.org/jira/browse/CASSANDRA-2353 On Fri, May 13, 2011 at 10:08 AM, Maki Watanabe watanabe.m...@gmail.com wrote: I wrote a small JMX client to invoke getNaturalEndpoints. It works fine at my test environment, but throws NPE for keyspace we will use for our application (both 0.7.5). Does anyone know quick resolution of that before I setting up cassandra on eclipse to inspect what happens :) thanks Exception in thread main javax.management.RuntimeMBeanException: java.lang.NullPointerException at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:877) [snip] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993) at my.test.getNaturalEndpoints.main(getNaturalEndpoints.java:32) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:266) at org.apache.cassandra.db.Table.init(Table.java:212) at org.apache.cassandra.db.Table.open(Table.java:106) at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1497) [snip] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 2011/5/11 Jonathan Ellis jbel...@gmail.com: Thanks! On Wed, May 11, 2011 at 10:20 AM, Maki Watanabe watanabe.m...@gmail.com wrote: Add a new faq: http://wiki.apache.org/cassandra/FAQ#jconsole_array_arg 2011/5/11 Nick Bailey n...@datastax.com: Yes. On Wed, May 11, 2011 at 8:25 AM, Maki Watanabe watanabe.m...@gmail.com wrote: Thanks, So my options are: 1. Write a thrift client code to call describe_ring with hashed key or 2. Write a JMX client code to call getNaturalEndpoints right? 2011/5/11 Nick Bailey n...@datastax.com: As far as I know you can not call getNaturalEndpoints from jconsole because it takes a byte array as a parameter and jconsole doesn't provide a way for inputting a byte array. You might be able to use the thrift call 'describe_ring' to do what you want though. You will have to manually hash your key to see what range it falls in however. On Wed, May 11, 2011 at 6:14 AM, Maki Watanabe watanabe.m...@gmail.com wrote: Hello, It's a question on jconsole rather than cassandra, how can I invoke getNaturalEndpoints with jconsole? org.apache.cassandra.service.StorageService.Operations.getNaturalEndpoints I want to run this method to find nodes which are responsible to store data for specific row key. I can find this method on jconsole but I can't invoke it because the button is gray out and doesn't accept click. Thanks, -- maki -- w3m -- w3m -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- w3m -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Inconsistent data issues when running nodetool move.
Hi, I'm running Cassandra (0.7.4) on a 4 node ring. It was a 3 node ring, but we ended up expanding it to 4... So then I followed the many suggestions to rebalance the ring. I found a script that suggested I use: # ~/nodes_calc.py How many nodes are in your cluster? 4 node 0: 0 node 1: 42535295865117307932921825928971026432 node 2: 85070591730234615865843651857942052864 node 3: 127605887595351923798765477786913079296 So I started to migrate each node to those tokens. I have my replication factor set to 2, so I guess I was expecting to be able to continue to use this ring without issues. But it seems that the node still accepts writes while it's decommissioning? I say this because if I interrupt the decommission by stopping Cassandra and starting it again, it appears to run through several commit logs. And as soon as it's through with those commit logs, I no longer get consistency issues. The issue I'm seeing is that writes to this ring will succeed, but it's possible for a subsequent read to return an older object. For several minutes even. I'm not sure if I did something wrong... learning as I go here and this list archive has been very useful. But, is there anyway I can rebalance the node and get better consistency? Thanks, Ryan
Re: How to invoke getNaturalEndpoints with jconsole?
I did not drop the keyspace, but your comment lead me to resolution. I found cassandra-cli is not case sensitive on keyspace. I used keyspace name FooBar on cassandra-cli, but it was Foobar in correct. cassandra-cli didn't complain on my mistake, but the JMX interface is less tolerance. If I use correct name, the tool runs fine. Thanks. 2011/5/13 Alex Araujo cassandra-us...@alex.otherinbox.com: On 5/13/11 10:08 AM, Maki Watanabe wrote: I wrote a small JMX client to invoke getNaturalEndpoints. It works fine at my test environment, but throws NPE for keyspace we will use for our application (both 0.7.5). Does anyone know quick resolution of that before I setting up cassandra on eclipse to inspect what happens :) thanks Exception in thread main javax.management.RuntimeMBeanException: java.lang.NullPointerException at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:877) [snip] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993) at my.test.getNaturalEndpoints.main(getNaturalEndpoints.java:32) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:266) at org.apache.cassandra.db.Table.init(Table.java:212) at org.apache.cassandra.db.Table.open(Table.java:106) at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1497) [snip] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Did you by chance see this after dropping the keyspace? I believe I've seen this as well. If so (and if I'm interpreting the stack trace and code correctly) it might be related to queuing an op for a keyspace that's been dropped without checking if its metadata is null rather than your code. -- w3m
problems using property snitch file
Hi All, I have built a cassandra cluster which is using property snitch file to specify my network topology to cassandra.The keyspace I have use network topology strategy. When I tried to add new node to this cluster,I get the following error: RROR [main] 2011-05-13 23:45:44,152 AbstractCassandraDaemon.java (line 333) Exception encountered during startup. java.lang.NullPointerException at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:96) at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:87) at org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:196) at org.apache.cassandra.locator.AbstractReplicationStrategy.getPendingAddressRanges(AbstractReplicationStrategy.java:230) at org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:195) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:403) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) Please Advise. Thanks Anurag
Re: Crash when uploading large data sets
You should upgrade to the latest Sun JVM. OpenJDK is almost a year behind in bug fixes. On Fri, May 13, 2011 at 11:40 AM, James Cipar jci...@cmu.edu wrote: It is a 64 bit VM. I didn't notice the hs_err_pid.log files since I'm staring over ssh, so they're in my home directory instead of my working directory. I've attached one of those below. I don't know much about Java, so I'm not sure how to interpret this file. # # An unexpected error has been detected by Java Runtime Environment: # # Internal Error (nmethod.cpp:1707), pid=10068, tid=1085823312 # Error: guarantee(cont_offset != 0,unhandled implicit exception in compiled code) # # Java VM: OpenJDK 64-Bit Server VM (1.6.0_0-b11 mixed mode linux-amd64) # If you would like to submit a bug report, please visit: # http://icedtea.classpath.org/bugzilla # --- T H R E A D --- Current thread (0x7f7e441a9c00): JavaThread EXPIRING-MAP-TIMER-1 daemon [_thread_in_Java, id=10095, stack(0x40b65000,0x40b86000)] Stack: [0x40b65000,0x40b86000], sp=0x40b83eb0, free space=123k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x5d198f] V [libjvm.so+0x5d1cf1] V [libjvm.so+0x27e506] V [libjvm.so+0x4970d6] V [libjvm.so+0x514cd9] V [libjvm.so+0x4b0b64] --- P R O C E S S --- Java Threads: ( = current thread ) 0x02913c00 JavaThread NonPeriodicTasks:1 [_thread_blocked, id=10233, stack(0x401fc000,0x4021d000)] 0x026eec00 JavaThread FILEUTILS-DELETE-POOL:1 [_thread_blocked, id=10222, stack(0x42533000,0x42554000)] 0x7f7e3c01c000 JavaThread Thread-22 [_thread_in_native, id=10212, stack(0x42512000,0x42533000)] 0x0253c000 JavaThread Thread-21 [_thread_in_native, id=10211, stack(0x40dfc000,0x40e1d000)] 0x0253a800 JavaThread Thread-20 [_thread_in_native, id=10210, stack(0x40ddb000,0x40dfc000)] 0x02537400 JavaThread Thread-19 [_thread_in_native, id=10209, stack(0x424f1000,0x42512000)] 0x02725c00 JavaThread pool-1-thread-1 [_thread_in_native, id=10208, stack(0x4007d000,0x4009e000)] 0x026f7400 JavaThread Thread-18 [_thread_in_native, id=10207, stack(0x4034f000,0x4037)] 0x02902800 JavaThread Thread-17 [_thread_in_native, id=10206, stack(0x40d6b000,0x40d8c000)] 0x02901400 JavaThread Thread-16 [_thread_in_native, id=10205, stack(0x424d,0x424f1000)] 0x02613c00 JavaThread Thread-15 [_thread_in_native, id=10204, stack(0x4024e000,0x4026f000)] 0x026ad800 JavaThread Thread-14 [_thread_in_native, id=10203, stack(0x40d11000,0x40d32000)] 0x0276f000 JavaThread Thread-13 [_thread_in_native, id=10202, stack(0x424af000,0x424d)] 0x026b2c00 JavaThread Thread-12 [_thread_in_native, id=10201, stack(0x4049,0x404b1000)] 0x026aec00 JavaThread Thread-11 [_thread_in_native, id=10200, stack(0x4248e000,0x424af000)] 0x0254ec00 JavaThread Thread-10 [_thread_in_native, id=10199, stack(0x4246d000,0x4248e000)] 0x0254d000 JavaThread Thread-9 [_thread_in_native, id=10198, stack(0x4244c000,0x4246d000)] 0x02505000 JavaThread Thread-8 [_thread_in_native, id=10197, stack(0x4242b000,0x4244c000)] 0x02502400 JavaThread Thread-7 [_thread_in_native, id=10196, stack(0x4240a000,0x4242b000)] 0x02500400 JavaThread WRITE-/172.19.149.80 [_thread_blocked, id=10195, stack(0x407dd000,0x407fe000)] 0x024ff000 JavaThread WRITE-/172.19.149.80 [_thread_blocked, id=10194, stack(0x40c8e000,0x40caf000)] 0x024f4400 JavaThread WRITE-/172.19.149.64 [_thread_blocked, id=10193, stack(0x423e9000,0x4240a000)] 0x024f3000 JavaThread WRITE-/172.19.149.64 [_thread_blocked, id=10192, stack(0x423c8000,0x423e9000)] 0x024f1400 JavaThread WRITE-/172.19.149.71 [_thread_blocked, id=10191, stack(0x40a2d000,0x40a4e000)] 0x024f JavaThread WRITE-/172.19.149.71 [_thread_blocked, id=10190, stack(0x423a7000,0x423c8000)] 0x024ee400 JavaThread WRITE-/172.19.149.62 [_thread_blocked, id=10189, stack(0x42386000,0x423a7000)] 0x026bf800 JavaThread WRITE-/172.19.149.62 [_thread_blocked, id=10188, stack(0x42365000,0x42386000)] 0x026bdc00 JavaThread WRITE-/172.19.149.72 [_thread_blocked, id=10187, stack(0x40dba000,0x40ddb000)] 0x026bc400 JavaThread WRITE-/172.19.149.72 [_thread_blocked,
Re: Commitlog Disk Full
our write happen in bursts. So often times, clients write data as fast as they can. Conceivably one can write 5G in one hour. The other setting that we have is that our replication factor is 3 and we write using QUORUM. Not sure if that will affect things. On Fri, May 13, 2011 at 12:04 AM, Peter Schuller peter.schul...@infidyne.com wrote: I haven't explictly set a value for the memtable_flush_after_mins parameter. Looks like the default is 60minutes. I will try to play around this value to see if that fixes things. Is the amount of data in the commit log consistent with what you might have been writing during 60 minutes? Including overwrites. If not, I'm not sure what's going on. Since you said it took about a day of traffic it feels fishy. -- / Peter Schuller
Re: Commitlog Disk Full
5G in one hour is actually very low. Something else is wrong. Peter pointed to something related to memtable size could be causing this problem, can you turn down memtable_throughput and see if that helps. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Commitlog-Disk-Full-tp6356797p6362301.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
