Re: Cassandra event notification on INSERT/DELETE of records
+1 to what Eric said, a queue is a classic C* anti-pattern. Something like Kafka or RabbitMQ might fit your use case better. Mark On 24 May 2016 at 18:03, Eric Stevenswrote: > It sounds like you're trying to build a queue in Cassandra, which is one > of the classic anti-pattern use cases for Cassandra. > > You may be able to do something clever with triggers, but I highly > recommend you look at purpose-built queuing software such as Kafka to solve > this instead. > > On Tue, May 24, 2016 at 9:49 AM Aaditya Vadnere wrote: > >> Hi experts, >> >> We are evaluating Cassandra as messaging infrastructure for a project. >> >> In our workflow Cassandra database will be synchronized across two nodes, >> a component will INSERT/UPDATE records on one node and another component >> (who has registered for the specific table) on second node will get >> notified of record change. >> >> The second component will then try to read the database to find out the >> specific message. >> >> Is it possible for Cassandra to support such workflow? Basically, is >> there a way for Cassandra to generate a notification anytime schema changes >> (so we can set processes to listen for schema changes). As I understand, >> polling the database periodically or database triggers might work but they >> are costly operations. >> >> >> -- >> Aaditya Vadnere >> >
Re: Record deletion job
No, this is what TTL is for. If you want to schedule a cron to delete data this will need to be an external task. On 19 June 2015 at 07:41, anil_ah anil...@yahoo.co.in wrote: Hi Is their any way to schedule a job in cassandra to delete the recrods which are older than a specific time period. Excluding the option of TTL. Regards Anil Sent from Samsung Mobile
Re: connection refused error for cassandra-cli
Have you configured the rpc_address to listen on that ip? You can check your nodes casandra.yaml for the rpc_address setting. On a side node why are you testing on such an old version of Cassandra? The latest versions of 1.2.x is 1.2.19. There are also two branches of 2.x which are 2.0.15 and 2.1.5. Mark On 26 May 2015 at 10:38, Mahmood Naderan nt_mahm...@yahoo.com wrote: Hello, I have downloaded cassandra-1.2.0 for a test. The output of ./bin/cassandra and ./bin/cassandra-cli seems to be fine. Please see http://pastebin.com/Tgs7qyjS Next, I created a setup-ycsb.cql and wrote [mahmood@tiger ycsb-0.1.4]$ cat setup-ycsb.cql CREATE KEYSPACE usertable with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options = {replication_factor:2}; use usertable; create column family data with comparator=UTF8Type and default_validation_class=UTF8Type and key_validation_class=UTF8Type; Problem is, when I execute cassandra-cli commnad, I get connection refused error [mahmood@tiger ycsb-0.1.4]$ cassandra-cli -h 192.168.1.5 -f setup-ycsb.cql org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused Everything seems to be normal as I disabled the firewall. So I wonder why I receive that error. Please see this output for more detail http://pastebin.com/Bnd3TNCM Did I miss anything? Regards, Mahmood
Re: java.io.IOException: Failed during snapshot creation
Can you check your logs for any other other error message around the time of the repair? Something to look for would be Error occurred during snapshot phase. Regards, Mark On 25 May 2015 at 14:56, Sachin PK sachinpray...@gmail.com wrote: Hey I'm new to Cassandra ,I have 4 node cluster with each node 16GB VPS, initially I had one seed node, I added one of the existing nodes as seed nodes , restarted nodes one by one after that one of my node went down . I ran nodetool repair on it when I checked the log i could find some errors ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,905 RepairSession.java:303 - [repair #968bce30-02e1-11e5-9310-45191f4c93ae] session completed with the following error java.io.IOException: Failed during snapshot creation. at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.5.jar:2.1.5] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80] ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,907 CassandraDaemon.java:223 - Exception in thread Thread[AntiEntropySessions:3,5,RMI Runtime] java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation. at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.5.jar:2.1.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_80] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80] Caused by: java.io.IOException: Failed during snapshot creation. at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.5.jar:2.1.5] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] ... 3 common frames omitted
Re: Disabling auto snapshots
To disable auto snapshots, set the property auto_snapshot: false in your cassandra.yaml file. Mark On 21 May 2015 at 08:30, Ali Akhtar ali.rac...@gmail.com wrote: Is there a config setting where automatic snapshots can be disabled? I have a use case where a table is truncated quite often, and would like to not have snapshots. I can't find anything on google. Thanks.
Re: Drop/Create table with same CF Name
Yes, it's a known issue. For more information on the topic see this support post from DataStax: https://support.datastax.com/hc/en-us/articles/204226339-How-to-drop-and-recreate-a-table-in-Cassandra-versions-older-than-2-1 Mark On 21 May 2015 at 15:31, Ken Hancock ken.hanc...@schange.com wrote: We've been running into the reused key cache issue (CASSANDRA-5202) with dropping and recreating the same table in Cassandra 1.2.18 so we've been testing with key caches disabled which does not seem to solve the issue. In the latest logs it seems that old SSTables metadata gets read after the tables have been deleted by the previous drop, eventually causing an exception and the Thrift interface shut down. At this point is it a known issue that one CANNOT reuse a table name prior to Cassandra 2.1 ?
Re: Steps to do after schema changes
It's always good to run nodetool describecluster after a schema change, this will show you all the nodes in your cluster and what schema version they have. If they have different versions you have a schema disagreement and should follow this guide to resolution: http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_handle_schema_disagree_t.html Regards, Mark On 12 March 2015 at 05:47, Phil Yang ud1...@gmail.com wrote: Usually, you have nothing to do. Changes will be synced to every nodes automatically. 2015-03-12 13:21 GMT+08:00 Ajay ajay.ga...@gmail.com: Hi, Are there any steps to do (like nodetool or restart node) or any precautions after schema changes are done in a column family say adding a new column or modifying any table properties? Thanks Ajay -- Thanks, Phil Yang
Re: unsubscribe
Raman, To unsubscribe send a mail to user-unsubscr...@cassandra.apache.org On 22 February 2015 at 03:59, Raman ra...@assetcorporation.net wrote: unsubscribe
Re: Why no virtual nodes for Cassandra on EC2?
Hey Clint, Someone for DataStax can correct me here, but I'm assuming that they have disabled vnodes because the AMI is built to make it easy to set up a pre-configured mixed workload cluster. A mixture of Real-Time/Transactional (Cassandra), Analytics (Hadoop), or Search (Solr). If you take a look at the getting started guide for both Hadoop and Solr you will see a paragraph instructing the user to disable vnodes for a mix workload cluster. http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax _enterprise/srch/srchIntro.html http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax _enterprise/ana/anaStrt.html This is specific to the example AMI and that type of workload. This is by no means a warning for users to disable vnodes on their Real-Time/Transactional Cassandra only clusters on EC2. I've used vnodes on EC2 without issue. Regards, Mark On 20 February 2015 at 05:08, Clint Kelly clint.ke...@gmail.com wrote: Hi all, The guide for installing Cassandra on EC2 says that Note: The DataStax AMI does not install DataStax Enterprise nodes with virtual nodes enabled. http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html Just curious why this is the case. It was my understanding that virtual nodes make taking Cassandra nodes on and offline an easier process, and that seems like something that an EC2 user would want to do quite frequently. -Clint
Re: Cancel subscription
Please use user-unsubscr...@cassandra.apache.org to unsubscribe from this mailing list. Thanks Regards, Mark On 19 February 2015 at 09:14, Hilary Albutt - CEO hil...@incrediblesoftwaresolutions.com wrote: Cancel subscription
Re: Node joining take a long time
What is a long time in your scenario? What is the data size in your cluster? I'm sure Rob will be along shortly to say that 2.1.2 is, in his opinion, broken for production use...an opinion I'd agree with. So bare that in mind if you are running a production cluster. Regards, Mark On 19 February 2015 at 12:19, 曹志富 cao.zh...@gmail.com wrote: Hi guys: I have a 20 nodes C* cluster with vnodes,version is 2.1.2. When I add a node to my cluster,it take a long time ,and somes exists node nodetool nestats show this: Mode: NORMAL Unbootstrap cfe03590-b02a-11e4-95c5-b5f6ad9c7711 /172.19.105.49 Receiving 68 files, 23309801005 bytes total I want know ,is there some problem with my cluster? -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/
Re: Cassandra install on JRE vs JDK
Yes you can use Oracle JDK if your prefer, I've been using the JDK with Cassandra in production for years without issue. Regards, Mark On 18 February 2015 at 19:49, cass savy casss...@gmail.com wrote: Can we install Oracle JDK instead of JRE in Cassandra servers? We have few clusters running JDK when we upgraded to C*2.0. Is there any known issue or impact with using JDK vs JRE? What is the reason to not use Oracle JDK in C* servers? Is there any performance impact ? Please advice.
Re: High GC activity on node with 4TB on data
Hey Jiri, While I don't have any experience running 4TB nodes (yet), I would recommend taking a look at a presentation by Arron Morton on large nodes: http://planetcassandra.org/blog/cassandra-community-webinar-videoslides-large-nodes-with-cassandra-by-aaron-morton/ to see if you can glean anything from that. I would note that at the start of his talk he mentions that in version 1.2 we can now talk about nodes around 1 - 3 TB in size, so if you are storing anything more than that you are getting into very specialised use cases. If you could provide us with some more information about your cluster setup (No. of CFs, read/write patterns, do you delete / update often, etc.) that may help in getting you to a better place. Regards, Mark On 8 February 2015 at 21:10, Kevin Burton bur...@spinn3r.com wrote: Do you have a lot of individual tables? Or lots of small compactions? I think the general consensus is that (at least for Cassandra), 8GB heaps are ideal. If you have lots of small tables it’s a known anti-pattern (I believe) because the Cassandra internals could do a better job on handling the in memory metadata representation. I think this has been improved in 2.0 and 2.1 though so the fact that you’re on 1.2.18 could exasperate the issue. You might want to consider an upgrade (though that has its own issues as well). On Sun, Feb 8, 2015 at 12:44 PM, Jiri Horky ho...@avast.com wrote: Hi all, we are seeing quite high GC pressure (in old space by CMS GC Algorithm) on a node with 4TB of data. It runs C* 1.2.18 with 12G of heap memory (2G for new space). The node runs fine for couple of days when the GC activity starts to raise and reaches about 15% of the C* activity which causes dropped messages and other problems. Taking a look at heap dump, there is about 8G used by SSTableReader classes in org.apache.cassandra.io.compress.CompressedRandomAccessReader. Is this something expected and we have just reached the limit of how many data a single Cassandra instance can handle or it is possible to tune it better? Regards Jiri Horky -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: Compacted_at timestamp
Hi Santo, If you are seeing the compacted_at value as timestamp and want to convert it to a human readable date, this is not possible via nodetool. You will always write a script to make the compactionhistory call and then convert the output (the fourth column - compacted_at) to a readable date. If you are seeing something other than an expected timestamp value, can you post an example of what you are getting? Regards, Mark On 8 February 2015 at 13:20, Havere Shanmukhappa, Santhosh santhosh_havereshanmukha...@intuit.com wrote: When I run nodetool compactionhistory command, it displays ‘compacted_at’ timestamp in non-readable format. Any way to read that column in readable format? I am using c*2.0.11 version. Thanks, Santo
Re: Error while starting Cassandra for the first time
Hey Krish, If you want to use these config values rather than comment them out, feel free to post a sample of the config that is giving you trouble, I'd be happy to take a look at it and see if I could see where your issue is. The fact that all the config options are directories may be an indication that there are some values that need to be escaped or the paths need to be quoted. Regards, Mark On 5 February 2015 at 04:29, Krish Donald gotomyp...@gmail.com wrote: I have used the yaml validator but tried to fixed based on error messages , I had to comment data_directories , commitlog_directory and saved_caches directory and after that it worked. Thanks a lot for the help ... On Wed, Feb 4, 2015 at 2:32 PM, Mark Reddy mark.l.re...@gmail.com wrote: INFO 22:17:19 Loading settings from file:/home/csduser/cassandra/ conf/cassandra.yaml ERROR 22:17:20 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml You have an malformed cassandra.yaml config file that is resulting in Cassandra not being able to start. You should be able to validate your yaml file with some online validator such as http://www.yamllint.com/ Regards, Mark On 4 February 2015 at 22:23, Krish Donald gotomyp...@gmail.com wrote: Hi, I am getting below error: Not able to understand why ?? [csduser@master bin]$ ./cassandra -f CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I INFO 22:17:19 Hostname: master.my.com INFO 22:17:19 Loading settings from file:/home/csduser/cassandra/conf/cassandra.yaml ERROR 22:17:20 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:120) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:158) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:133) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96) [apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448) [apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537) [apache-cassandra-2.1.2.jar:2.1.2] Caused by: org.yaml.snakeyaml.scanner.ScannerException: while scanning a simple key; could not found expected ':'; in 'reader', line 33, column 1: # See http://wiki.apache.org/cas ... ^ at org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:460) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:558) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105) ~[snakeyaml-1.11.jar:na
Re: Case Study for Learning Cassandra
Hey Krish, Here is a list of use cases from the industry: http://planetcassandra.org/apache-cassandra-use-cases/ You may find some examples on how people/companies are using and running their C* clusters. Also DataStax provide online training that will get you up to speed on several aspects of C* usage: https://academy.datastax.com/ Regards, Mark On 5 February 2015 at 04:38, Krish Donald gotomyp...@gmail.com wrote: Hi, I am new to Cassandra and have setup 4 nodes Cassandra cluster using VMs. Looking for any case study which I can do to understand the Cassandra Administration and put in my resume as well. Any help is appreciated. Thanks Krish
Re: Newly added column not visible
Hey Saurabh, I can't say that I have experienced this before, however if you can reliably reproduce the issue it would be worth commenting on the JIRA issue you linked to or alternatively creating a new JIRA with as much info (setup, test case, debug logs, etc) as possible. Regards, Mark On 5 February 2015 at 00:50, Saurabh Sethi saurabh_se...@symantec.com wrote: I have a 3 node cluster running Cassandra version 2.1.2. Through my unit test, I am creating a column family with 3 columns, inserting a row, asserting that the values got inserted and then truncating the column family. After that I am adding a fourth column to the column family and inserting a new row with four values but this insertion is failing because it can’t find the newly added column. If I go via debug mode, it works fine but not otherwise. I also tried putting a Thread.sleep() for 10 seconds after adding the new column but to no avail. Anyone has any idea what might be going on here? I see that a jira was filed related to similar issue in May 2014 but they closed it in December 2014 stating not reproducible. https://issues.apache.org/jira/browse/CASSANDRA-7186 Thanks, Saurabh
Re: Error while starting Cassandra for the first time
INFO 22:17:19 Loading settings from file:/home/csduser/cassandra/ conf/cassandra.yaml ERROR 22:17:20 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml You have an malformed cassandra.yaml config file that is resulting in Cassandra not being able to start. You should be able to validate your yaml file with some online validator such as http://www.yamllint.com/ Regards, Mark On 4 February 2015 at 22:23, Krish Donald gotomyp...@gmail.com wrote: Hi, I am getting below error: Not able to understand why ?? [csduser@master bin]$ ./cassandra -f CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I INFO 22:17:19 Hostname: master.my.com INFO 22:17:19 Loading settings from file:/home/csduser/cassandra/conf/cassandra.yaml ERROR 22:17:20 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:120) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:158) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:133) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96) [apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448) [apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537) [apache-cassandra-2.1.2.jar:2.1.2] Caused by: org.yaml.snakeyaml.scanner.ScannerException: while scanning a simple key; could not found expected ':'; in 'reader', line 33, column 1: # See http://wiki.apache.org/cas ... ^ at org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:460) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:558) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481) ~[snakeyaml-1.11.jar:na] at org.yaml.snakeyaml.Yaml.load(Yaml.java:412) ~[snakeyaml-1.11.jar:na] at org.apache.cassandra.config.YamlConfigurationLoader.logConfig(YamlConfigurationLoader.java:126) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:104) ~[apache-cassandra-2.1.2.jar:2.1.2] ... 6 common frames omitted Invalid yaml Fatal configuration error; unable to start. See log for stacktrace. Thanks Krish
Re: Question about use scenario with fulltext search
If you have a list of usernames stored in your cassandra database, how could you find all usernames starting with 'Jo'? Cassandra does not support full text search on its own, if you are looking into DataStax enterprise Cassandra there is an integration with Slor that gives you this functionality. Personally for projects I work on that use Cassandra and require full text search, the necessary data is indexed into Elasticsearch. Or ... if this is not possible, what are you using cassandra for? If you are looking for use cases here is a comprehensive set from companies spanning many industries: http://planetcassandra.org/apache-cassandra-use-cases/ Regards, Mark On 1 February 2015 at 16:05, anton anto...@gmx.de wrote: Hi, I was just reading about cassandra and playing a little with it (using django www.djangoproject.com on the web server). One thing that I realized now is that fulltext search as in a normal sql statement (example): select name from users where name like 'Jo%'; Simply does not work because this functionality does not exist. After reading and googeling and reading ... I still do not understand how I could use a db without this functionality (If I do not want to restrict myself on numerical data). So my question is: If you have a list of usernames stored in your cassandra database, how could you find all usernames starting with 'Jo'? Or ... if this is not possible, what are you using cassandra for? Actually I still did not get the point of how I could use cassandra :-( Anton
Re: A questiion to adding a new data center
Hi Boying, I'm not sure I fully understand your question here, so some clarification may be needed. However, if you are asking what steps need to be performed on the current datacenter or on the new datacenter: Step 1 - Current DC Step 2 - New DC Step 3 - Depending on the snitch you may need to make changes on both the current and new DCs Step 4 - Client config Step 5 - Client config Step 6 - New DC Step 7 - New DC Step 8 - New DC Mark On 21 November 2014 03:27, Lu, Boying boying...@emc.com wrote: Hi, all, I read the document about how to adding a new data center to existing clusters posted at http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html But I have a question: Are all those steps executed *only* at the new adding cluster or on existing clusters also? ( Step 7 is to be executed on the new cluster according to the document). Thanks Boying
Re: [jira] Akhtar Hussain shared a search result with you
I believe you were attempting to share: https://issues.apache.org/jira/browse/CASSANDRA-8352 Your cassandra logs outputs the following: DEBUG [Thrift:4] 2014-11-20 15:36:50,653 CustomTThreadPoolServer.java (line 204) Thrift transport error occurred during processing of message. org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.) Which indicates that your server is under pressure at that moment and points you to look at your server logs for further diagnosis. Mark On 21 November 2014 11:15, Akhtar Hussain (JIRA) j...@apache.org wrote: Akhtar Hussain shared a search result with you - https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=truejqlQuery=reporter+%3D+currentUser%28%29+ORDER+BY+createdDate+DESC We have a Geo-red setup with 2 Data centers having 3 nodes each. When we bring down a single Cassandra node down in DC2 by kill -9 Cassandra-pid, reads fail on DC1 with TimedOutException for a brief amount of time (15-20 sec~). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: max ttl for column
Hi Rajanish, Cassandra imposes a max TTL of 20 years. public static final int MAX_TTL = 20 * 365 * 24 * 60 * 60; // 20 years in seconds See: https://github.com/apache/cassandra/blob/8d8fed52242c34b477d0384ba1d1ce3978efbbe8/src/java/org/apache/cassandra/db/ExpiringCell.java#L37 Mark On 21 November 2014 17:29, Rajanish GJ rajan...@apigee.com wrote: Does hector or cassandra imposes a limit on max ttl value for a column? I am trying to insert record into one of the column family and seeing the following error.. Cassandra version : 1.1.12 Hector : 1.1-4 Any pointers appreciated. me.prettyprint.hector.api.exceptions.HInvalidRequestException: InvalidRequestException(why:ttl is too large. requested (951027277) maximum (63072)) at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:52) ~[hector-core-1.1-4.jar:na] at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:260) ~[hector-core-1.1-4.jar:na] at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113) ~[hector-core-1.1-4.jar:na] at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243) ~[hector-core-1.1-4.jar:na] at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115) ~[hector-core-1.1-4.jar:na] at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163) ~[hector-core-1.1-4.jar:na] at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69) ~[hector-core-1.1-4.jar:na] = *Also tried using cql, and it seems to hangs and not responding.. trying again with few combinations* *INSERT INTO users (key,id) values ('test6','951027277 secs') USING TTL 951027277 ; * Regards Rajanish GJ apigee | rajan...@apigee.com
Re: logging over multi-datacenter
Hi Adil, What Cassandra version are you using? Are you using the default user or a non-default user? Mark On 20 November 2014 08:20, Adil adil.cha...@gmail.com wrote: Hi, We have two data-center, we configured PasswordAuthenticator on each node, we increment the RF of system_auth to the number of nodes (each data-center) as recommended. We can logged-in via cqlsh without problem, but when i stop cassandra on all nodes of a data-center we can't logged in in the other data-center...this error is displayed as output: Bad credentials] message=org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM' from what i understand we should be able to logged in even if there is only one node UP but it seems that has to reach QUORUM consistency level (2 data-center). my question is if the java driver cql uses the same condition and if there is a way to set the consistency level to like LOCAL_ONE. Thanks Adil
Re: logging over multi-datacenter
Hi Adil, When using the default superuser ('cassandra') a consistency level of QUORUM is used. When using other users ONE is used. You are not supposed to use 'cassandra' user directly, except to create another superuser and use that one from that point on. Mark On 20 November 2014 14:40, Adil adil.cha...@gmail.com wrote: cassandra version 2.1.2 with the default user we create another user and with this one we could login even if only one node is up 2014-11-20 15:16 GMT+01:00 Mark Reddy mark.l.re...@gmail.com: Hi Adil, What Cassandra version are you using? Are you using the default user or a non-default user? Mark On 20 November 2014 08:20, Adil adil.cha...@gmail.com wrote: Hi, We have two data-center, we configured PasswordAuthenticator on each node, we increment the RF of system_auth to the number of nodes (each data-center) as recommended. We can logged-in via cqlsh without problem, but when i stop cassandra on all nodes of a data-center we can't logged in in the other data-center...this error is displayed as output: Bad credentials] message=org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM' from what i understand we should be able to logged in even if there is only one node UP but it seems that has to reach QUORUM consistency level (2 data-center). my question is if the java driver cql uses the same condition and if there is a way to set the consistency level to like LOCAL_ONE. Thanks Adil
Re: any way to get nodetool proxyhistograms data for an entire cluster?
To be clear, I don't believe DSE is required to use OpsCenter. My understanding is that it can be used with vanilla Apache Cassandra, but I have never actually tried to do so. Correct Rob, you can use Opscenter with vanilla Apache Cassandra. Mark On 20 November 2014 16:10, Robert Coli rc...@eventbrite.com wrote: On Wed, Nov 19, 2014 at 7:20 PM, Clint Kelly clint.ke...@gmail.com wrote: We have DSE so I can use opscenter. I was just looking for something more precise than the graphs that I get from opscenter. To be clear, I don't believe DSE is required to use OpsCenter. My understanding is that it can be used with vanilla Apache Cassandra, but I have never actually tried to do so. =Rob
Re: What time range does nodetool cfhistograms use?
Hi Clint, The values of SSTables, Write Latency and Read Latency will be reset on node start/restart and after running the cfhistograms command itself. The values of Row Size and Column Count are calculated at startup and then re-evaluated during compaction. Mark On 16 November 2014 17:12, Clint Kelly clint.ke...@gmail.com wrote: Hi all, Over what time range does nodetool cfhistograms operate? I am using Cassandra 2.0.8.39. I am trying to debug some very high 95th and 99th percentile read latencies in an application that I'm working on. I tried running nodetool cfhistograms to get a flavor for the distribution of read latencies and also to see how many SSTables our reads are using, and I saw different results yesterday versus this morning, so I assume the time window is fairly tight (like an hour or so). I vaguely recall Aaron Morton talking about this during the training course I took at Cassandra Summit, but I cannot find my notes and I think the behavior of the tool with regard to time windows changed from version to version. Thanks! -Clint
Re: Netstats 100% streaming
HI Eric, It looks like you are running into CASSANDRA-7878 https://issues.apache.org/jira/browse/CASSANDRA-7878 which is fixed in 2.0.11 / 2.1.1 Mark On 1 November 2014 14:08, Eric Stevens migh...@gmail.com wrote: We've been commissioning some new nodes on a 2.0.10 community edition cluster, and we're seeing streams that look like they're shipping way more data than they ought for individual files during bootstrap. /var/lib/cassandra/data/x/y/x-y-jb-11748-Data.db 3756423/3715409 bytes(101%) sent to /1.2.3.4 /var/lib/cassandra/data/x/y/x-y-jb-11043-Data.db 584745/570432 bytes(102%) sent to /1.2.3.4 /var/lib/cassandra/data/x/z/x-z-jb-525-Data.db 13020828/11141590 bytes(116%) sent to /1.2.3.4 /var/lib/cassandra/data/x/w/x-w-jb-539-Data.db 1044124/51404 bytes(2031%) sent to /1.2.3.4 /var/lib/cassandra/data/x/v/x-v-jb-546-Data.db 971447/22253 bytes(4365%) sent to /1.2.3.4 /var/lib/cassandra/data/x/y/x-y-jb-10404-Data.db 6225920/23215181 bytes(26%) sent to /1.2.3.4 Has anyone else seen something like this, and this something we should be worried about? I haven't been able to find any information about this symptom. -Eric
Re: EC2 Snitch load imbalance
Oleg, If you are running nodetool status, be sure to specify the keyspace also. If you don't specify the keyspace the results will be nonsense. https://issues.apache.org/jira/browse/CASSANDRA-7173 Regards, Mark On 28 October 2014 10:35, Oleg Dulin oleg.du...@gmail.com wrote: I have a setup with 6 cassandra nodes (1.2.18), using RandomPartition, not using vnodes -- this is a legacy cluster. We went from 3 nodes to 6 in the last few days to add capacity. However, there appears to be an imbalance: Datacenter: us-east == Replicas: 2 Address RackStatus State LoadOwns Token 113427455640312821154458202477256070484 x.x.x.73 1d Up Normal 154.64 GB 33.33% 85070591730234615865843651857942052863 x.x.x.2511a Up Normal 62.26 GB16.67% 28356863910078205288614550619314017621 x.x.x.238 1b Up Normal 243.7 GB50.00% 56713727820156410577229101238628035242 x.x.x.25 1a Up Normal 169.3 GB33.33% 210 x.x.x.162 1b Up Normal 118.24 GB 50.00% 141784319550391026443072753096570088105 x.x.x.208 1d Up Normal 226.85 GB 16.67% 113427455640312821154458202477256070484 What is the cause of this imbalance ? How can I rectify it ? Regards, Oleg
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Hi stephen, I have never added a node via OpsCenter, so this may be a short coming of that process. However in non OpsCenter installs you would have to create the data directories first: sudo mkdir -p /mnt/cassandra/commitlog sudo mkdir -p /mnt/cassandra/data sudo mkdir -p /mnt/cassandra/saved_caches And then give the cassandra user ownership of those directories: sudo chown -R cassandra:cassandra /mnt/cassandra Once this is done Cassandra will have the correct directories and permission to start up. Mark On 27 August 2014 09:50, Stephen Portanova sport...@gmail.com wrote: I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect. I check the system.log and output.log, and they both say: INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory /mnt/cassandra/data doesn't exist* *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory* INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down... ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.AssertionError at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263) at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) My agent.log file says: Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip. I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it! -- Stephen Portanova (480) 495-2634
Re: Installing Cassandra Multinode on CentOs coming up with exception
It is telling you that your yaml is invalid, from looking at the snippet you have provided it looks like the seed_provider.parameters is not correctly indented, it should look something like: seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 192.168.1.32 Regards, Mark On 26 August 2014 08:12, Vineet Mishra clearmido...@gmail.com wrote: Hi All, I am installing Cassandra Multinode Setup on a 4 node CentOs Cluster, my cassandra.yaml looks like so cluster_name: 'node' initial_token: 0 num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 192.168.1.32 listen_address: 192.168.1.32 rpc_address: 0.0.0.0 endpoint_snitch: RackInferringSnitch Similarly cassandra.yaml for second node cluster_name: 'node' initial_token: 2305843009213693952 num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 192.168.1.32 listen_address: 192.168.1.36 rpc_address: 0.0.0.0 endpoint_snitch: RackInferringSnitch and so on. . . While trying to start the Server running on the Seed node(192.168.1.32) it throws this nasty exception and didn't start, -bash-4.1$ sudo bin/cassandra -bash-4.1$ INFO 12:19:46,653 Logging initialized INFO 12:19:46,688 Loading settings from file:/home/cluster/cassandra/conf/cassandra.yaml ERROR 12:19:46,985 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:100) at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:135) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:111) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:156) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) Caused by: Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=seed_provider for JavaBean=org.apache.cassandra.config.Config@676c6370; java.lang.reflect.InvocationTargetException in 'reader', line 1, column 1: cluster_name: 'pcross' ^ at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:333) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127) at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481) at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:475) at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:93) ... 5 more Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=seed_provider for JavaBean=org.apache.cassandra.config.Config@676c6370; java.lang.reflect.InvocationTargetException at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:299) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:189) at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:331) ... 11 more Caused by: org.yaml.snakeyaml.error.YAMLException: java.lang.reflect.InvocationTargetException at org.yaml.snakeyaml.constructor.Constructor$ConstructSequence.construct(Constructor.java:542) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:296) ... 13 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.yaml.snakeyaml.constructor.Constructor$ConstructSequence.construct(Constructor.java:540) ... 15 more Caused by: java.lang.NullPointerException at org.apache.cassandra.config.SeedProviderDef.init(SeedProviderDef.java:33) ... 20 more Invalid yaml I am not sure exactly whats making it to throw NullPointer and halt the process. Expert Advice would be appreciated! URGENT! Thanks!
Re: Installing Cassandra Multinode on CentOs coming up with exception
You are missing commitlog_sync in your cassandra.yaml. Are you generating your own cassandra.yaml or editing the package default? If you are generating your own there are several configuration options that are required and if not present, Cassandra will fail to start. Regards, Mark On 26 August 2014 11:14, Vineet Mishra clearmido...@gmail.com wrote: Thanks Mark, That was indeed yaml formatting issue. Moreover I am getting the underlying error now, INFO 15:33:43,770 Loading settings from file:/home/cluster/cassandra/conf/cassandra.yaml INFO 15:33:44,100 Data files directories: [/var/lib/cassandra/data] INFO 15:33:44,101 Commit log directory: /var/lib/cassandra/commitlog ERROR 15:33:44,103 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Missing required directive CommitLogSync at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:147) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:111) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:156) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) Missing required directive CommitLogSync Fatal configuration error; unable to start. See log for stacktrace. Do you have any idea about this. Thanks! On Tue, Aug 26, 2014 at 3:07 PM, Mark Reddy mark.l.re...@gmail.com wrote: It is telling you that your yaml is invalid, from looking at the snippet you have provided it looks like the seed_provider.parameters is not correctly indented, it should look something like: seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 192.168.1.32 Regards, Mark On 26 August 2014 08:12, Vineet Mishra clearmido...@gmail.com wrote: Hi All, I am installing Cassandra Multinode Setup on a 4 node CentOs Cluster, my cassandra.yaml looks like so cluster_name: 'node' initial_token: 0 num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 192.168.1.32 listen_address: 192.168.1.32 rpc_address: 0.0.0.0 endpoint_snitch: RackInferringSnitch Similarly cassandra.yaml for second node cluster_name: 'node' initial_token: 2305843009213693952 num_tokens: 256 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: 192.168.1.32 listen_address: 192.168.1.36 rpc_address: 0.0.0.0 endpoint_snitch: RackInferringSnitch and so on. . . While trying to start the Server running on the Seed node(192.168.1.32) it throws this nasty exception and didn't start, -bash-4.1$ sudo bin/cassandra -bash-4.1$ INFO 12:19:46,653 Logging initialized INFO 12:19:46,688 Loading settings from file:/home/cluster/cassandra/conf/cassandra.yaml ERROR 12:19:46,985 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:100) at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:135) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:111) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:156) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) Caused by: Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=seed_provider for JavaBean=org.apache.cassandra.config.Config@676c6370; java.lang.reflect.InvocationTargetException in 'reader', line 1, column 1: cluster_name: 'pcross' ^ at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:333) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127) at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481) at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:475) at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:93) ... 5 more Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=seed_provider for JavaBean=org.apache.cassandra.config.Config@676c6370; java.lang.reflect.InvocationTargetException at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:299) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:189) at org.yaml.snakeyaml.constructor.Constructor
Re: Secondary index or dedicated CF?
Hi, As a general rule of thumb I would steer clear of secondary indexes, this is also the official stand that DataStax take (see p5 of their best practices doc: http://www.datastax.com/wp-content/uploads/2014/04/WP-DataStax-Enterprise-Best-Practices.pdf). “It is best to avoid using Cassandra's built-in secondary indexes where possible. Instead, it is recommended to denormalize data and manually maintain a dynamic table as a form of an index instead of using a secondary index. If and when secondary indexes are to be used, they should be created only on columns containing low-cardinality data (for example: fields with less than 1000 states).“ Mark On 22 Aug 2014, at 15:58, DuyHai Doan doanduy...@gmail.com wrote: Hello Eric Under the hood what is the difference of the both solutions? 1. Cassandra secondary index: distributed index, supports better high volume of data, the index itself is distributed so there is no bottleneck. The tradeoff is that depending on the cardinality of data having the same bucketname+tenantID the performance may drop sharply. Please read this: http://www.datastax.com/documentation/cql/3.1/cql/ddl/ddl_when_use_index_c.html?scroll=concept_ds_sgh_yzz_zj__when-no-index. There are several restrictions to secondary index 2. Manual index: easy to design, but potentially wide row and not well balance if data having the same bucketname+tenantID is very large. Furthermore you need to manage index consistency manually so that it is synced with source data updates. The best thing to do is to benchmark both solutions and takes the approach giving you the best results. Be careful with benchmarks, it should be representative of the data pattern you likely have in production. On Fri, Aug 22, 2014 at 7:47 AM, Leleu Eric eric.le...@worldline.com wrote: Hi, I’m new with Cassandra and I wondering what is the best design for my case. I have a set of buckets that contain one or thousands of contents. Here is my Content CF : CREATE TABLE IF NOT EXISTS contents (tenantID varchar, key varchar, type varchar, bucket varchar, owner varchar, workspace varchar, public_read boolean PRIMARY KEY ((key, tenantID), type, workspace)); To retrieve all contents that belong to a bucket, I have created an index on the bucket column. CREATE INDEX IF NOT EXISTS bucket_to_contents ON contents (bucket); The column value “bucket” is concatenated with the tenantId (bucket = bucketname+tenantID) in order to avoid filtering on the tenantID on my application. Is it the rights way to do or should I create another column family to link each content to the bucket ? CREATE TABLE IF NOT EXISTS bucket_to_contents (tenantID varchar, key varchar, type varchar, bucket varchar, owner varchar, workspace varchar, public_read boolean PRIMARY KEY ((bucket, tenantID), key)); Under the hood what is the difference of the both solutions? According to my understanding, the result will be the same. Both will have the rowkey equals to the “bucketname” and the “tenantID”. Excepted that the secondary index can have a replication delay… Can you help me on this point? Regards, Eric Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Re: 答复: can not query data from cassandra
Hi, It looks like you are running into a known issue: https://issues.apache.org/jira/browse/CASSANDRA-7780 and its being worked on here: https://issues.apache.org/jira/browse/CASSANDRA-7145 Mark On 20 August 2014 09:06, 鄢来琼 laiqiong@gtafe.com wrote: Is it related to MAX_HEAP_SIZE and HEAP_NEWSIZE? My system memory size is 4G,but the MAX_HEAP_SIZE and HEAP_NEWSIZE are set to 2G and 800M, respectively. -- Thanks and regards, GTA Data *Peter.YAN* E-Mail:laiqiong@gtafe.com http://www.gtafe.com/ [image: 说明: cid:image001.png@01CF5897.E1268DE0] *发件人:* 鄢来琼 [mailto:laiqiong@gtafe.com] *发送时间:* 2014年8月20日 14:13 *收件人:* user@cassandra.apache.org *主题:* can not query data from cassandra HI ALL, I setup Cassandra on a linux host. I have insert some data into “mykeyspace.cffex_l23” table. The following error are raised during query data from “mykeyspace.cffex_l23”. Could you give me any suggestion to fix it? According to “top” cmd, I found that most of the memory are used by Cassandra. *Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st* *Mem: 4043660k total, 3281336k used, 762324k free,42704k buffers* *Swap:0k total,0k used,0k free, 741456k cached* * PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND* *7583 root 20 0 4437m 2.3g 65m S 0.0 58.7 531:32.23 java* *10907 root 15 0 575m 27m 2024 S 0.0 0.7 0:03.07 mongod* The system.log is attached. INFO [MemoryMeter:1] 2014-08-20 14:00:24,612 Memtable.java (line 481) CFS(Keyspace='mykeyspace', ColumnFamily='cffex_l23') liveRatio is 6.294228546622461 (just-counted was 6.294228546622461). calculation took 2907ms for 279972 cells ERROR [ReadStage:871] 2014-08-20 14:01:51,197 CassandraDaemon.java (line 199) Exception in thread Thread[ReadStage:871,5,main] java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /opt/apache-cassandra-2.0.9/cassandraData/data/mykeyspace/cffex_l23/mykeyspace-cffex_l23-jb-799-Data.db (No such file or directory) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1931) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /opt/apache-cassandra-2.0.9/cassandraData/data/mykeyspace/cffex_l23/mykeyspace-cffex_l23-jb-799-Data.db (No such file or directory) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48) at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39) at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:396) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:366) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144) at org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:87) at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) at
Re: Cassandra Consistency Level
Hi, As you are writing as CL.ONE and cqlsh by default reads at CL.ONE, there is a probability that you are reading stale data i.e. the node you have contacted for the read may not have the most recent data. If you have a higher consistency requirement, you should look at increasing your consistency level, for a more detailed look at this see: http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html If you want to continue using CL.ONE, you could look at increasing the read_repair_chance for better consistency. http://www.datastax.com/documentation/cassandra/2.0/cassandra/reference/referenceTableAttributes.html Just to verify that this is in fact a consistency issue could you run a nodetool repair on that table and run the same queries again? Mark Regards, Mark On 20 August 2014 00:14, Check Peck comptechge...@gmail.com wrote: We have cassandra cluster in three different datacenters (DC1, DC2 and DC3) and we have 10 machines in each datacenter. We have few tables in cassandra in which we have less than 100 records. What we are seeing - some tables are out of sync between machines in DC3 as compared to DC1 or DC2 when we do select count(*) on it. As an example we did select count(*) while connecting to one cassandra machine in dc3 datacenter as compared to one cassandra machine in dc1 datacenter and the results were different. root@machineA:/home/david/apache-cassandra/bin# python cqlsh dc3114.dc3.host.com Connected to TestCluster at dc3114.dc3.host.com:9160. [cqlsh 2.3.0 | Cassandra 1.2.9 | CQL spec 3.0.0 | Thrift protocol 19.36.0] Use HELP for help. cqlsh use testingkeyspace ; cqlsh:testingkeyspace select count(*) from test_metadata ; count --- 12 cqlsh:testingkeyspace exit root@machineA:/home/david/apache-cassandra/bin# python cqlsh dc18b0c.dc1.host.com Connected to TestCluster at dc18b0c.dc1.host.com:9160. [cqlsh 2.3.0 | Cassandra 1.2.9 | CQL spec 3.0.0 | Thrift protocol 19.36.0] Use HELP for help. cqlsh use testingkeyspace ; cqlsh:testingkeyspace select count(*) from test_metadata ; count --- 16 What could be the reason for this sync issue? Can anyone shed some light on this? Since our java driver code and datastax c++ driver code are using these tables with CONSISTENCY LEVEL ONE.
Re: question about OpsCenter agent
Hi Clint, You need to configure the datastax-agents so they know what machine OpsCenter is running on. To do this you will need to edit the address.yaml of the datastax-agent, located in /var/lib/datastax-agent/conf/. In this file you need to add the following line: stomp_interface: opscenter-ip-address This will allow your agents to connect to the OpsCenter server (opscenterd) over the stomp protocol (stomp.github.io/). For a more detailed look at the configuration options available to the datastax-agent see this page: datastax.com/documentation/opscenter/5.0/opsc/configure/agentAddressConfiguration.html Mark On Fri, Aug 15, 2014 at 3:32 AM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I just installed DataStax Enterprise 4.5. I installed OpsCenter Server on one of my four machines. The port that OpsCenter usually uses () was used by something else, so I modified /usr/share/opscenter/conf/opscenterd.conf to set the port to 8889. When I log into OpsCenter, it says 0 of 4 agents connected. I have started datastax-agent on all four of the nodes in my Cassandra cluster. Is there another OpsCenter agent that I have to start? I'm not sure how to fix this. I clicked on the Fix link and it prompted me You have nodes that need the OpsCenter agent installed. How do I install the OpsCenter agent without using this GUI? I don't recall getting prompted for that when I installed DSE. Best regards, Clint
Re: Bootstrap failures: unable to find sufficient sources for streaming range
Hi Peter, At the time of the IllegalStateException, do you see the node that it should be streaming from marked as down by the failure detector? Mark On Fri, Aug 15, 2014 at 5:45 AM, Peter Haggerty peter.hagge...@librato.com wrote: When adding nodes via bootstrap to a 27 node 2.0.9 cluster with a cluster-wide phi_convict_threshold of 12 the nodes fail to bootstrap. This worked a half dozen times in the past few weeks as we've scaled this cluster from 21 to 24 and then to 27 nodes. There have been no configuration or Cassandra version changes since the cluster was booted several weeks ago. This is adding a single node at a time just as we've done before. When using a phi of 16 on the node that is bootstrapping we've had one failure and one success. Adjusting RING_DELAY from 30 seconds to 45 doesn't appear to help. We have DEBUG level logs for a successful attempt and a failed attempt but they seem quite similar in behavior, even bad behavior like ERRORS reported by MigrationTask.java and FailureDetector.java. In the failed attempts the bootstrap immediately falls over with unable to find ...: java.lang.IllegalStateException: unable to find sufficient sources for streaming range (1445230693272554509,1449885986247309687] The logs clearly show that it knows where those ranges can be found: DEBUG [main] 2014-08-14 18:56:45,019 RangeStreamer.java (line 122) Bootstrap: range (1445230693272554509,1449885986247309687] exists on /10.11.12.13 DEBUG [main] 2014-08-14 18:56:45,019 RangeStreamer.java (line 122) Bootstrap: range (1445230693272554509,1449885986247309687] exists on /10.11.12.14 If you look for the various non-debug log lines for these addresses you'll see some odd behavior, but you see similar odd behavior in the logs of the node that is able to successfully bootstrap. $ grep '10.11.12.13' system.log | grep -v ^DEBUG INFO [GossipStage:1] 2014-08-14 18:56:09,870 Gossiper.java (line 910) Node /10.11.12.13 is now part of the cluster INFO [GossipStage:1] 2014-08-14 18:56:15,267 Gossiper.java (line 910) Node /10.11.12.13 is now part of the cluster INFO [HANDSHAKE-/10.11.12.13] 2014-08-14 18:56:45,571 OutboundTcpConnection.java (line 386) Handshaking version with /10.11.12.13 INFO [RequestResponseStage:5] 2014-08-14 18:56:45,579 Gossiper.java (line 876) InetAddress /10.11.12.13 is now UP INFO [RequestResponseStage:2] 2014-08-14 18:56:45,579 Gossiper.java (line 876) InetAddress /10.11.12.13 is now UP INFO [RequestResponseStage:7] 2014-08-14 18:56:45,580 Gossiper.java (line 876) InetAddress /10.11.12.13 is now UP INFO [RequestResponseStage:1] 2014-08-14 18:56:45,579 Gossiper.java (line 876) InetAddress /10.11.12.13 is now UP INFO [RequestResponseStage:8] 2014-08-14 18:56:45,579 Gossiper.java (line 876) InetAddress /10.11.12.13 is now UP $ grep '10.11.12.14' system.log | grep -v ^DEBUG INFO [GossipStage:1] 2014-08-14 18:56:09,865 Gossiper.java (line 910) Node /10.11.12.14 is now part of the cluster INFO [GossipStage:1] 2014-08-14 18:56:15,106 Gossiper.java (line 910) Node /10.11.12.14 is now part of the cluster The general order of events, as told by JOINING lines: $ grep 'JOINING' system.log INFO [main] 2014-08-14 18:56:10,638 StorageService.java (line 1009) JOINING: waiting for ring information INFO [main] 2014-08-14 18:56:14,640 StorageService.java (line 1009) JOINING: schema complete, ready to bootstrap INFO [main] 2014-08-14 18:56:14,640 StorageService.java (line 1009) JOINING: waiting for pending range calculation INFO [main] 2014-08-14 18:56:14,641 StorageService.java (line 1009) JOINING: calculation complete, ready to bootstrap INFO [main] 2014-08-14 18:56:14,641 StorageService.java (line 1009) JOINING: getting bootstrap token INFO [main] 2014-08-14 18:56:14,712 StorageService.java (line 1009) JOINING: sleeping 3 ms for pending range setup INFO [main] 2014-08-14 18:56:44,713 StorageService.java (line 1009) JOINING: Starting to bootstrap... Thanks in advance for any suggestions. Peter
Re: How to prevent the removed DC comes back automactically?
How can we prevent a disconnected DC from coming back automatically? You could use firewall rules to prevent the disconnected DC from contacting your live DCs when it becomes live again Mark On Thu, Aug 14, 2014 at 6:48 AM, Lu, Boying boying...@emc.com wrote: Hi, All, We are using Cassandra 2.0.7 in a multi DCs environments. If a connected DC is powered off, we use the ‘nodetool removenode’ command to remove it from the connected DCs. But we found that once the disconnected DC is powered on, it will connect to other DCs automatically. How can we prevent a disconnected DC from coming back automatically? Thanks a lot Boying
Re: A question to nodetool removenode command
Hi, Gossip will propagate to all nodes in a cluster. So if you have a cluster spanning DC1, DC2 and DC3 and you then remove all nodes in DC3 via nodetool removenode from a node in DC1, all nodes in both DC1 and DC2 will be informed of the nodes removal so no need to run it from a node in DC2. Mark On Thu, Aug 14, 2014 at 10:20 AM, Lu, Boying boying...@emc.com wrote: Hi, All, We have a Cassandra 2.0.7 running in three connected DCs, say DC1, DC2 and DC3. DC3 is powered off, so we run ‘nodetool removenode’ command on DC1 to remove all nodes of DC3. Do we need to run the same command on DC2? Thanks Boying
Re: Cassandra corrupt column family
Hi, Without more information (Cassandra version, setup, topology, schema, queries performed) this list won't be able to assist you. If you can provide a more detailed explanation of the steps you took to reach your current state that would be great. Mark On Tue, Aug 12, 2014 at 12:21 PM, Batranut Bogdan batra...@yahoo.com wrote: Hello all, I have altered a table in cassandra and on one node it somehow got corrupted. I the changes did not propagate ok. Ran repair keyspace columnfamily... noting changed... Is there a way to repair this?
Re: Nodetool Repair questions
Hi Vish, 1. This tool repairs inconsistencies across replicas of the row. Since latest update always wins, I dont see inconsistencies other than ones resulting from the combination of deletes, tombstones, and crashed nodes. Technically, if data is never deleted from cassandra, then nodetool repair does not need to be run at all. Is this understanding correct? If wrong, can anyone provide other ways inconsistencies could occur? Even if you never delete data you should run repairs occasionally to ensure overall consistency. While hinted handoffs and read repairs do lead to better consistency, they are only helpers/optimization and are not regarded as operations that ensure consistency. 2. Want to understand the performance of 'nodetool repair' in a Cassandra multi data center setup. As we add nodes to the cluster in various data centers, does the performance of nodetool repair on each node increase linearly, or is it quadratic ? Its difficult to calculate the performance of a repair, I've seen the time to completion fluctuate between 4hrs to 10hrs+ on the same node. However in theory adding more nodes would spread the data and free up machine resources, thus resulting in more performant repairs. The essence of this question is: If I have a keyspace with x number of replicas in each data center, do I have to deal with an upper limit on the number of data centers/nodes? Could you expand on why you believe there would be an upper limit of dc/nodes due to running repairs? Mark On Tue, Aug 12, 2014 at 10:06 PM, Viswanathan Ramachandran vish.ramachand...@gmail.com wrote: Some questions on nodetool repair. 1. This tool repairs inconsistencies across replicas of the row. Since latest update always wins, I dont see inconsistencies other than ones resulting from the combination of deletes, tombstones, and crashed nodes. Technically, if data is never deleted from cassandra, then nodetool repair does not need to be run at all. Is this understanding correct? If wrong, can anyone provide other ways inconsistencies could occur? 2. Want to understand the performance of 'nodetool repair' in a Cassandra multi data center setup. As we add nodes to the cluster in various data centers, does the performance of nodetool repair on each node increase linearly, or is it quadratic ? The essence of this question is: If I have a keyspace with x number of replicas in each data center, do I have to deal with an upper limit on the number of data centers/nodes? Thanks Vish
Re: Cassandra bootstrap error: java.lang.IllegalStateException: No sources found
Hi, When you configured the new node, did you correctly configure your seeds and listen address? Mark On Mon, Aug 11, 2014 at 6:12 PM, Rahul Gupta rgu...@dekaresearch.com wrote: I have a 3 node Cassandra cluster and I am adding a new node when I get below error during bootstrap. Running DataStax ver 4.5 (Cassandra 2.0.8.39) on all the nodes. Nodes are VMs running on VMWare . Any ideas what might be wrong here? INFO [main] 2014-08-11 13:01:09,427 StorageService.java (line 1007) JOINING: Starting to bootstrap... ERROR [main] 2014-08-11 13:01:09,443 CassandraDaemon.java (line 513) Exception encountered during startup java.lang.IllegalStateException: No sources found for (85070591730234615865843651857942052864,134599085099335834412322738856718602703] at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:159) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:117) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:72) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1035) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:797) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:614) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:504) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:394) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:574) INFO [Thread-1] 2014-08-11 13:01:09,451 DseDaemon.java (line 477) DSE shutting down... INFO [StorageServiceShutdownHook] 2014-08-11 13:01:09,476 Gossiper.java (line 1279) Announcing shutdown INFO [Thread-1] 2014-08-11 13:01:09,521 PluginManager.java (line 317) All plugins are stopped. INFO [Thread-1] 2014-08-11 13:01:09,522 CassandraDaemon.java (line 463) Cassandra shutting down... ERROR [Thread-1] 2014-08-11 13:01:09,522 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.NullPointerException at org.apache.cassandra.service.CassandraDaemon.stop(CassandraDaemon.java:464) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:480) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) INFO [StorageServiceShutdownHook] 2014-08-11 13:01:11,477 MessagingService.java (line 683) Waiting for messaging service to quiesce INFO [ACCEPT-/172.17.0.122] 2014-08-11 13:01:11,478 MessagingService.java (line 923) MessagingService has terminated the accept() thread *Rahul Gupta* *DEKA* *Research Development* http://www.dekaresearch.com/ 340 Commercial St Manchester, NH 03101 P: 603.666.3908 extn. 6504 | C: 603.718.9676 This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. -- This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. Thank you. Please consider the environment before printing this email.
Re: Unable to connect to cassandra server on EC2?
Hi, While I have no direct experience with the Python driver itself I took a quick and it uses Cassandra's native transport protocol, so setting the port to 9160 (Thrift protocol) won't work. You will need to set it to the native transport port, which is 9042. Also make sure that you have the rpc_address (and if needed broadcast_rpc_address) set correctly. On a side note, if you are trying out 2.1.0, 2.1.0-rc5 is out and with many fixes and improvements. Mark On Sun, Aug 10, 2014 at 1:04 PM, Subodh Nijsure subodh.nijs...@gmail.com wrote: Hello, I have setup a new EC2 instance to run cassandra on EC2, have gone through bunch of questions that don't seem to help. I am running apache-cassandra-2.1.0-rc3 I have opened port 9160, 9042 on my EC2 instance say its IP address is 1.2.3.4 Since this is single node system I haven't opened other ports as pointed in http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMISecurityGroup.html When I try to connect to this host using the python driver with cluster config like this: cluster = Cluster(contact_points=['1.2.3.4',], port=9160) I am getting error: File cluster.py, line 1649, in _reconnect_internal raise NoHostAvailable(Unable to connect to any servers, errors) cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'1.2.3.4': error(111, 'Connection refused')}) Any ideas on what incantation I need to have in cassandra.yml file to allow external clients to connect cassandra instance running on EC2 ? -Subodh
Re: clarification on 100k tombstone limit in indexes
Hi Ian, The issues here, which relates to normal and index column families, is scanning over a large number of tombstones can cause Cassandra to fall over due to increased GC pressure. This pressure is caused because tombstones will create DeletedColumn objects which consume heap. Also these DeletedColumn objects will have to be serialized and sent back to the coordinator, thus increasing your response times. Take for example a row that does deletes and you query it with a limit of 100. In a worst case scenario you could end up reading say 50k tombstones to reach the 100 'live' column limit, all of which has to be put on heap and then sent over the wire to the coordinator. This would be considered a Cassandra anti-pattern.[1] With that in mind there was a debug warning added to 1.2 to inform the user when they were querying a row with 1000 tombstones [2]. Then in 2.0 the action was taken to drop requests reaching 100k tombstones[3] rather than just printing out a warning. This is a safety measure, as it is not advised to perform such a query and is a result of most people 'doing it wrong'. For those people who understand the risk of scanning over large numbers of tombstones there is a configuration option in the cassandra.yaml to increase this threshold, tombstone_failure_threshold.[4] Mark [1] http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets [2] https://issues.apache.org/jira/browse/CASSANDRA-6042 [3] https://issues.apache.org/jira/browse/CASSANDRA-6117 [4] https://github.com/jbellis/cassandra/blob/4ac18ae805d28d8f4cb44b42e2244bfa6d2875e1/conf/cassandra.yaml#L407-L417 On Sun, Aug 10, 2014 at 7:19 PM, Ian Rose ianr...@fullstory.com wrote: Hi - On this page ( http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html), the docs state: Do not use an index [...] On a frequently updated or deleted column and *Problems using an index on a frequently updated or deleted column*¶ http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html?scroll=concept_ds_sgh_yzz_zj__upDatIndx Cassandra stores tombstones in the index until the tombstone limit reaches 100K cells. After exceeding the tombstone limit, the query that uses the indexed value will fail. I'm afraid I don't really understand this limit from its (brief) description. I also saw this recent thread http://mail-archives.apache.org/mod_mbox/cassandra-user/201403.mbox/%3CCABNXB2Bf4aeoDVpMNOxJ_e7aDez2EuZswMJx=jWfb8=oyo4...@mail.gmail.com%3E but I'm afraid it didn't help me much... *SHORT VERSION* If I have tens or hundreds of thousands of rows in a keyspace, where every row has an indexed column that is updated O(10) times during the lifetime of each row, is that going to cause problems for me? If that 100k limit is *per row* then I should be fine but if that 100k limit is *per keyspace* then I'd definitely exceed it quickly. *FULL EXPLANATION* In our system, items are created at a rate of ~10/sec. Each item is updated ~10 times over the next few minutes (although in rare cases the number of updates, and the duration, might be several times as long). Once the last update is received for an item, we select it from Cassandra, process the data, then delete the entire row. The tricky bit is that sometimes (maybe 30-40% of the time) we don't actually know when the last update has been received so we use a timeout: if an item hasn't been updated for 30 minutes, then we assume it is done and should process it as before (select, then delete). So I am trying to design a schema that will allow for efficient queries of the form find me all items that have not been updated in the past 30 minutes. We plan to call this query once a minute. Here is my tentative schema: CREATE TABLE items ( item_id ascii, last_updated timestamp, item_data listblob, PRIMARY KEY (item_id) ) plus an index on last_updated. So updates to an existing item would just be lookup by item_id, append new data to item_data, and set last_updated to now. And queries to find items that have timed out would use the index on last_updated: find all items where last_updated [now - 30 minutes]. Assuming, that is, that the aforementioned 100k tombstone limit won't bring this index crashing to a halt... Any clarification on this limit and/or suggestions on a better way to model/implement this system would be greatly appreciated! Cheers, Ian
Re: clarification on 100k tombstone limit in indexes
Hi Ian Are these tombstones ever GCed out of the index? How frequently? Yes, tombstones are removed after the time specified by gc_grace_seconds has elapsed, which by default is 10 days and is configurable. Knowing and understanding how Cassandra handles distributed deletes is key to designing an efficient schema if you plan to delete often. There are lots of resources on how deletes are handled in Cassandra, take a look at these links for example: http://wiki.apache.org/cassandra/DistributedDeletes http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_about_deletes_c.html Mark On Sun, Aug 10, 2014 at 9:02 PM, Ian Rose ianr...@fullstory.com wrote: Hi Mark - Thanks for the clarification but as I'm not too familiar with the nuts bolts of Cassandra I'm not sure how to apply that info to my current situation. It sounds like this 100k limit is, indeed, a global limit as opposed to a per-row limit. Are these tombstones ever GCed out of the index? How frequently? If not, then it seems like *any* index is at risk of reaching this tipping point; it's just that indexes on frequently updated columns will reach this pointer faster the indexes on rarely updated columns. Basically I'm trying to get some kind of sense for what frequently updated http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html means quantitatively. As written, the docs make it sound dangerous to create an index on a column that is *ever* deleted or updated since there is no sense of how frequent is too frequent. Cheers, Ian On Sun, Aug 10, 2014 at 3:02 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Ian, The issues here, which relates to normal and index column families, is scanning over a large number of tombstones can cause Cassandra to fall over due to increased GC pressure. This pressure is caused because tombstones will create DeletedColumn objects which consume heap. Also these DeletedColumn objects will have to be serialized and sent back to the coordinator, thus increasing your response times. Take for example a row that does deletes and you query it with a limit of 100. In a worst case scenario you could end up reading say 50k tombstones to reach the 100 'live' column limit, all of which has to be put on heap and then sent over the wire to the coordinator. This would be considered a Cassandra anti-pattern.[1] With that in mind there was a debug warning added to 1.2 to inform the user when they were querying a row with 1000 tombstones [2]. Then in 2.0 the action was taken to drop requests reaching 100k tombstones[3] rather than just printing out a warning. This is a safety measure, as it is not advised to perform such a query and is a result of most people 'doing it wrong'. For those people who understand the risk of scanning over large numbers of tombstones there is a configuration option in the cassandra.yaml to increase this threshold, tombstone_failure_threshold.[4] Mark [1] http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets [2] https://issues.apache.org/jira/browse/CASSANDRA-6042 [3] https://issues.apache.org/jira/browse/CASSANDRA-6117 [4] https://github.com/jbellis/cassandra/blob/4ac18ae805d28d8f4cb44b42e2244bfa6d2875e1/conf/cassandra.yaml#L407-L417 On Sun, Aug 10, 2014 at 7:19 PM, Ian Rose ianr...@fullstory.com wrote: Hi - On this page ( http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html), the docs state: Do not use an index [...] On a frequently updated or deleted column and *Problems using an index on a frequently updated or deleted column*¶ http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html?scroll=concept_ds_sgh_yzz_zj__upDatIndx Cassandra stores tombstones in the index until the tombstone limit reaches 100K cells. After exceeding the tombstone limit, the query that uses the indexed value will fail. I'm afraid I don't really understand this limit from its (brief) description. I also saw this recent thread http://mail-archives.apache.org/mod_mbox/cassandra-user/201403.mbox/%3CCABNXB2Bf4aeoDVpMNOxJ_e7aDez2EuZswMJx=jWfb8=oyo4...@mail.gmail.com%3E but I'm afraid it didn't help me much... *SHORT VERSION* If I have tens or hundreds of thousands of rows in a keyspace, where every row has an indexed column that is updated O(10) times during the lifetime of each row, is that going to cause problems for me? If that 100k limit is *per row* then I should be fine but if that 100k limit is *per keyspace* then I'd definitely exceed it quickly. *FULL EXPLANATION* In our system, items are created at a rate of ~10/sec. Each item is updated ~10 times over the next few minutes (although in rare cases the number of updates, and the duration, might be several times as long). Once the last update is received for an item, we select it from Cassandra, process the data, then delete the entire
Re: Reasonable range for the max number of tables?
Hi Phil, In theory, the max number of column families would be in the low number of hundreds. In practice the limit is related the amount of heap you have, as each column family will consume 1 MB of heap due to arena allocation. To segregate customer data, you could: - Use customer specific column families under a single keyspace - Use a keyspace per customer - Use the same column families and have a column that identifies the customer. On the application layer ensure that there are sufficient checks so one customer can't read another customers data Mark On Tue, Aug 5, 2014 at 9:09 AM, Phil Luckhurst phil.luckhu...@powerassure.com wrote: Is there any mention of this limitation anywhere in the Cassandra documentation? I don't see it mentioned in the 'Anti-patterns in Cassandra' section of the DataStax 2.0 documentation or anywhere else. When starting out with Cassandra as a store for a multi-tenant application it seems very attractive to segregate data for each tenant using a tenant specific keyspace each with their own set of tables. It's not until you start browsing through forums such as this that you find out that it isn't going to scale above a few tenants. If you want to be able to segregate customer data in Cassandra is it the accepted practice to have multiple Cassandra installations? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Reasonable-range-for-the-max-number-of-tables-tp7596094p7596106.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Make an existing cluster multi data-center compatible.
Yes, you must run a full repair for the reasons stated in the yaml file. Mark On Tue, Aug 5, 2014 at 11:52 AM, Rene Kochen rene.koc...@schange.com wrote: Hi all, I want to add a data-center to an existing single data-center cluster. First I have to make the existing cluster multi data-center compatible. The existing cluster is a 12 node cluster with: - Replication factor = 3 - Placement strategy = SimpleStrategy - Endpoint snitch = SimpleSnitch If I change the following: - Placement strategy = NetworkTopologyStrategy - Endpoint snitch = PropertyFileSnitch - all 12 nodes in this file belong to the same data-center and rack. Do I have to run full repairs after this change? Because the yaml file states: IF YOU CHANGE THE SNITCH AFTER DATA IS INSERTED INTO THE CLUSTER, YOU MUST RUN A FULL REPAIR, SINCE THE SNITCH AFFECTS WHERE REPLICAS ARE PLACED. Thanks! Rene
Re: Node bootstrap
Yes num_tokens is set to 256. initial_token is blank on all nodes including the new one. Ok so you have num_tokens set to 256 for all nodes with initial_token commented out, this means you are using vnodes and the new node will automatically grab a list of tokens to take over responsibility for. Pool NameActive Pending Completed Blocked All time blocked FlushWriter 0 0 1136 0 512 Looks like about 50% of flushes are blocked. This is a problem as it indicates that the IO system cannot keep up. Just ran this on the new node: nodetool netstats | grep Streaming from | wc -l 10 This is normal as the new node will most likely take tokens from all nodes in the cluster. Sorry for the multiple updates, but another thing I found was all the other existing nodes have themselves in the seeds list, but the new node does not have itself in the seeds list. Can that cause this issue? Seeds are only used when a new node is bootstrapping into the cluster and needs a set of ips to contact and discover the cluster, so this would have no impact on data sizes or streaming. In general it would be considered best practice to have a set of 2-3 seeds from each data center, with all nodes having the same seed list. What is the current output of 'nodetool compactionstats'? Could you also paste the output of nodetool status keyspace? Mark On Tue, Aug 5, 2014 at 3:59 PM, Ruchir Jha ruchir@gmail.com wrote: Sorry for the multiple updates, but another thing I found was all the other existing nodes have themselves in the seeds list, but the new node does not have itself in the seeds list. Can that cause this issue? On Tue, Aug 5, 2014 at 10:30 AM, Ruchir Jha ruchir@gmail.com wrote: Just ran this on the new node: nodetool netstats | grep Streaming from | wc -l 10 Seems like the new node is receiving data from 10 other nodes. Is that expected in a vnodes enabled environment? Ruchir. On Tue, Aug 5, 2014 at 10:21 AM, Ruchir Jha ruchir@gmail.com wrote: Also not sure if this is relevant but just noticed the nodetool tpstats output: Pool NameActive Pending Completed Blocked All time blocked FlushWriter 0 0 1136 0 512 Looks like about 50% of flushes are blocked. On Tue, Aug 5, 2014 at 10:14 AM, Ruchir Jha ruchir@gmail.com wrote: Yes num_tokens is set to 256. initial_token is blank on all nodes including the new one. On Tue, Aug 5, 2014 at 10:03 AM, Mark Reddy mark.re...@boxever.com wrote: My understanding was that if initial_token is left empty on the new node, it just contacts the heaviest node and bisects its token range. If you are using vnodes and you have num_tokens set to 256 the new node will take token ranges dynamically. What is the configuration of your other nodes, are you setting num_tokens or initial_token on those? Mark On Tue, Aug 5, 2014 at 2:57 PM, Ruchir Jha ruchir@gmail.com wrote: Thanks Patricia for your response! On the new node, I just see a lot of the following: INFO [FlushWriter:75] 2014-08-05 09:53:04,394 Memtable.java (line 400) Writing Memtable INFO [CompactionExecutor:3] 2014-08-05 09:53:11,132 CompactionTask.java (line 262) Compacted 12 sstables to so basically it is just busy flushing, and compacting. Would you have any ideas on why the 2x disk space blow up. My understanding was that if initial_token is left empty on the new node, it just contacts the heaviest node and bisects its token range. And the heaviest node is around 2.1 TB, and the new node is already at 4 TB. Could this be because compaction is falling behind? Ruchir On Mon, Aug 4, 2014 at 7:23 PM, Patricia Gorla patri...@thelastpickle.com wrote: Ruchir, What exactly are you seeing in the logs? Are you running major compactions on the new bootstrapping node? With respect to the seed list, it is generally advisable to use 3 seed nodes per AZ / DC. Cheers, On Mon, Aug 4, 2014 at 11:41 AM, Ruchir Jha ruchir@gmail.com wrote: I am trying to bootstrap the thirteenth node in a 12 node cluster where the average data size per node is about 2.1 TB. The bootstrap streaming has been going on for 2 days now, and the disk size on the new node is already above 4 TB and still going. Is this because the new node is running major compactions while the streaming is going on? One thing that I noticed that seemed off was the seeds property in the yaml of the 13th node comprises of 1..12. Where as the seeds property on the existing 12 nodes consists of all the other nodes except the thirteenth node. Is this an issue? Any other insight is appreciated? Ruchir. -- Patricia Gorla @patriciagorla Consultant Apache Cassandra Consulting http://www.thelastpickle.com http://thelastpickle.com
Re: A question about using 'update keyspace with strategyoptions' command
So the ‘strategy’ change may not be seen by all nodes when the ‘upgrade keyspace …’ command returns and I can use ’describe cluster’ to check if the change has taken effect on all nodes right? Correct, the change may take time to propagate to all nodes. As Rahul said you can check describe cluster in cli to be sure. Mark On Tue, Aug 5, 2014 at 3:06 PM, Lu, Boying boying...@emc.com wrote: Thanks a lot. So the ‘strategy’ change may not be seen by all nodes when the ‘upgrade keyspace …’ command returns and I can use ’describe cluster’ to check if the change has taken effect on all nodes right? *From:* Rahul Neelakantan [mailto:ra...@rahul.be] *Sent:* 2014年8月5日 18:46 *To:* user@cassandra.apache.org *Subject:* Re: A question about using 'update keyspace with strategyoptions' command Try running describe cluster from Cassandra-CLI to see if all nodes have the same schema version. Rahul Neelakantan On Aug 5, 2014, at 6:13 AM, Sylvain Lebresne sylv...@datastax.com wrote: On Tue, Aug 5, 2014 at 11:40 AM, Lu, Boying boying...@emc.com wrote: What I want to know is “are the *strategy* changed ?’ after the ‘udpate keyspace with strategy_options…’ command returns successfully Like all schema changes, not necessarily on all nodes. You will have to check for schema agreement between nodes. Not the *data* change. e.g. say I run the command ‘update keyspace with strategy_opitons [dc1: 3, dc2:3]’ , when this command returns, are the *strategy* options already changed? Or I need to wait some time for the strategy to be changed? *From:* Sylvain Lebresne [mailto:sylv...@datastax.com] *Sent:* 2014年8月5日 16:59 *To:* user@cassandra.apache.org *Subject:* Re: A question about using 'update keyspace with strategyoptions' command Changing the strategy options, and in particular the replication factor, does not perform any data replication by itself. You need to run a repair to ensure data is replicated following the new replication. On Tue, Aug 5, 2014 at 10:52 AM, Lu, Boying boying...@emc.com wrote: Thanks. yes. I can use the ‘show keyspace’ command to check and see the strategy does changed. But what I want to know is if the ‘update keyspace with strategy_options …’ command is a ‘sync’ operation or a ‘async’ operation. *From:* Rahul Menon [mailto:ra...@apigee.com] *Sent:* 2014年8月5日 16:38 *To:* user *Subject:* Re: A question about using 'update keyspace with strategyoptions' command Try the show keyspaces command and look for Options under each keyspace. Thanks Rahul On Tue, Aug 5, 2014 at 2:01 PM, Lu, Boying boying...@emc.com wrote: Hi, All, I want to run ‘update keyspace with strategy_options={dc1:3, dc2:3}’ from cassandra-cli to update the strategy options of some keyspace in a multi-DC environment. When the command returns successfully, does it mean that the strategy options have been updated successfully or I need to wait some time for the change to be propagated to all DCs? Thanks Boying
Re: Node bootstrap
Hi Ruchir, With the large number of blocked flushes and the number of pending compactions would still indicate IO contention. Can you post the output of 'iostat -x 5 5' If you do in fact have spare IO, there are several configuration options you can tune such as increasing the number of flush writers and compaction_throughput_mb_per_sec Mark On Tue, Aug 5, 2014 at 5:22 PM, Ruchir Jha ruchir@gmail.com wrote: Also Mark to your comment on my tpstats output, below is my iostat output, and the iowait is at 4.59%, which means no IO pressure, but we are still seeing the bad flush performance. Should we try increasing the flush writers? Linux 2.6.32-358.el6.x86_64 (ny4lpcas13.fusionts.corp) 08/05/2014 _x86_64_(24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.80 10.250.654.590.00 78.72 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 103.83 9630.62 11982.60 3231174328 4020290310 dm-0 13.57 160.1781.12 53739546 27217432 dm-1 7.5916.9443.775682200 14686784 dm-2 5792.76 32242.66 45427.12 10817753530 15241278360 sdb 206.09 22789.19 33569.27 7646015080 11262843224 On Tue, Aug 5, 2014 at 12:13 PM, Ruchir Jha ruchir@gmail.com wrote: nodetool status: Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.10.20.27 1.89 TB256 25.4% 76023cdd-c42d-4068-8b53-ae94584b8b04 rack1 UN 10.10.20.62 1.83 TB256 25.5% 84b47313-da75-4519-94f3-3951d554a3e5 rack1 UN 10.10.20.47 1.87 TB256 24.7% bcd51a92-3150-41ae-9c51-104ea154f6fa rack1 UN 10.10.20.45 1.7 TB 256 22.6% 8d6bce33-8179-4660-8443-2cf822074ca4 rack1 UN 10.10.20.15 1.86 TB256 24.5% 01a01f07-4df2-4c87-98e9-8dd38b3e4aee rack1 UN 10.10.20.31 1.87 TB256 24.9% 1435acf9-c64d-4bcd-b6a4-abcec209815e rack1 UN 10.10.20.35 1.86 TB256 25.8% 17cb8772-2444-46ff-8525-33746514727d rack1 UN 10.10.20.51 1.89 TB256 25.0% 0343cd58-3686-465f-8280-56fb72d161e2 rack1 UN 10.10.20.19 1.91 TB256 25.5% 30ddf003-4d59-4a3e-85fa-e94e4adba1cb rack1 UN 10.10.20.39 1.93 TB256 26.0% b7d44c26-4d75-4d36-a779-b7e7bdaecbc9 rack1 UN 10.10.20.52 1.81 TB256 25.4% 6b5aca07-1b14-4bc2-a7ba-96f026fa0e4e rack1 UN 10.10.20.22 1.89 TB256 24.8% 46af9664-8975-4c91-847f-3f7b8f8d5ce2 rack1 Note: The new node is not part of the above list. nodetool compactionstats: pending tasks: 1649 compaction typekeyspace column family completed total unit progress Compaction iprod customerorder 1682804084 17956558077 bytes 9.37% Compactionprodgatecustomerorder 1664239271 1693502275 bytes98.27% Compaction qa_config_bkupfixsessionconfig_hist 2443 27253 bytes 8.96% Compactionprodgatecustomerorder_hist 1770577280 5026699390 bytes35.22% Compaction iprodgatecustomerorder_hist 2959560205312350192622 bytes 0.95% On Tue, Aug 5, 2014 at 11:37 AM, Mark Reddy mark.re...@boxever.com wrote: Yes num_tokens is set to 256. initial_token is blank on all nodes including the new one. Ok so you have num_tokens set to 256 for all nodes with initial_token commented out, this means you are using vnodes and the new node will automatically grab a list of tokens to take over responsibility for. Pool NameActive Pending Completed Blocked All time blocked FlushWriter 0 0 1136 0 512 Looks like about 50% of flushes are blocked. This is a problem as it indicates that the IO system cannot keep up. Just ran this on the new node: nodetool netstats | grep Streaming from | wc -l 10 This is normal as the new node will most likely take tokens from all nodes in the cluster. Sorry for the multiple updates, but another thing I found was all the other existing nodes have themselves in the seeds list, but the new node does not have itself in the seeds list. Can that cause this issue? Seeds are only used when a new node is bootstrapping into the cluster and needs a set of ips to contact and discover the cluster, so this would have no impact on data sizes or streaming. In general it would be considered best practice to have a set of 2-3 seeds from each data center, with all nodes having the same seed list. What is the current output of 'nodetool compactionstats'? Could you also paste the output of nodetool status keyspace? Mark On Tue, Aug 5, 2014 at 3
Re: Node stuck during nodetool rebuild
Hi Vasilis, To further on what Rob said I believe you might be able to tune the phi detector threshold to help this operation complete, hopefully someone with direct experience of same will chime in. I have been through this operation where streams break due to a node falsely being marked down (flapping). In an attempt to mitigate this I increase the phi_convict_threshold in cassandra.yaml from 8 to 10, after which the rebuild was able to successfully complete. The default value for phi_convict_threshold is 8 with 12 being the maximum recommended value. Mark On Tue, Aug 5, 2014 at 7:22 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Aug 5, 2014 at 1:28 AM, Vasileios Vlachos vasileiosvlac...@gmail.com wrote: The problem is that the nodetool seems to be stuck, and nodetool netstats on node1 of DC2 appears to be stuck at 10% streaming a 5G file from node2 at DC1. This doesn't tally with nodetool netstats when running it against either of the DC1 nodes. The DC1 nodes don't think they stream anything to DC2. Yes, streaming is fragile and breaks and hangs forever and your only option in most cases is to stop the rebuilding node, nuke its data, and start again. I believe you might be able to tune the phi detector threshold to help this operation complete, hopefully someone with direct experience of same will chime in. =Rob
Re: `system` keyspace replication
Hi Jens, The system keyspace is configured with LocalStrategy, this strategy only stores data on the the local node. This strategy is reserved for internal use only and is used for other things such as SecondaryIndexes. You cannot change the replication factor of the 'system' keyspace. If you attempt to do so, the command will not succeed and will return the following message: Bad Request: Cannot alter system keyspace Mark On Thu, Jul 31, 2014 at 10:58 AM, Jens Rantil jens.ran...@tink.se wrote: Hi, Datastax has a documentation page about configuring replication[1]. It mentions a couple of system keyspaces that they recommend increasing replication for. However, it does not mention the `system` keyspace. Question: Is it recommended to increase replication factor for the `system` keyspace for production system? [1] http://www.datastax.com/docs/datastax_enterprise3.1/deploy/configuring_replication Thanks, Jens
Re: bootstrapping new nodes on 1.2.12
Hi Parag, 1) Would anyone be able to help me interrupt this information from OpsCenter? At a high level bootstrapping a new node has two phases, streaming and secondary index builds. I believe OpsCenter will only report active streams, the pending stream will be listed as such in OpsCenter as well. In OpsCenter rather than looking at the Data Size check the used space on the Storage Capacity pie chart, this will show how much data is on disk but not necessarily live on the node yet. Personally I would check 'nodetool netstats' to see what streams are remaining, this will list all active / pending stream and what files are to be streamed, at the moment you might just be streaming some very large files and once complete you will see a dramatic increase in data size. If streaming is complete and you use secondary indexes, check 'nodetool compcationstats' for any secondary index builds that may be taking place. 2) Is there anything I can do to speed this up? If you have the capacity you could increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml If you don't have the capacity you could add more nodes to spread the data so you stream less in future. Finally you could upgrade to 2.0.x as it contains a complete refactor of streaming and should make your streaming sessions more robust and transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286 Mark On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel ppa...@clearpoolgroup.com wrote: Hi, It’s taking a while to boostrap a 13th node into a 12 node cluster. The average node size is about 1.7TB. At the beginning of today we were close to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it would have finished by now because when I was looking on OpsCenter, there were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look again now and those same nodes haven’t progressed all day. Instead I see 9 more transfers (some of which are progressing). 1) Would anyone be able to help me interrupt this information from OpsCenter? 2) Is there anything I can do to speed this up? Thanks, Parag
Re: bootstrapping new nodes on 1.2.12
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before? This could indicate that the bootstrap process is hung due to a failed streaming session. Can you check your logs for the following line: AbstractStreamSession.java (line 110) Stream failed because /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active in background, but further streams won't be started) If that is the case you will need to wipe the node and begin the bootstrapping process again Mark On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel ppa...@clearpoolgroup.com wrote: Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before? Side question – does a new node stream from other nodes in any particular order? Perhaps this is a coincidence, but if I were to sort my hostnames in alphabetical order, it’s currently streaming from the last 2. *From:* Mark Reddy [mailto:mark.re...@boxever.com] *Sent:* Wednesday, July 30, 2014 4:42 AM *To:* user@cassandra.apache.org *Subject:* Re: bootstrapping new nodes on 1.2.12 Hi Parag, 1) Would anyone be able to help me interrupt this information from OpsCenter? At a high level bootstrapping a new node has two phases, streaming and secondary index builds. I believe OpsCenter will only report active streams, the pending stream will be listed as such in OpsCenter as well. In OpsCenter rather than looking at the Data Size check the used space on the Storage Capacity pie chart, this will show how much data is on disk but not necessarily live on the node yet. Personally I would check 'nodetool netstats' to see what streams are remaining, this will list all active / pending stream and what files are to be streamed, at the moment you might just be streaming some very large files and once complete you will see a dramatic increase in data size. If streaming is complete and you use secondary indexes, check 'nodetool compcationstats' for any secondary index builds that may be taking place. 2) Is there anything I can do to speed this up? If you have the capacity you could increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml If you don't have the capacity you could add more nodes to spread the data so you stream less in future. Finally you could upgrade to 2.0.x as it contains a complete refactor of streaming and should make your streaming sessions more robust and transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286 Mark On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel ppa...@clearpoolgroup.com wrote: Hi, It’s taking a while to boostrap a 13th node into a 12 node cluster. The average node size is about 1.7TB. At the beginning of today we were close to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it would have finished by now because when I was looking on OpsCenter, there were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look again now and those same nodes haven’t progressed all day. Instead I see 9 more transfers (some of which are progressing). 1) Would anyone be able to help me interrupt this information from OpsCenter? 2) Is there anything I can do to speed this up? Thanks, Parag
Re: bootstrapping new nodes on 1.2.12
HI Parag, I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them). Firstly, why are you forcing a GC? Do you have sufficient evidence that Cassandra is not managing the heap in the way in which your application requires? Also how are you accomplishing this full GC? Are you removing the node from the ring, forcing a GC and then adding it back in? Or are you forcing a GC while it is in the ring? 1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions? If the node that is leaving the ring is streaming data to the bootstrapping node, yes, this will break the streaming session and no further streams will be started from that node. To my knowledge, there is nothing in newer / future versions that will prevent this. 2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ? Again I'll need some more information around these manual GC's. But yes, increasing the phi value would reduce the chance of a node in the ring being marked down during a heavy gc cycle. Mark On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel ppa...@clearpoolgroup.com wrote: Mark, I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them). Few Questions 1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions? 2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ? Parag *From:* Mark Reddy [mailto:mark.re...@boxever.com] *Sent:* Wednesday, July 30, 2014 7:58 AM *To:* user@cassandra.apache.org *Subject:* Re: bootstrapping new nodes on 1.2.12 Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before? This could indicate that the bootstrap process is hung due to a failed streaming session. Can you check your logs for the following line: AbstractStreamSession.java (line 110) Stream failed because /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active in background, but further streams won't be started) If that is the case you will need to wipe the node and begin the bootstrapping process again Mark On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel ppa...@clearpoolgroup.com wrote: Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before? Side question – does a new node stream from other nodes in any particular order? Perhaps this is a coincidence, but if I were to sort my hostnames in alphabetical order, it’s currently streaming from the last 2. *From:* Mark Reddy [mailto:mark.re...@boxever.com] *Sent:* Wednesday, July 30, 2014 4:42 AM *To:* user@cassandra.apache.org *Subject:* Re: bootstrapping new nodes on 1.2.12 Hi Parag, 1) Would anyone be able to help me interrupt this information from OpsCenter? At a high level bootstrapping a new node has two phases, streaming and secondary index builds. I believe OpsCenter will only report active streams, the pending stream will be listed as such in OpsCenter as well. In OpsCenter rather than looking at the Data Size check the used space on the Storage Capacity pie chart, this will show how much data is on disk but not necessarily live on the node yet. Personally I would check 'nodetool netstats' to see what streams are remaining, this will list all active / pending stream and what files are to be streamed, at the moment you might just be streaming some very large files and once complete you will see a dramatic increase in data size. If streaming is complete and you use secondary indexes, check 'nodetool compcationstats' for any secondary index builds that may be taking place. 2) Is there anything I can do to speed this up? If you have the capacity you could increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml If you don't have the capacity you
Re: bootstrapping new nodes on 1.2.12
Our Full GC’s take about 9 seconds. If we were to increase the phi_convict_threshold to not take a node offline for a 9 second unavailability, what negative side effects can there be? When you observe these GC's do you also see the node being marked down and then back up ~9 seconds later? GC's can often happen and have no effect on gossip marking a node as down, in which case the streaming session will remain intact. The side effect of long GC's is increased latency from that node during that period. Mark On Wed, Jul 30, 2014 at 2:24 PM, Parag Patel ppa...@clearpoolgroup.com wrote: As to why we do it, we need to reevaluate because the GC optimizations we’ve made recently probably don’t require it anymore. However, prior to our optimizations we observed a benefit at our peak time. When we force a GC, we don’t remove it from the ring. This seems like a fundamental flaw in our approach. Thanks for pointing this out. For the purposes of bootstrapping, we will disable the manual GC’s to make sure we don’t interrupt the joining process. However, one unpredictable problem can always remain – a Full GC happens causing the node to go offline and causing the bootstrap to fail. To solve this, we’ll try increasing the phi_convict_threshold. Our Full GC’s take about 9 seconds. If we were to increase the phi_convict_threshold to not take a node offline for a 9 second unavailability, what negative side effects can there be? Parag *From:* Mark Reddy [mailto:mark.re...@boxever.com] *Sent:* Wednesday, July 30, 2014 9:06 AM *To:* user@cassandra.apache.org *Subject:* Re: bootstrapping new nodes on 1.2.12 HI Parag, I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them). Firstly, why are you forcing a GC? Do you have sufficient evidence that Cassandra is not managing the heap in the way in which your application requires? Also how are you accomplishing this full GC? Are you removing the node from the ring, forcing a GC and then adding it back in? Or are you forcing a GC while it is in the ring? 1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions? If the node that is leaving the ring is streaming data to the bootstrapping node, yes, this will break the streaming session and no further streams will be started from that node. To my knowledge, there is nothing in newer / future versions that will prevent this. 2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ? Again I'll need some more information around these manual GC's. But yes, increasing the phi value would reduce the chance of a node in the ring being marked down during a heavy gc cycle. Mark On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel ppa...@clearpoolgroup.com wrote: Mark, I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them). Few Questions 1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions? 2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ? Parag *From:* Mark Reddy [mailto:mark.re...@boxever.com] *Sent:* Wednesday, July 30, 2014 7:58 AM *To:* user@cassandra.apache.org *Subject:* Re: bootstrapping new nodes on 1.2.12 Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before? This could indicate that the bootstrap process is hung due to a failed streaming session. Can you check your logs for the following line: AbstractStreamSession.java (line 110) Stream failed because /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active in background, but further streams won't be started) If that is the case you will need to wipe the node and begin the bootstrapping process again Mark On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel ppa...@clearpoolgroup.com wrote: Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending
Re: nodetool status returning negative load values on 2.1.0-rc4
Looks like you are running into this issue: https://issues.apache.org/jira/browse/CASSANDRA-7239 Mark On Tue, Jul 29, 2014 at 5:32 PM, Jeronimo de A. Barros jeronimo.bar...@gmail.com wrote: Hi everyone, After upgrade and clean install on a 5 nodes cluster: Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.3.50 -123645 bytes 256 61.1% 8ca12c95-9f20-4121-b6c2-0d4e09f255d9 rack1 UN 192.168.3.20 -142445 bytes 256 59.3% a1e2db2b-53a2-4cf7-8246-ea86104276d4 rack1 UN 192.168.3.40 -180088 bytes 256 56.7% 842107c0-91c6-4dbc-b6e4-81d04f3f492b rack1 UN 192.168.3.25 -123678 bytes 256 63.5% 6a50a6e7-d3bb-4b3f-a3a7-cb442b26d188 rack1 UN 192.168.3.24 -106578 bytes 256 59.3% 71713907-b403-4590-8079-5b750ce0b8d4 rack1 Did I do something wrong ? Thanks for any help, Jero
Re: Full GC in cassandra
What is your data size and number of columns in Cassandra. Do you do many deletions? On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha ruchir@gmail.com wrote: Really curious to know what's causing the spike in Columns and DeletedColums below : 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class Histogram: num #instances #bytes class name -- 1: 132626060 6366050880 java.nio.HeapByteBuffer 2: 28194918 3920045528 [B 3: 78124737 3749987376 edu.stanford.ppl.concurrent.SnapTreeMap$Node * 4: 67650128 2164804096 2164804096 org.apache.cassandra.db.Column* * 5: 16315310 522089920 org.apache.cassandra.db.DeletedColumn* 6: 6818 392489608 [I 7: 2844374 273059904 edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch 8: 5727000 22908 java.util.TreeMap$Entry 9:767742 182921376 [J 10: 2932832 140775936 edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder 11: 2844375 9102 edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch 12: 4145131 66322096 java.util.concurrent.atomic.AtomicReference 13:437874 64072392 [C 14: 2660844 63860256 java.util.concurrent.ConcurrentSkipListMap$Node 15: 4920 62849864 [[B 16: 1632063 52226016 edu.stanford.ppl.concurrent.SnapTreeMap
Re: Cassandra version upgrade path
Check https://github.com/apache/cassandra/blob/trunk/NEWS.txt to see the steps required, if any, when upgrading. As a general practice you should also reviw the changes at https://github.com/apache/cassandra/blob/trunk/CHANGES.txt If upgrading major versions it is often recommended to be on the latest version of your current major version first before attempting an upgrade and test in a non production environment beforehand. Mark On Sat, Jul 26, 2014 at 8:01 AM, Rahul Neelakantan ra...@rahul.be wrote: Is there a tool/guide/document that shows you upgrade paths for different versions of Cassandra? For example if you are on version X and want to go to version Y, here are the intermediate versions you need to upgrade to and here are some special precautions/Steps you need to take for so and so version while upgrading Rahul Neelakantan 678-451-4545
Re: How to get rid of stale info in gossip
After removing a node, it's information can persist in the Gossiper for up to 3 days, after which time it should be removed. Are you having issues with a removed node state persisting for longer? Mark On Fri, Jul 25, 2014 at 11:33 AM, Rahul Neelakantan ra...@rahul.be wrote: Is there a way to get rid of stale information that shows up for removed/dead nodes in gossip, without a complete cluster bounce? Rahul Neelakantan
Re: do all nodes actually send the data to the coordinator when doing a read?
However what's about timestamp checking ? You're saying that the coordinator checks for the digest of data (cell value) from both nodes but if the cell name have different timestamp would it still request a full data read to the node having the most recent time ? When generating the hash to be returned to the coordinator, the possible cell values that are used are name, value, timestamp, serialisationFlag and depending on the cell type possibly other values. From there the hashes are compared and if there is a mismatch the data is requested from the replicas. At this stage the RowDataResolver will compute the most recent version of each column, and send diffs to out-of-date replicas. On Fri, Jul 25, 2014 at 11:32 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Yes. Digest includes following: {name, value, timestamp, flags(deleted, expired, etc.)} On Fri, Jul 25, 2014 at 2:33 PM, DuyHai Doan doanduy...@gmail.com wrote: Thanks Mark for the very detailed explanation. However what's about timestamp checking ? You're saying that the coordinator checks for the digest of data (cell value) from both nodes but if the cell name have different timestamp would it still request a full data read to the node having the most recent time ? On Fri, Jul 25, 2014 at 11:25 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Brian, A read request will be handled in the following manner: Once the coordinator receives a read request it will firstly determine the replicas responsible for the data. From there those replicas are sorted by proximity to the coordinator. The closest node as determined by proximity sorting will be sent a command to perform an actual data read i.e. return the data to the coordinator If you have a Replication Factor (RF) of 3 and are reading at CL.QUORUM, one additional node will be sent a digest query. A digest query is like a read query except that instead of the receiving node actually returning the data, it only returns a digest (hash) of the would-be data. The reason for this is to discover whether the two nodes contacted agree on what the current data is, without sending the data over the network. Obviously for large data sets this is an effective bandwidth saver. Back on the coordinator node if the data and the digest match the data is returned to the client. If the data and digest do not match, a full data read is performed against the contacted replicas in order to guarantee that the most recent data is returned. Asynchronously in the background, the third replica is checked for consistency with the first two, and if needed, a read repair is initiated for that node. Mark On Fri, Jul 25, 2014 at 9:12 PM, Brian Tarbox briantar...@gmail.com wrote: We're considering a C* setup with very large columns and I have a question about the details of read. I understand that a read request gets handled by the coordinator which sends read requests to quorum of the nodes holding replicas of the data, and once quorum nodes have replied with consistent data it is returned to the client. My understanding is that each of the nodes actually sends the full data being requested to the coordinator (which in the case of very large columns would involve lots of network traffic). Is that right? The alternative (which I don't think is the case but I've been asked to verify) is that the replicas first send meta-data to the coordinator which then asks one replica to send the actual data. Again, I don't think this is the case but was asked to confirm. Thanks. -- http://about.me/BrianTarbox
Re: What is C*?
Yes you are correct, Cassandra is often abbreviated as C*. With most languages and applications being referenced by their acronym / abbreviation, I guess you just have to pick one that is available. I assume if someone creates a new language and wants to name it C*, they will see that it is taken and avoid the conflict. Mark On Wed, Jul 23, 2014 at 11:50 PM, jcllings jclli...@gmail.com wrote: Keep seeing refs to C*. I assume that C* == Cassandra? IMHO not a good ref to use what with C, C++, C#. A language called C* can't be far behind assuming it doesn't already exist. ;-) Jim C.
Re: Case Study from Migrating from RDBMS to Cassandra
PlanetCassandra has a collection of migration use cases: http://planetcassandra.org/mysql-to-cassandra-migration/ http://planetcassandra.org/oracle-to-cassandra-migration/ If they don't contain the information you need I'm sure you could reach out to the companies involved or DataStax itself to get what you require. Mark On Wed, Jul 23, 2014 at 5:28 AM, Surbhi Gupta surbhi.gupt...@gmail.com wrote: Thansk Shane, Howover i am looking for any Proof of Concepts kind of document . Does anybody has complete end to end document which contains the application overview, How they have migrated from RDBMS to Cassandra? What are the things to consider? How they have converted data model and after the new data model? How they have loaded the data into cassadnra ? Performance test after and before migartion etc. Thanks Surbhi On 23 July 2014 08:51, Shane Hansen shanemhan...@gmail.com wrote: There's lots of info on migrating from a relational database to Cassandra here: http://www.datastax.com/relational-database-to-nosql On Tue, Jul 22, 2014 at 7:45 PM, Surbhi Gupta surbhi.gupt...@gmail.com wrote: Hi, Does anybody has the case study for Migrating from RDBMS to Cassandra ? Thanks
Re: Why is the cassandra documentation such poor quality?
Hi Kevin, The difference here is that the Apache Cassandra site is maintained by the community whereas the DataStax site is maintained by paid employees with a vested interest in producing documentation. With DataStax having some comprehensive docs, I guess the desire for people to maintain the Apache site has dwindled. However, if you are interested in contributing to it and bringing it back up to standard you can, thus is the freedom of open source. Mark On Wed, Jul 23, 2014 at 2:54 AM, Kevin Burton bur...@spinn3r.com wrote: This document: https://wiki.apache.org/cassandra/Operations … for example. Is extremely out dated… does NOT reflect 2.x releases certainly. Mentions commands that are long since removed/deprecated. Instead of giving bad documentation, maybe remove this and mark it as obsolete. The datastax documentation… is … acceptable I guess. My main criticism there is that a lot of it it is in their blog. Kevin -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: When will a node's host ID change?
It won't. Cassandra stores a node's host ID in an end-point to host ID mapping, each ID must be unique and cannot be changed after the fact. Mark On Mon, Jul 21, 2014 at 4:44 PM, John Sanda john.sa...@gmail.com wrote: Under what circumstances, if any, will a node's host ID change? - John
Re: Only one node in Cassandra AMI cluster
Have you setup your security group correctly so that all nodes can communicate. Is there anything in the logs to suggest nodes could not communicate to each other? When you log into the other instances, is Cassandra running correctly? On Tue, Jul 15, 2014 at 6:28 PM, Lars Schouw schou...@yahoo.com wrote: When I start Cassandra AMI cluster with 4 nodes on AWS and open the OpsCenter there is only one node shown. I followed the doc http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMILaunch.html I choose Region AMI us-east-1 ami-ada2b6c4 I set the number of Ec2 instances to 4 and advanced details to --clustername myDSCcluster --totalnodes 4 --version community [image: enter image description here] Then after launching and waiting a bit I can only see one node in the cluster ubuntu@ip-:~$ nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns Host ID Rack UN XXX 59.24 KB 256 100.0% 1c
Re: Only one node in Cassandra AMI cluster
Good to hear. Mark On Tue, Jul 15, 2014 at 8:16 PM, Lars Schouw schou...@yahoo.com wrote: Yes, it works when setting the security group so all traffic is allowed from anywhere. I also found the doc that tells me how to add the more restrictive security http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMISecurityGroup.html On Tuesday, July 15, 2014 6:43 PM, Mark Reddy mark.re...@boxever.com wrote: Have you setup your security group correctly so that all nodes can communicate. Is there anything in the logs to suggest nodes could not communicate to each other? When you log into the other instances, is Cassandra running correctly? On Tue, Jul 15, 2014 at 6:28 PM, Lars Schouw schou...@yahoo.com wrote: When I start Cassandra AMI cluster with 4 nodes on AWS and open the OpsCenter there is only one node shown. I followed the doc http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMILaunch.html I choose Region AMI us-east-1 ami-ada2b6c4 I set the number of Ec2 instances to 4 and advanced details to --clustername myDSCcluster --totalnodes 4 --version community [image: enter image description here] Then after launching and waiting a bit I can only see one node in the cluster ubuntu@ip-:~$ nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns Host ID Rack UN XXX 59.24 KB 256 100.0% 1c
Re: Small bug on Cassandra 2.1.0-rc3 and fix
Hi Carlos, I have just downloaded 2.1.0-rc3 to validate this and the cassandra.in.sh contains the correct reference to the jamm-0.2.6.jar This can also been seen on the trunk: https://github.com/apache/cassandra/blob/trunk/bin/cassandra.in.sh I am unsure why you are seeing jamm-0.2.5.jar in your cassandra.in.sh file if you have taken down a fresh 2.1.0-rc, can you give any further information? Mark On Fri, Jul 11, 2014 at 9:49 PM, Carlos Scheidecker nando@gmail.com wrote: Hello all, Probably not here that I should submit it, but I have been testing Cassandra 2.1.0 for quite a while. I have tried to subscribe to the dev list as well. Even now on rc3, there is still a small bug on cassandra.in.sh used, among others, for nodetool settings. The error is that it is still loading jamm-0.2.5.jar while the package has been updated to 0.2.6 on this version. Where it says: then JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar fi Should be inside cassandra.in.sh then JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.6.jar fi Otherwise, of course, nodetool will not start unless you have version 0.2.5 from somewhere else in your classpath.
Re: UnavailableException
Can you post the output of nodetool status and your Astyanax connection settings? On Fri, Jul 11, 2014 at 9:06 PM, Ruchir Jha ruchir@gmail.com wrote: This is how we create our keyspace. We just ran this command once through a cqlsh session on one of the nodes, so don't quite understand what you mean by check that your DC names match up CREATE KEYSPACE prod WITH replication = { 'class': 'NetworkTopologyStrategy', 'datacenter1': '3' }; On Fri, Jul 11, 2014 at 3:48 PM, Chris Lohfink clohf...@blackbirdit.com wrote: What replication strategy are you using? if using NetworkTopolgyStrategy double check that your DC names match up (case sensitive) Chris On Jul 11, 2014, at 9:38 AM, Ruchir Jha ruchir@gmail.com wrote: Here's the complete stack trace: com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: TokenRangeOfflineException: [host=ny4lpcas5.fusionts.corp(10.10.20.47):9160, latency=22784(42874), attempts=3]UnavailableException() at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28) at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151) at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:485) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:79) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:123) Caused by: UnavailableException() at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20841) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964) at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:129) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:126) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) ... 12 more On Fri, Jul 11, 2014 at 9:11 AM, Prem Yadav ipremya...@gmail.com wrote: Please post the full exception. On Fri, Jul 11, 2014 at 1:50 PM, Ruchir Jha ruchir@gmail.com wrote: We have a 12 node cluster and we are consistently seeing this exception being thrown during peak write traffic. We have a replication factor of 3 and a write consistency level of QUORUM. Also note there is no unusual Or Full GC activity during this time. Appreciate any help. Sent from my iPhone
Re: Small bug on Cassandra 2.1.0-rc3 and fix
This looks to have been fixed only yesterday: https://github.com/apache/cassandra/commit/420c68511417e917925898e8c0637c9df3e317b0 On Fri, Jul 11, 2014 at 10:49 PM, Carlos Scheidecker nando@gmail.com wrote: Mark, Sure! Actually it might be a package problem. Not installing Cassandra by hand as I need to test it across a bigger dev cluster. As I am using Ubuntu 14.04 on the test nodes using the following repository: ### Cassandra deb http://www.apache.org/dist/cassandra/debian 21x main # disabled on upgrade to saucy deb-src http://www.apache.org/dist/cassandra/debian 21x main # disabled on upgrade to saucy ### Cassandra Therefore, it seems to be a packaging problem as the file /usr/share/cassandra/cassandra.in.sh is the culprid. Here is its entire content after the fix: root@hp-dv7-7243cl:/etc/apt# cat /usr/share/cassandra/cassandra.in.sh # The directory where Cassandra's configs live (required) CASSANDRA_CONF=/etc/cassandra CASSANDRA_HOME=/usr/share/cassandra # The java classpath (required) if [ -n $CLASSPATH ]; then CLASSPATH=$CLASSPATH:$CASSANDRA_CONF else CLASSPATH=$CASSANDRA_CONF fi for jar in /usr/share/cassandra/lib/*.jar; do CLASSPATH=$CLASSPATH:$jar done for jar in /usr/share/cassandra/*.jar; do CLASSPATH=$CLASSPATH:$jar done CLASSPATH=$CLASSPATH:$EXTRA_CLASSPATH # set JVM javaagent opts to avoid warnings/errors if [ $JVM_VENDOR != OpenJDK -o $JVM_VERSION \ 1.6.0 ] \ || [ $JVM_VERSION = 1.6.0 -a $JVM_PATCH_VERSION -ge 23 ] then JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.6.jar fi On Fri, Jul 11, 2014 at 3:02 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Carlos, I have just downloaded 2.1.0-rc3 to validate this and the cassandra.in.sh contains the correct reference to the jamm-0.2.6.jar This can also been seen on the trunk: https://github.com/apache/cassandra/blob/trunk/bin/cassandra.in.sh I am unsure why you are seeing jamm-0.2.5.jar in your cassandra.in.sh file if you have taken down a fresh 2.1.0-rc, can you give any further information? Mark On Fri, Jul 11, 2014 at 9:49 PM, Carlos Scheidecker nando@gmail.com wrote: Hello all, Probably not here that I should submit it, but I have been testing Cassandra 2.1.0 for quite a while. I have tried to subscribe to the dev list as well. Even now on rc3, there is still a small bug on cassandra.in.sh used, among others, for nodetool settings. The error is that it is still loading jamm-0.2.5.jar while the package has been updated to 0.2.6 on this version. Where it says: then JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar fi Should be inside cassandra.in.sh then JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.6.jar fi Otherwise, of course, nodetool will not start unless you have version 0.2.5 from somewhere else in your classpath.
Re: TTransportException (java.net.SocketException: Broken pipe)
Hi Bhaskar, Can you check your limits using 'ulimit -a'? The default is 1024, which needs to be increased if you have not done so already. Here you will find a list of recommended production settings: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html Mark On Tue, Jul 8, 2014 at 5:30 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: Hi, I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad core ubuntu server with 32gb ram) and trying to ingest 1MB values using cassandra-stress. It works fine for a while(1600secs) but after ingesting around 120GB data, I start getting the following error: Operation [70668] retried 10 times - error inserting key 0070668 ((TTransportException): java.net.SocketException: Broken pipe) The cassandra server is still running but in the system.log I see the below mentioned errors. ERROR [COMMIT-LOG-ALLOCATOR] 2014-07-07 22:39:23,617 CassandraDaemon.java (line 198) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.lang.NoClassDefFoundError: org/apache/cassandra/db/commitlog/CommitLog$4 at org.apache.cassandra.db.commitlog.CommitLog.handleCommitError(CommitLog.java:374) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.db.commitlog.CommitLog$4 at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 4 more Caused by: java.io.FileNotFoundException: /path/2.0.7/cassandra/build/classes/main/org/apache/cassandra/db/commitlog/CommitLog$4.class (Too many open files) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:146) at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1086) at sun.misc.Resource.cachedInputStream(Resource.java:77) at sun.misc.Resource.getByteBuffer(Resource.java:160) at java.net.URLClassLoader.defineClass(URLClassLoader.java:436) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ... 10 more ERROR [FlushWriter:7] 2014-07-07 22:39:24,924 CassandraDaemon.java (line 198) Exception in thread Thread[FlushWriter:7,5,main] FSWriteError in /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417) at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db (Too many open files) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.init(FileOutputStream.java:221) at java.io.FileOutputStream.init(FileOutputStream.java:110) at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466) ... 9 more There are around 9685 open files by the Cassandra server process (using lsof), 3938 commit log segments in /cassandra/commitlog and around 572 commit log segments deleted during the course of the test. I am wondering what is causing Cassandra to open so many files, is the flushing slow? or something else? I tried increasing the flush writers, but that didn't help. Regards, Bhaskar CREATE KEYSPACE Keyspace1 WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '1' }; CREATE TABLE Standard1 ( key blob, C0 blob, PRIMARY KEY (key) ) WITH COMPACT STORAGE AND
Re: Disable vnode
Hi Rameez, I have never done a migration from vnodes to non-vnodes however I would imagine that the procedure would be the same as its counterpart. As always testing in dev should be done first. To move from vnodes to non-vodes I would add a new datacenter to the cluster with vnodes disabled and rebuild from your vnode cluster. You can find some more details about adding a data center to your cluster here: http://datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html?scroll=task_ds_hmp_54q_gk__task_ds_hmp_54q_gk_unique_1 Mark On Fri, Jul 4, 2014 at 2:43 PM, Rameez Thonnakkal ssram...@gmail.com wrote: hello Team, I am looking for standard operating procedure to disable vnode in a production cluster. This is to enable solr which doesn't work with a cassandra cluster having vnode enabled. Any suggestions/ Thanks, Rameez
Re: why aren't my nodes show same percentage of ownership
You should run nodetool ring and specify a keyspace, otherwise the ownership information will be nonsense. https://issues.apache.org/jira/browse/CASSANDRA-7173 Mark On Mon, Jun 30, 2014 at 6:13 PM, S C as...@outlook.com wrote: When I run nodetoool ring I see the ownership with different percentages. However, LOAD column shows not a huge deviation. Why is that? I am using Datastax 3.0. http://pastebin.com/EcWbZn26
Re: Index with same Name but different keyspace
Can you share your schema and the commands you are running? On Thu, May 15, 2014 at 7:54 PM, mahesh rajamani rajamani.mah...@gmail.comwrote: Hi, I am using Cassandra 2.0.5 version. I trying to setup 2 keyspace with same tables for different testing. While creating index on the tables, I realized I am not able to use the same index name though the tables are in different keyspaces. Is maintaining unique index name across keyspace is must/feature? -- Regards, Mahesh Rajamani
Re: Mutation messages dropped
Yes, please see http://wiki.apache.org/cassandra/FAQ#dropped_messages for further details. Mark On Fri, May 9, 2014 at 12:52 PM, Raveendran, Varsha IN BLR STS varsha.raveend...@siemens.com wrote: Hello, I am writing around 10Million records continuously into a single node Cassandra (2.0.5) . In the Cassandra log file I see an entry “*272 MUTATION messages dropped in last 5000ms*” . Does this mean that 272 records were not written successfully? Thanks, Varsha
Re: Can the seeds list be changed at runtime?
Hi Boying, From Datastax documentation: http://www.datastax.com/documentation/cassandra/1.2/cassandra/architecture/architectureGossipAbout_c.html The seed node designation has no purpose other than bootstrapping the gossip process for new nodes joining the cluster. Seed nodes are not a single point of failure, nor do they have any other special purpose in cluster operations beyond the bootstrapping of nodes. For this reason you can change the seed list on existing node at any time, as the node itself will already be aware of the cluster and would not need to rely on the seed list to join. For new nodes that you want to bootstrap into the cluster you can specify any nodes you wish. Mark On Tue, Apr 29, 2014 at 2:57 AM, Lu, Boying boying...@emc.com wrote: Hi, All, I wonder if I can change the seeds list at runtime. i.e. without change the yaml file and restart DB service? Thanks Boying
Re: JDK 8
Datastax recommended during a long time (still now ?) to use Java 6 Java 6 is recommended for version 1.2 Java 7 is required for version 2.0 Mark On Tue, Apr 29, 2014 at 10:19 AM, Alain RODRIGUEZ arodr...@gmail.comwrote: Looks like it will be like with the version 7... Cassandra has been compatible with this version for a long time, but there were no official validations and Datastax recommended during a long time (still now ?) to use Java 6. The best thing would be to use older versions. If for some reason you use Java 8, run some tests and let us know how things goes :). Good luck with this. 2014-04-29 1:09 GMT+02:00 Colin co...@clark.ws: It seems to run ok, but I havent seen it yet in production on 8. -- *Colin Clark* +1-320-221-9531 On Apr 28, 2014, at 4:01 PM, Ackerman, Mitchell mitchell.acker...@pgi.com wrote: I’ve been searching around, but cannot find any information as to whether Cassandra runs on JRE 8. Any information on that? Thanks, Mitchell
Re: Bootstrap Timing
If all streams have completed, the node could be still rebuilding secondary indexes? Try looking at 'nodetool compactionstats' for this. On Fri, Apr 18, 2014 at 1:05 PM, Phil Burress philburress...@gmail.comwrote: nodetool netstats shows 84 files. They are all at 100%. Nothing showing in Pending or Active for Read Repair Stats. I'm assuming this means it's done. But it still shows JOINING. Is there an undocumented step I'm missing here? This whole process seems broken to me. On Thu, Apr 17, 2014 at 4:32 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Apr 16, 2014 at 1:56 PM, Phil Burress philburress...@gmail.comwrote: I've shut down two of the nodes and am bootstrapping one right now. Is there any way to tell when it will finish bootstrapping? nodetool netstats will show the progress of the streams involved, which could help you estimate. =Rob
Re: How to clear all data using CQL?
To get a list of all keyspaces via CQL you can run: *describe keyspaces *or *select keyspace_name from system.schema_keyspaces;* On Wed, Apr 16, 2014 at 9:23 AM, Sebastian Schmidt isib...@gmail.comwrote: Hey I want to do unit tests for software that uses Cassandra as storage backend. Using the Hector client, I could just iterate over all keyspaces and delete them. In CQL, there is no query to list all keyspaces. How can I drop all Keyspaces from Cassandra, without knowing every single Keyspace name? Regards, Sebastian
Re: Unable to complete request: one or more nodes were unavailable.
create keyspace twitter with replication = {'class':'SimpleStrategy', 'replication_factor' : 3} Your replication factor is your issue here, you have a single node and a RF=3. For a single node setup your RF should be 1. You can find more info about replication here: http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html On Wed, Apr 16, 2014 at 1:44 PM, Vivek Mishra mishra.v...@gmail.com wrote: Hi, Mine is a simple case. Running on single node only. Keyspace is: create keyspace twitter with replication = {'class':'SimpleStrategy', 'replication_factor' : 3} -Vivek On Wed, Apr 16, 2014 at 1:27 AM, Tupshin Harper tups...@tupshin.comwrote: Please provide your keyspace definition, and the output of nodetool ring -Tupshin On Apr 15, 2014 3:52 PM, Vivek Mishra mishra.v...@gmail.com wrote: Hi, I am trying Cassandra light weight transaction support with Cassandra 2.0.4 cqlsh:twitter create table user(user_id text primary key, namef text); cqlsh:twitter insert into user(user_id,namef) values('v','ff') if not exists; *Unable to complete request: one or more nodes were unavailable.* Any suggestions? -Vivek
Re: Unable to complete request: one or more nodes were unavailable.
The Paxos protocol used for CAS operations will always use at least a consistency level effectively equivalent to QUORUM (called SERIAL) when writing, even if you explicitly specify a lower level, e.g. ANY or ONE. Setting consistency level to ALL will make the write execute on all replicas if the condition is met, but the comparison itself is executed against a QUORUM number of nodes. As a result, a write operation with ALL consistency level that fails to meet the specified check may not throw an Exception, even if some replica nodes are not accessible. On Wed, Apr 16, 2014 at 2:00 PM, Tupshin Harper tups...@tupshin.com wrote: No, but you do need a quorum of nodes. http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html SERIAL A write must be written conditionally to the commit log and memory table on a quorum of replica nodes. Used to achievelinearizable consistencyhttp://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_tunable_consistency_c.html#concept_ds_f4h_hwx_zjfor lightweight transactions by preventing unconditional updates. On Apr 16, 2014 5:56 AM, Vivek Mishra mishra.v...@gmail.com wrote: Thanks Mark. does this mean with RF=3, all 3 nodes must be up and running for CAS updates? -Vivek On Wed, Apr 16, 2014 at 6:22 PM, Mark Reddy mark.re...@boxever.comwrote: create keyspace twitter with replication = {'class':'SimpleStrategy', 'replication_factor' : 3} Your replication factor is your issue here, you have a single node and a RF=3. For a single node setup your RF should be 1. You can find more info about replication here: http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html On Wed, Apr 16, 2014 at 1:44 PM, Vivek Mishra mishra.v...@gmail.comwrote: Hi, Mine is a simple case. Running on single node only. Keyspace is: create keyspace twitter with replication = {'class':'SimpleStrategy', 'replication_factor' : 3} -Vivek On Wed, Apr 16, 2014 at 1:27 AM, Tupshin Harper tups...@tupshin.comwrote: Please provide your keyspace definition, and the output of nodetool ring -Tupshin On Apr 15, 2014 3:52 PM, Vivek Mishra mishra.v...@gmail.com wrote: Hi, I am trying Cassandra light weight transaction support with Cassandra 2.0.4 cqlsh:twitter create table user(user_id text primary key, namef text); cqlsh:twitter insert into user(user_id,namef) values('v','ff') if not exists; *Unable to complete request: one or more nodes were unavailable.* Any suggestions? -Vivek
Re: GC histogram analysis
Do you delete and/or set TTLs on your data? On Wed, Apr 16, 2014 at 2:14 PM, Ruchir Jha ruchir@gmail.com wrote: Hi, I am trying to investigate ParNew promotion failures happening routinely in production. As part of this exercise, I enabled -XX:PrintHistogramBeforeFullGC and saw the following output. As you can see there are a ton of Columns, ExpiringColumns and DeletedColumns before GC ran and these numbers go down significantly right after GC. Why are there so many expiring and deleted columns? *Before GC:* num #instances #bytes class name -- 1: 113539896 5449915008 java.nio.*HeapByteBuffer* 2: 15979061 2681431488 [B 3: 36364545 1745498160 edu.stanford.ppl.concurrent.SnapTreeMap$Node 4: 23583282 754665024 org.apache.cassandra.db.*Column* 5: 8745428 209890272 java.util.concurrent.ConcurrentSkipListMap$Node 6: 5062619 202504760 org.apache.cassandra.db. *ExpiringColumn* 7: 45261 198998216 [I 8: 1801535 172947360 edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch 9: 1473677 169570040 [J 10: 4713304 113119296 java.lang.Double 11: 3246729 103895328 org.apache.cassandra.db. *DeletedColumn* *After GC:* num #instances #bytes class name -- 1: 11807204 1505962728 [B 2: 12525536 601225728 java.nio.*HeapByteBuffer* 3: 8839073 424275504 edu.stanford.ppl.concurrent.SnapTreeMap$Node 4: 8194496 262223872 org.apache.cassandra.db.*Column* cache.KeyCacheKey 17:432119 17284760 org.apache.cassandra.db.*ExpiringColumn* 21:351096 11235072 org.apache.cassandra.db.*DeletedColumn*
Re: Datastax OPSCENTER
Datastax have a comprehensive installation and configuration guide here: http://www.datastax.com/documentation/opscenter/4.1/opsc/install/opscInstallation_g.html On Tue, Apr 15, 2014 at 9:13 AM, Aravindan T aravinda...@tcs.com wrote: Hi, Could anyone please guide me on installation procedure of Cassandra using Datastax OPSCENTER? I couldnt find any documentation in regards to the same. Your help is much appreciated. Thanks, AT =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Re: bloom filter + suddenly smaller CF
Hi Will, You can run 'nodetool upgradesstables', this will rewrite the SSTables and regenerate the bloom filters for those tables, This will reduce their usage. Mark On Mon, Apr 14, 2014 at 3:16 PM, William Oberman ober...@civicscience.comwrote: Ah, so I could change the chance value to poke it. Good to know! On Mon, Apr 14, 2014 at 10:12 AM, Michal Michalski michal.michal...@boxever.com wrote: Sorry, I misread the question - I thought you've also changed FP chance value, not only removed the data. Kind regards, Michał Michalski, michal.michal...@boxever.com On 14 April 2014 15:07, Michal Michalski michal.michal...@boxever.comwrote: Did you set Bloom Filter's FP chance before or after the step 3) above? If you did it before, C* should build Bloom Filters properly. If not - that's the reason. Kind regards, Michał Michalski, michal.michal...@boxever.com On 14 April 2014 15:04, William Oberman ober...@civicscience.comwrote: I didn't cross link my thread, but the basic idea is I've done: 1.) Process that deleted ~900M of ~1G rows from a CF 2.) Set GCGraceSeconds to 0 on CF 3.) Run nodetool compact on all N nodes And I checked, and all N nodes have bloom filters using 1.5 +/- .2 GB of RAM (I didn't explicitly write down the before numbers, but they seem about the same) . So, compaction didn't change the BF's (unless cassandra needs a 2nd compaction to see all of the data cleared by the 1st compaction). will On Mon, Apr 14, 2014 at 9:52 AM, Michal Michalski michal.michal...@boxever.com wrote: Bloom filters are built on creation / rebuild of SSTable. If you removed the data, but the old SSTables weren't compacted or you didn't rebuild them manually, bloom filters will stay the same size. M. Kind regards, Michał Michalski, michal.michal...@boxever.com On 14 April 2014 14:44, William Oberman ober...@civicscience.comwrote: I had a thread on this forum about clearing junk from a CF. In my case, it's ~90% of ~1 billion rows. One side effect I had hoped for was a reduction in the size of the bloom filter. But, according to nodetool cfstats, it's still fairly large (~1.5GB of RAM). Do bloom filters ever resize themselves when the CF suddenly gets smaller? My next test will be restarting one of the instances, though I'll have to wait on that operation so I thought I'd ask in the meantime. will
Re: Cassandra disk usage
What are you storing these 15 chars as; string, int, double, etc.? 15 chars does not translate to 15 bytes. You may be mixing up replication factor and quorum when you say Cassandra cluster has 3 servers, and data is stored in quorum ( 2 servers ). You read and write at quorum (N/2)+1 where N=total_number_of_nodes and your data is replicated to the number of nodes you specify in your replication factor. Could you clarify? Also if you are concerned about disk usage, why are you storing the same 15 char value in both the column name and value? You could just store it as the name and half your data usage :) On Sun, Apr 13, 2014 at 4:26 PM, Yulian Oifa oifa.yul...@gmail.com wrote: I have column family with 2 raws. 2 raws have overall 100 million columns. Each columns have name of 15 chars ( digits ) and same 15 chars in value ( also digits ). Each column should have 30 bytes. Therefore all data should contain approximately 3GB. Cassandra cluster has 3 servers , and data is stored in quorum ( 2 servers ). Therefore each server should have 3GB*2/3=2GB of data for this column family. Table is almost never changed , data is only removed from this table , which possibly created tombstones , but it should not increase the usage. However when i check the data i see that each server has more then 4GB of data ( more then twice of what should be). server 1: -rw-r--r-- 1 root root 3506446057 Dec 26 12:02 freeNumbers-g-264-Data.db -rw-r--r-- 1 root root 814699666 Dec 26 12:24 freeNumbers-g-281-Data.db -rw-r--r-- 1 root root 198432466 Dec 26 12:27 freeNumbers-g-284-Data.db -rw-r--r-- 1 root root 35883918 Apr 12 20:07 freeNumbers-g-336-Data.db server 2: -rw-r--r-- 1 root root 3448432307 Dec 26 11:57 freeNumbers-g-285-Data.db -rw-r--r-- 1 root root 762399716 Dec 26 12:22 freeNumbers-g-301-Data.db -rw-r--r-- 1 root root 220887062 Dec 26 12:23 freeNumbers-g-304-Data.db -rw-r--r-- 1 root root 54914466 Dec 26 12:26 freeNumbers-g-306-Data.db -rw-r--r-- 1 root root 53639516 Dec 26 12:26 freeNumbers-g-305-Data.db -rw-r--r-- 1 root root 53007967 Jan 8 15:45 freeNumbers-g-314-Data.db -rw-r--r-- 1 root root 413717 Apr 12 18:33 freeNumbers-g-359-Data.db server 3: -rw-r--r-- 1 root root 4490657264 Apr 11 18:20 freeNumbers-g-358-Data.db -rw-r--r-- 1 root root 389171 Apr 12 20:58 freeNumbers-g-360-Data.db -rw-r--r-- 1 root root 4276 Apr 11 18:20 freeNumbers-g-358-Statistics.db -rw-r--r-- 1 root root 4276 Apr 11 18:24 freeNumbers-g-359-Statistics.db -rw-r--r-- 1 root root 4276 Apr 12 20:58 freeNumbers-g-360-Statistics.db -rw-r--r-- 1 root root976 Apr 11 18:20 freeNumbers-g-358-Filter.db -rw-r--r-- 1 root root208 Apr 11 18:24 freeNumbers-g-359-Data.db -rw-r--r-- 1 root root 78 Apr 11 18:20 freeNumbers-g-358-Index.db -rw-r--r-- 1 root root 52 Apr 11 18:24 freeNumbers-g-359-Index.db -rw-r--r-- 1 root root 52 Apr 12 20:58 freeNumbers-g-360-Index.db -rw-r--r-- 1 root root 16 Apr 11 18:24 freeNumbers-g-359-Filter.db -rw-r--r-- 1 root root 16 Apr 12 20:58 freeNumbers-g-360-Filter.db When i try to compact i get the following notification from first server : INFO [CompactionExecutor:1604] 2014-04-13 18:23:07,260 CompactionController.java (line 146) Compacting large row USER_DATA/freeNumbers:8bdf9678-6d70-11e3-85ab-80e385abf85d (4555076689 bytes) incrementally Which confirms that there is around 4.5GB of data on that server only. Why does cassandra takes so much data??? Best regards Yulian Oifa
Re: Cassandra disk usage
i I will change the data i am storing to decrease the usage , in value i will find some small value to store.Previously i used same value since this table is index only for search purposed and does not really has value. If you don't need a value, you don't have to store anything. You can store the column name and leave the value empty, this is a common practice. 1) What should be recommended read and write consistency and replication factor for 3 nodes with option of future increase server numbers? Both consistency level and replication factor are tuneable depending on your application constraints. I'd say a CL or quorum and RF of 3 is the general practice. Still it has 1.5X of overall data how can this be resolved and what is reason for that? As Michał pointed out there is a 15 byte column overhead to consider here, where: total_column_size = column_name_size + column_value_size + 15 This link might shed some light on this: http://www.datastax.com/documentation/cassandra/1.2/cassandra/architecture/architecturePlanningUserData_t.html Also i see that data is in different size on all nodes , does that means that servers are out of sync How much is it out by? Data size may differ due to deletes, as you mentioned you do deletes. What is the output of 'nodetool ring'? On Sun, Apr 13, 2014 at 6:42 PM, Michal Michalski michal.michal...@boxever.com wrote: Each columns have name of 15 chars ( digits ) and same 15 chars in value ( also digits ). Each column should have 30 bytes. Remember about the standard Cassandra's column overhead which is, as far as I remember, 15 bytes, so it's 45 bytes in total - 50% more than you estimated, which kind of matches your 3 GB vs 4.5 GB case. There's also a per-row overhead, but I'm not sure about its size in current C* versions - I remember it was about 25 bytes or so some time ago, but it's not important in your case. Kind regards, Michał Michalski, michal.michal...@boxever.com On 13 April 2014 17:48, Yulian Oifa oifa.yul...@gmail.com wrote: Hello Mark and thanks for you reply. 1) i store is as UTF8String.All digits are from 0x30 to 0x39 and should take 1 byte each digit. Since all characters are digits it should have 15 bytes. 2) I will change the data i am storing to decrease the usage , in value i will find some small value to store.Previously i used same value since this table is index only for search purposed and does not really has value. 3) You are right i read and write in quorum and it was my mistake ( i though that if i write in quorum then data will be written to 2 nodes only). If i check the keyspace create keyspace USER_DATA with placement_strategy = 'NetworkTopologyStrategy' and strategy_options = [{19 : 3}] and durable_writes = true; it has replication factor of 3. Therefore i have several questions 1) What should be recommended read and write consistency and replication factor for 3 nodes with option of future increase server numbers? 2) Still it has 1.5X of overall data how can this be resolved and what is reason for that? 3) Also i see that data is in different size on all nodes , does that means that servers are out of sync??? Thanks and best regards Yulian Oifa On Sun, Apr 13, 2014 at 7:03 PM, Mark Reddy mark.re...@boxever.comwrote: What are you storing these 15 chars as; string, int, double, etc.? 15 chars does not translate to 15 bytes. You may be mixing up replication factor and quorum when you say Cassandra cluster has 3 servers, and data is stored in quorum ( 2 servers ). You read and write at quorum (N/2)+1 where N=total_number_of_nodes and your data is replicated to the number of nodes you specify in your replication factor. Could you clarify? Also if you are concerned about disk usage, why are you storing the same 15 char value in both the column name and value? You could just store it as the name and half your data usage :) On Sun, Apr 13, 2014 at 4:26 PM, Yulian Oifa oifa.yul...@gmail.comwrote: I have column family with 2 raws. 2 raws have overall 100 million columns. Each columns have name of 15 chars ( digits ) and same 15 chars in value ( also digits ). Each column should have 30 bytes. Therefore all data should contain approximately 3GB. Cassandra cluster has 3 servers , and data is stored in quorum ( 2 servers ). Therefore each server should have 3GB*2/3=2GB of data for this column family. Table is almost never changed , data is only removed from this table , which possibly created tombstones , but it should not increase the usage. However when i check the data i see that each server has more then 4GB of data ( more then twice of what should be). server 1: -rw-r--r-- 1 root root 3506446057 Dec 26 12:02 freeNumbers-g-264-Data.db -rw-r--r-- 1 root root 814699666 Dec 26 12:24 freeNumbers-g-281-Data.db -rw-r--r-- 1 root root 198432466 Dec 26 12:27 freeNumbers-g-284-Data.db -rw-r--r-- 1 root root 35883918 Apr 12 20:07
Re: clearing tombstones?
Correct, a tombstone will only be removed after gc_grace period has elapsed. The default value is set to 10 days which allows a great deal of time for consistency to be achieved prior to deletion. If you are operationally confident that you can achieve consistency via anti-entropy repairs within a shorter period you can always reduce that 10 day interval. Mark On Fri, Apr 11, 2014 at 3:16 PM, William Oberman ober...@civicscience.comwrote: I'm seeing a lot of articles about a dependency between removing tombstones and GCGraceSeconds, which might be my problem (I just checked, and this CF has GCGraceSeconds of 10 days). On Fri, Apr 11, 2014 at 10:10 AM, tommaso barbugli tbarbu...@gmail.comwrote: compaction should take care of it; for me it never worked so I run nodetool compaction on every node; that does it. 2014-04-11 16:05 GMT+02:00 William Oberman ober...@civicscience.com: I'm wondering what will clear tombstoned rows? nodetool cleanup, nodetool repair, or time (as in just wait)? I had a CF that was more or less storing session information. After some time, we decided that one piece of this information was pointless to track (and was 90%+ of the columns, and in 99% of those cases was ALL columns for a row). I wrote a process to remove all of those columns (which again in a vast majority of cases had the effect of removing the whole row). This CF had ~1 billion rows, so I expect to be left with ~100m rows. After I did this mass delete, everything was the same size on disk (which I expected, knowing how tombstoning works). It wasn't 100% clear to me what to poke to cause compactions to clear the tombstones. First I tried nodetool cleanup on a candidate node. But, afterwards the disk usage was the same. Then I tried nodetool repair on that same node. But again, disk usage is still the same. The CF has no snapshots. So, am I misunderstanding something? Is there another operation to try? Do I have to just wait? I've only done cleanup/repair on one node. Do I have to run one or the other over all nodes to clear tombstones? Cassandra 1.2.15 if it matters, Thanks! will
Re: clearing tombstones?
Yes, running nodetool compact (major compaction) creates one large SSTable. This will mess up the heuristics of the SizeTiered strategy (is this the compaction strategy you are using?) leading to multiple 'small' SSTables alongside the single large SSTable, which results in increased read latency. You will incur the operational overhead of having to manage compactions if you wish to compact these smaller SSTables. For all these reasons it is generally advised to stay away from running compactions manually. Assuming that this is a production environment and you want to keep everything running as smoothly as possible I would reduce the gc_grace on the CF, allow automatic minor compactions to kick in and then increase the gc_grace once again after the tombstones have been removed. On Fri, Apr 11, 2014 at 3:44 PM, William Oberman ober...@civicscience.comwrote: So, if I was impatient and just wanted to make this happen now, I could: 1.) Change GCGraceSeconds of the CF to 0 2.) run nodetool compact (*) 3.) Change GCGraceSeconds of the CF back to 10 days Since I have ~900M tombstones, even if I miss a few due to impatience, I don't care *that* much as I could re-run my clean up tool against the now much smaller CF. (*) A long long time ago I seem to recall reading advice about don't ever run nodetool compact, but I can't remember why. Is there any bad long term consequence? Short term there are several: -a heavy operation -temporary 2x disk space -one big SSTable afterwards But moving forward, everything is ok right? CommitLog/MemTable-SStables, minor compactions that merge SSTables, etc... The only flaw I can think of is it will take forever until the SSTable minor compactions build up enough to consider including the big SSTable in a compaction, making it likely I'll have to self manage compactions. On Fri, Apr 11, 2014 at 10:31 AM, Mark Reddy mark.re...@boxever.comwrote: Correct, a tombstone will only be removed after gc_grace period has elapsed. The default value is set to 10 days which allows a great deal of time for consistency to be achieved prior to deletion. If you are operationally confident that you can achieve consistency via anti-entropy repairs within a shorter period you can always reduce that 10 day interval. Mark On Fri, Apr 11, 2014 at 3:16 PM, William Oberman ober...@civicscience.com wrote: I'm seeing a lot of articles about a dependency between removing tombstones and GCGraceSeconds, which might be my problem (I just checked, and this CF has GCGraceSeconds of 10 days). On Fri, Apr 11, 2014 at 10:10 AM, tommaso barbugli tbarbu...@gmail.comwrote: compaction should take care of it; for me it never worked so I run nodetool compaction on every node; that does it. 2014-04-11 16:05 GMT+02:00 William Oberman ober...@civicscience.com: I'm wondering what will clear tombstoned rows? nodetool cleanup, nodetool repair, or time (as in just wait)? I had a CF that was more or less storing session information. After some time, we decided that one piece of this information was pointless to track (and was 90%+ of the columns, and in 99% of those cases was ALL columns for a row). I wrote a process to remove all of those columns (which again in a vast majority of cases had the effect of removing the whole row). This CF had ~1 billion rows, so I expect to be left with ~100m rows. After I did this mass delete, everything was the same size on disk (which I expected, knowing how tombstoning works). It wasn't 100% clear to me what to poke to cause compactions to clear the tombstones. First I tried nodetool cleanup on a candidate node. But, afterwards the disk usage was the same. Then I tried nodetool repair on that same node. But again, disk usage is still the same. The CF has no snapshots. So, am I misunderstanding something? Is there another operation to try? Do I have to just wait? I've only done cleanup/repair on one node. Do I have to run one or the other over all nodes to clear tombstones? Cassandra 1.2.15 if it matters, Thanks! will
Re: clearing tombstones?
To clarify, you would want to manage compactions only if you were concerned about read latency. If you update rows, those rows may become spread across an increasing number of SSTables leading to increased read latency. Thanks for providing some insight into your use case as it does differ from the norm. If you would consider 50GB a small CF and your data ingestion sufficient enough to result in more SSTables of similar size soon, yes you could run a major compaction will little operational overhead and the compaction strategies heuristics would level out after some time. On Fri, Apr 11, 2014 at 4:52 PM, Laing, Michael michael.la...@nytimes.comwrote: I have played with this quite a bit and recommend you set gc_grace_seconds to 0 and use 'nodetool compact [keyspace] [cfname]' on your table. A caveat I have is that we use C* 2.0.6 - but the space we expect to recover is in fact recovered. Actually, since we never delete explicitly (just ttl) we always have gc_grace_seconds set to 0. Another important caveat is to be careful with repair: having set gc to 0 and compacted on a node, if you then repair it, data may come streaming in from the other nodes. We don't run into this, as our gc is always 0, but others may be able to comment. ml On Fri, Apr 11, 2014 at 11:26 AM, William Oberman ober...@civicscience.com wrote: Yes, I'm using SizeTiered. I totally understand the mess up the heuristics issue. But, I don't understand You will incur the operational overhead of having to manage compactions if you wish to compact these smaller SSTables. My understanding is the small tables will still compact. The problem is that until I have 3 other (by default) tables of the same size as the big table, it won't be compacted. In my case, this might not be terrible though, right? To get into the trees, I have 9 nodes with RF=3 and this CF is ~500GB/node. I deleted like 90-95% of the data, so I expect the data to be 25-50GB after the tombstones are cleared, but call it 50GB. That means I won't compact this 50GB file until I gather another 150GB (50,50,50,50-200). But, that's not *horrible*. Now, if I only deleted 10% of the data, waiting to compact 450GB until I had another 1.3TB would be rough... I think your advice is great for people looking for normal answers in the forum, but I don't think my use case is very normal :-) will On Fri, Apr 11, 2014 at 11:12 AM, Mark Reddy mark.re...@boxever.comwrote: Yes, running nodetool compact (major compaction) creates one large SSTable. This will mess up the heuristics of the SizeTiered strategy (is this the compaction strategy you are using?) leading to multiple 'small' SSTables alongside the single large SSTable, which results in increased read latency. You will incur the operational overhead of having to manage compactions if you wish to compact these smaller SSTables. For all these reasons it is generally advised to stay away from running compactions manually. Assuming that this is a production environment and you want to keep everything running as smoothly as possible I would reduce the gc_grace on the CF, allow automatic minor compactions to kick in and then increase the gc_grace once again after the tombstones have been removed. On Fri, Apr 11, 2014 at 3:44 PM, William Oberman ober...@civicscience.com wrote: So, if I was impatient and just wanted to make this happen now, I could: 1.) Change GCGraceSeconds of the CF to 0 2.) run nodetool compact (*) 3.) Change GCGraceSeconds of the CF back to 10 days Since I have ~900M tombstones, even if I miss a few due to impatience, I don't care *that* much as I could re-run my clean up tool against the now much smaller CF. (*) A long long time ago I seem to recall reading advice about don't ever run nodetool compact, but I can't remember why. Is there any bad long term consequence? Short term there are several: -a heavy operation -temporary 2x disk space -one big SSTable afterwards But moving forward, everything is ok right? CommitLog/MemTable-SStables, minor compactions that merge SSTables, etc... The only flaw I can think of is it will take forever until the SSTable minor compactions build up enough to consider including the big SSTable in a compaction, making it likely I'll have to self manage compactions. On Fri, Apr 11, 2014 at 10:31 AM, Mark Reddy mark.re...@boxever.comwrote: Correct, a tombstone will only be removed after gc_grace period has elapsed. The default value is set to 10 days which allows a great deal of time for consistency to be achieved prior to deletion. If you are operationally confident that you can achieve consistency via anti-entropy repairs within a shorter period you can always reduce that 10 day interval. Mark On Fri, Apr 11, 2014 at 3:16 PM, William Oberman ober...@civicscience.com wrote: I'm seeing a lot of articles about a dependency between removing tombstones and GCGraceSeconds, which
Re: clearing tombstones?
Thats great Will, if you could update the thread with the actions you decide to take and the results that would be great. Mark On Fri, Apr 11, 2014 at 5:53 PM, William Oberman ober...@civicscience.comwrote: I've learned a *lot* from this thread. My thanks to all of the contributors! Paulo: Good luck with LCS. I wish I could help there, but all of my CF's are SizeTiered (mostly as I'm on the same schema/same settings since 0.7...) will On Fri, Apr 11, 2014 at 12:14 PM, Mina Naguib mina.nag...@adgear.comwrote: Levelled Compaction is a wholly different beast when it comes to tombstones. The tombstones are inserted, like any other write really, at the lower levels in the leveldb hierarchy. They are only removed after they have had the chance to naturally migrate upwards in the leveldb hierarchy to the highest level in your data store. How long that takes depends on: 1. The amount of data in your store and the number of levels your LCS strategy has 2. The amount of new writes entering the bottom funnel of your leveldb, forcing upwards compaction and combining To give you an idea, I had a similar scenario and ran a (slow, throttled) delete job on my cluster around December-January. Here's a graph of the disk space usage on one node. Notice the still-diclining usage long after the cleanup job has finished (sometime in January). I tend to think of tombstones in LCS as little bombs that get to explode much later in time: http://mina.naguib.ca/images/tombstones-cassandra-LCS.jpg On 2014-04-11, at 11:20 AM, Paulo Ricardo Motta Gomes paulo.mo...@chaordicsystems.com wrote: I have a similar problem here, I deleted about 30% of a very large CF using LCS (about 80GB per node), but still my data hasn't shrinked, even if I used 1 day for gc_grace_seconds. Would nodetool scrub help? Does nodetool scrub forces a minor compaction? Cheers, Paulo On Fri, Apr 11, 2014 at 12:12 PM, Mark Reddy mark.re...@boxever.comwrote: Yes, running nodetool compact (major compaction) creates one large SSTable. This will mess up the heuristics of the SizeTiered strategy (is this the compaction strategy you are using?) leading to multiple 'small' SSTables alongside the single large SSTable, which results in increased read latency. You will incur the operational overhead of having to manage compactions if you wish to compact these smaller SSTables. For all these reasons it is generally advised to stay away from running compactions manually. Assuming that this is a production environment and you want to keep everything running as smoothly as possible I would reduce the gc_grace on the CF, allow automatic minor compactions to kick in and then increase the gc_grace once again after the tombstones have been removed. On Fri, Apr 11, 2014 at 3:44 PM, William Oberman ober...@civicscience.com wrote: So, if I was impatient and just wanted to make this happen now, I could: 1.) Change GCGraceSeconds of the CF to 0 2.) run nodetool compact (*) 3.) Change GCGraceSeconds of the CF back to 10 days Since I have ~900M tombstones, even if I miss a few due to impatience, I don't care *that* much as I could re-run my clean up tool against the now much smaller CF. (*) A long long time ago I seem to recall reading advice about don't ever run nodetool compact, but I can't remember why. Is there any bad long term consequence? Short term there are several: -a heavy operation -temporary 2x disk space -one big SSTable afterwards But moving forward, everything is ok right? CommitLog/MemTable-SStables, minor compactions that merge SSTables, etc... The only flaw I can think of is it will take forever until the SSTable minor compactions build up enough to consider including the big SSTable in a compaction, making it likely I'll have to self manage compactions. On Fri, Apr 11, 2014 at 10:31 AM, Mark Reddy mark.re...@boxever.comwrote: Correct, a tombstone will only be removed after gc_grace period has elapsed. The default value is set to 10 days which allows a great deal of time for consistency to be achieved prior to deletion. If you are operationally confident that you can achieve consistency via anti-entropy repairs within a shorter period you can always reduce that 10 day interval. Mark On Fri, Apr 11, 2014 at 3:16 PM, William Oberman ober...@civicscience.com wrote: I'm seeing a lot of articles about a dependency between removing tombstones and GCGraceSeconds, which might be my problem (I just checked, and this CF has GCGraceSeconds of 10 days). On Fri, Apr 11, 2014 at 10:10 AM, tommaso barbugli tbarbu...@gmail.com wrote: compaction should take care of it; for me it never worked so I run nodetool compaction on every node; that does it. 2014-04-11 16:05 GMT+02:00 William Oberman ober...@civicscience.com : I'm wondering what will clear tombstoned rows? nodetool cleanup, nodetool repair, or time (as in just wait)? I had a CF
Re: How to replace cluster name without any impact?
What version are you running? As of 1.2.x you can do the following: 1. Start the cqlsh connected locally to the node. 2. Run: update system.local set cluster_name='$CLUSTER_NAME' where key='local'; 3. Run nodetool flush on the node. 4. Update the cassandra.yaml file on the node, changing the cluster_name to the same as you set in step 2. 5. Restart the node. Please be aware that you will have two partial clusters until you complete your rolling restart. Also considering that the cluster name is only a cosmetic value my opinion would be to leave it, as the risk far outweighs the benefits of changing it. Mark On Thu, Apr 10, 2014 at 2:49 AM, Check Peck comptechge...@gmail.com wrote: We have around 36 node Cassandra cluster and we have three Datacenters. Each datacenter have 12 node. We already have data flowing in Cassandra now and we cannot wipe out all our data now. Considering this - what is the right way to rename the cluster name without any or minimal impact?
Re: Migrating to new datacenter
I would go with option 1. I think it is the safer of the two options, involves less work and if something were to go wrong mid migration you can remove the second DC from your keyspace replication and have a clean break. SimpleStrategy will work across DCs. It is generally advised to not use it across DCs as it is hard to maintain and can induce latency on writes depending on DC distance, however as you are just migrating and then killing the original DC you won't face these issues. Mark On Mon, Apr 7, 2014 at 5:25 PM, Brandon McCauslin bm3...@gmail.com wrote: We're currently running a small 5 node 2.0.5 cluster in a single datacenter using the SimpleStrategy replication strategy with replication factor of 3. We want to migrate our data from our current datacenter to a new datacenter, without incurring any downtime or data loss. There is no plan to maintain or keep the old (current) datacenter after the migration is complete. I'm wondering how best to go about this migration. *Shoud We:* 1. add a 2nd datacenter as described here - Adding a datacenter to clusterhttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html, then decommission the 1st datacenterhttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_decomission_dc_t.html. Can we even do this given we're currently setup with SimpleStrategy replication? 2. or should we just start adding new nodes from the new datacenter to our existing cluster as described herehttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html? Then once we've added all the new nodes, start decommissioning all the old nodes from the old/current datacenter as described herehttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html . Thanks for your help and time!
Re: Securing Cassandra database
Hi, If you want to just secure OpsCenter itself take a look here: http://www.datastax.com/documentation/opscenter/4.1/opsc/configure/opscAssigningAccessRoles_t.html If you want to enable internal authentication and still allow OpsCenter access, you can create an OpsCenter user and once you have auth turned within the cluster update the cluster config with the user name and password for the OpsCenter user. Depending on your installation type you will find the cluster config in one of the following locations: Packaged installs: /etc/opscenter/clusters/cluster_specific.conf Binary installs: install_location/conf/clusters/cluster_specific.conf Windows installs: Program Files (x86)\DataStax Community\opscenter\conf\clusters\cluster_specific.conf Open the file and update the username and password values under the [cassandra] section: [cassandra] username = seed_hosts = api_port = password = After changing properties in this file, restart OpsCenter for the changes to take effect. Mark On Sat, Apr 5, 2014 at 6:54 AM, Check Peck comptechge...@gmail.com wrote: Hi All, We would like to secure our Cassandra database. We don't want anybody to read/write on our Cassandra database leaving our team members only. We are using Cassandra 1.2.9 in Production and we have 36 node Cassandra cluster. 12 in each colo as we have three datacenters. But we would like to have OPSCENTER working as it is working currently. Is this possible to do anyhow? Is there any settings in yaml file which we can enforce? *Raihan Jamal*
Re: Securing Cassandra database
Ok so you want to enable auth on Cassandra itself. You will want to look into the authentication and authorisation functionality then. Here is a quick overview: http://www.datastax.com/dev/blog/a-quick-tour-of-internal-authentication-and-authorization-security-in-datastax-enterprise-and-apache-cassandra This section of the docs should give you the technical details needed to move forward on this: http://www.datastax.com/documentation/cassandra/1.2/cassandra/security/securityTOC.html Mark On Sat, Apr 5, 2014 at 7:31 AM, Check Peck comptechge...@gmail.com wrote: Just to add, nobody should be able to read and write into our Cassandra database through any API *or any CQL client as well *only our team should be able to do that. On Fri, Apr 4, 2014 at 11:29 PM, Check Peck comptechge...@gmail.comwrote: Thanks Mark. But what about Cassandra database? I don't want anybody to read and write into our Cassandra database through any API only just our team should be able to do that. We are using CQL based tables so data doesn't get shown on the OPSCENTER. In our case, we would like to secure database itself. Is this possible to do as well anyhow? On Fri, Apr 4, 2014 at 11:24 PM, Mark Reddy mark.re...@boxever.comwrote: Hi, If you want to just secure OpsCenter itself take a look here: http://www.datastax.com/documentation/opscenter/4.1/opsc/configure/opscAssigningAccessRoles_t.html If you want to enable internal authentication and still allow OpsCenter access, you can create an OpsCenter user and once you have auth turned within the cluster update the cluster config with the user name and password for the OpsCenter user. Depending on your installation type you will find the cluster config in one of the following locations: Packaged installs: /etc/opscenter/clusters/cluster_specific.conf Binary installs: install_location/conf/clusters/cluster_specific.conf Windows installs: Program Files (x86)\DataStax Community\opscenter\conf\clusters\cluster_specific.conf Open the file and update the username and password values under the [cassandra] section: [cassandra] username = seed_hosts = api_port = password = After changing properties in this file, restart OpsCenter for the changes to take effect. Mark On Sat, Apr 5, 2014 at 6:54 AM, Check Peck comptechge...@gmail.comwrote: Hi All, We would like to secure our Cassandra database. We don't want anybody to read/write on our Cassandra database leaving our team members only. We are using Cassandra 1.2.9 in Production and we have 36 node Cassandra cluster. 12 in each colo as we have three datacenters. But we would like to have OPSCENTER working as it is working currently. Is this possible to do anyhow? Is there any settings in yaml file which we can enforce?