Re: Cassandra event notification on INSERT/DELETE of records

[Mark Reddy]

+1 to what Eric said, a queue is a classic C* anti-pattern. Something like
Kafka or RabbitMQ might fit your use case better.


Mark

On 24 May 2016 at 18:03, Eric Stevens  wrote:

> It sounds like you're trying to build a queue in Cassandra, which is one
> of the classic anti-pattern use cases for Cassandra.
>
> You may be able to do something clever with triggers, but I highly
> recommend you look at purpose-built queuing software such as Kafka to solve
> this instead.
>
> On Tue, May 24, 2016 at 9:49 AM Aaditya Vadnere  wrote:
>
>> Hi experts,
>>
>> We are evaluating Cassandra as messaging infrastructure for a project.
>>
>> In our workflow Cassandra database will be synchronized across two nodes,
>> a component will INSERT/UPDATE records on one node and another component
>> (who has registered for the specific table) on second node will get
>> notified of record change.
>>
>> The second component will then try to read the database to find out the
>> specific message.
>>
>> Is it possible for Cassandra to support such workflow? Basically, is
>> there a way for Cassandra to generate a notification anytime schema changes
>> (so we can set processes to listen for schema changes). As I understand,
>> polling the database periodically or database triggers might work but they
>> are costly operations.
>>
>>
>> --
>> Aaditya Vadnere
>>
>

Re: Record deletion job

[Mark Reddy]

No, this is what TTL is for.

If you want to schedule a cron to delete data this will need to be an
external task.


On 19 June 2015 at 07:41, anil_ah anil...@yahoo.co.in wrote:

 Hi
Is their any way to schedule a job in cassandra to delete the recrods
 which are older than a specific time period.

 Excluding the option of TTL.

 Regards
 Anil


 Sent from Samsung Mobile

Re: connection refused error for cassandra-cli

[Mark Reddy]

Have you configured the rpc_address to listen on that ip? You can check
your nodes casandra.yaml for the rpc_address setting.

On a side node why are you testing on such an old version of Cassandra? The
latest versions of 1.2.x is 1.2.19. There are also two branches of 2.x
which are 2.0.15 and 2.1.5.


Mark

On 26 May 2015 at 10:38, Mahmood Naderan nt_mahm...@yahoo.com wrote:

 Hello,
 I have downloaded cassandra-1.2.0 for a test. The output of
 ./bin/cassandra and ./bin/cassandra-cli seems to be fine. Please see
 http://pastebin.com/Tgs7qyjS

 Next, I created a setup-ycsb.cql and wrote
 [mahmood@tiger ycsb-0.1.4]$ cat setup-ycsb.cql
 CREATE KEYSPACE usertable
 with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
 and strategy_options = {replication_factor:2};
 use usertable;
 create column family data with comparator=UTF8Type and
 default_validation_class=UTF8Type and key_validation_class=UTF8Type;


 Problem is, when I execute cassandra-cli commnad, I get connection refused
 error

 [mahmood@tiger ycsb-0.1.4]$ cassandra-cli -h 192.168.1.5 -f setup-ycsb.cql
 org.apache.thrift.transport.TTransportException:
 java.net.ConnectException: Connection refused




 Everything seems to be normal as I disabled the firewall. So I wonder why
 I receive that error.
 Please see this output for more detail  http://pastebin.com/Bnd3TNCM


 Did I miss anything?

 Regards,
 Mahmood

Re: java.io.IOException: Failed during snapshot creation

[Mark Reddy]

Can you check your logs for any other other error message around the time
of the repair? Something to look for would be Error occurred during
snapshot phase.


Regards,
Mark

On 25 May 2015 at 14:56, Sachin PK sachinpray...@gmail.com wrote:

 Hey I'm new to Cassandra ,I have 4 node cluster with each node 16GB VPS,
 initially I had one seed node, I added one of the existing nodes as seed
 nodes , restarted nodes one by one after that one of my node went down . I
 ran nodetool repair on it when I checked the log i could find some errors


 ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,905
 RepairSession.java:303 - [repair #968bce30-02e1-11e5-9310-45191f4c93ae]
 session completed with the following error
 java.io.IOException: Failed during snapshot creation.
 at
 org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
 ~[apache-cassandra-2.1.5.jar:2.1.5]
 at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146)
 ~[apache-cassandra-2.1.5.jar:2.1.5]
 at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
 ~[guava-16.0.jar:na]
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 [na:1.7.0_80]
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 [na:1.7.0_80]
 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
 ERROR [AntiEntropySessions:3] 2015-05-25 09:26:12,907
 CassandraDaemon.java:223 - Exception in thread
 Thread[AntiEntropySessions:3,5,RMI Runtime]
 java.lang.RuntimeException: java.io.IOException: Failed during snapshot
 creation.
 at com.google.common.base.Throwables.propagate(Throwables.java:160)
 ~[guava-16.0.jar:na]
 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
 ~[apache-cassandra-2.1.5.jar:2.1.5]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 ~[na:1.7.0_80]
 at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 ~[na:1.7.0_80]
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 [na:1.7.0_80]
 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
 Caused by: java.io.IOException: Failed during snapshot creation.
 at
 org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
 ~[apache-cassandra-2.1.5.jar:2.1.5]
 at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146)
 ~[apache-cassandra-2.1.5.jar:2.1.5]
 at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
 ~[guava-16.0.jar:na]
 ... 3 common frames omitted

Re: Disabling auto snapshots

[Mark Reddy]

To disable auto snapshots, set the property auto_snapshot: false in your
cassandra.yaml file.

Mark

On 21 May 2015 at 08:30, Ali Akhtar ali.rac...@gmail.com wrote:

 Is there a config setting where automatic snapshots can be disabled? I
 have a use case where a table is truncated quite often, and would like to
 not have snapshots. I can't find anything on google.

 Thanks.

Re: Drop/Create table with same CF Name

[Mark Reddy]

Yes, it's a known issue. For more information on the topic see this support
post from DataStax:

https://support.datastax.com/hc/en-us/articles/204226339-How-to-drop-and-recreate-a-table-in-Cassandra-versions-older-than-2-1

Mark

On 21 May 2015 at 15:31, Ken Hancock ken.hanc...@schange.com wrote:


 We've been running into the reused key cache issue (CASSANDRA-5202) with
 dropping and recreating the same table in Cassandra 1.2.18 so we've been
 testing with key caches disabled which does not seem to solve the issue.
 In the latest logs it seems that old SSTables metadata gets read after the
 tables have been deleted by the previous drop, eventually causing an
 exception and the Thrift interface shut down.

 At this point is it a known issue that one CANNOT reuse a table name prior
 to Cassandra 2.1 ?

Re: Steps to do after schema changes

[Mark Reddy]

It's always good to run nodetool describecluster after a schema change,
this will show you all the nodes in your cluster and what schema version
they have. If they have different versions you have a schema disagreement
and should follow this guide to resolution:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_handle_schema_disagree_t.html

Regards,
Mark

On 12 March 2015 at 05:47, Phil Yang ud1...@gmail.com wrote:

 Usually, you have nothing to do. Changes will be synced to every nodes
 automatically.

 2015-03-12 13:21 GMT+08:00 Ajay ajay.ga...@gmail.com:

 Hi,

 Are there any steps to do (like nodetool or restart node) or any
 precautions after schema changes are done in a column family say adding a
 new column or modifying any table properties?

 Thanks
 Ajay




 --
 Thanks,
 Phil Yang

Re: unsubscribe

[Mark Reddy]

Raman,

To unsubscribe send a mail to user-unsubscr...@cassandra.apache.org


On 22 February 2015 at 03:59, Raman ra...@assetcorporation.net wrote:

 unsubscribe

Re: Why no virtual nodes for Cassandra on EC2?

[Mark Reddy]

Hey Clint,

Someone for DataStax can correct me here, but I'm assuming that they have
disabled vnodes because the AMI is built to make it easy to set up a
pre-configured
mixed workload cluster. A mixture of Real-Time/Transactional (Cassandra),
Analytics (Hadoop), or Search (Solr). If you take a look at the getting
started guide for both Hadoop and Solr you will see a paragraph instructing
the user to disable vnodes for a mix workload cluster.

http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax
_enterprise/srch/srchIntro.html
http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax
_enterprise/ana/anaStrt.html

This is specific to the example AMI and that type of workload. This is by
no means a warning for users to disable vnodes on their
Real-Time/Transactional Cassandra only clusters on EC2.


I've used vnodes on EC2 without issue.

Regards,
Mark

On 20 February 2015 at 05:08, Clint Kelly clint.ke...@gmail.com wrote:

 Hi all,

 The guide for installing Cassandra on EC2 says that

 Note: The DataStax AMI does not install DataStax Enterprise nodes
 with virtual nodes enabled.


 http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html

 Just curious why this is the case.  It was my understanding that
 virtual nodes make taking Cassandra nodes on and offline an easier
 process, and that seems like something that an EC2 user would want to
 do quite frequently.

 -Clint

Re: Cancel subscription

[Mark Reddy]

Please use user-unsubscr...@cassandra.apache.org to unsubscribe from this
mailing list.


Thanks

Regards,
Mark

On 19 February 2015 at 09:14, Hilary Albutt - CEO 
hil...@incrediblesoftwaresolutions.com wrote:

 Cancel subscription

Re: Node joining take a long time

[Mark Reddy]

What is a long time in your scenario? What is the data size in your cluster?


I'm sure Rob will be along shortly to say that 2.1.2 is, in his opinion,
broken for production use...an opinion I'd agree with. So bare that in mind
if you are running a production cluster.

Regards,
Mark

On 19 February 2015 at 12:19, 曹志富 cao.zh...@gmail.com wrote:

 Hi guys:
 I have a 20 nodes C* cluster with vnodes,version is 2.1.2. When I add a
 node to my cluster,it take a long time ,and somes exists node nodetool
 nestats show this:

 Mode: NORMAL
 Unbootstrap cfe03590-b02a-11e4-95c5-b5f6ad9c7711
 /172.19.105.49
 Receiving 68 files, 23309801005 bytes total

 I want know ,is there some problem with my cluster?
 --
 曹志富
 手机：18611121927
 邮箱：caozf.zh...@gmail.com
 微博：http://weibo.com/boliza/

Re: Cassandra install on JRE vs JDK

[Mark Reddy]

Yes you can use Oracle JDK if your prefer, I've been using the JDK with
Cassandra in production for years without issue.

Regards,
Mark

On 18 February 2015 at 19:49, cass savy casss...@gmail.com wrote:

 Can we install Oracle JDK instead of JRE  in Cassandra servers? We have
 few clusters running JDK when we upgraded to C*2.0.

 Is there any known issue or impact with using  JDK vs JRE?
 What is the reason to not use Oracle JDK in C* servers?
 Is there any performance impact ?

 Please advice.

Re: High GC activity on node with 4TB on data

[Mark Reddy]

Hey Jiri,

While I don't have any experience running 4TB nodes (yet), I would
recommend taking a look at a presentation by Arron Morton on large nodes:
http://planetcassandra.org/blog/cassandra-community-webinar-videoslides-large-nodes-with-cassandra-by-aaron-morton/
to see if you can glean anything from that.

I would note that at the start of his talk he mentions that in version 1.2
we can now talk about nodes around 1 - 3 TB in size, so if you are storing
anything more than that you are getting into very specialised use cases.

If you could provide us with some more information about your cluster setup
(No. of CFs, read/write patterns, do you delete / update often, etc.) that
may help in getting you to a better place.


Regards,
Mark

On 8 February 2015 at 21:10, Kevin Burton bur...@spinn3r.com wrote:

 Do you have a lot of individual tables?  Or lots of small compactions?

 I think the general consensus is that (at least for Cassandra), 8GB heaps
 are ideal.

 If you have lots of small tables it’s a known anti-pattern (I believe)
 because the Cassandra internals could do a better job on handling the in
 memory metadata representation.

 I think this has been improved in 2.0 and 2.1 though so the fact that
 you’re on 1.2.18 could exasperate the issue.  You might want to consider an
 upgrade (though that has its own issues as well).

 On Sun, Feb 8, 2015 at 12:44 PM, Jiri Horky ho...@avast.com wrote:

 Hi all,

 we are seeing quite high GC pressure (in old space by CMS GC Algorithm)
 on a node with 4TB of data. It runs C* 1.2.18 with 12G of heap memory
 (2G for new space). The node runs fine for couple of days when the GC
 activity starts to raise and reaches about 15% of the C* activity which
 causes dropped messages and other problems.

 Taking a look at heap dump, there is about 8G used by SSTableReader
 classes in org.apache.cassandra.io.compress.CompressedRandomAccessReader.

 Is this something expected and we have just reached the limit of how
 many data a single Cassandra instance can handle or it is possible to
 tune it better?

 Regards
 Jiri Horky




 --

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog: http://burtonator.wordpress.com
 … or check out my Google+ profile
 https://plus.google.com/102718274791889610666/posts
 http://spinn3r.com

Re: Compacted_at timestamp

[Mark Reddy]

Hi Santo,

If you are seeing the compacted_at value as timestamp and want to convert
it to a human readable date, this is not possible via nodetool. You will
always write a script to make the compactionhistory call and then convert
the output (the fourth column - compacted_at) to a readable date.

If you are seeing something other than an expected timestamp value, can you
post an example of what you are getting?


Regards,
Mark

On 8 February 2015 at 13:20, Havere Shanmukhappa, Santhosh 
santhosh_havereshanmukha...@intuit.com wrote:

  When I run nodetool compactionhistory command, it displays
 ‘compacted_at’ timestamp in non-readable format. Any way to read that
 column in readable format? I am using c*2.0.11 version.

  Thanks,
 Santo

Re: Error while starting Cassandra for the first time

[Mark Reddy]

Hey Krish,

If you want to use these config values rather than comment them out, feel
free to post a sample of the config that is giving you trouble, I'd be
happy to take a look at it and see if I could see where your issue is.

The fact that all the config options are directories may be an indication
that there are some values that need to be escaped or the paths need to be
quoted.

Regards,
Mark

On 5 February 2015 at 04:29, Krish Donald gotomyp...@gmail.com wrote:

 I have used the yaml validator but tried to fixed based on error messages
 , I had to comment data_directories , commitlog_directory and saved_caches
 directory and after that it worked.

 Thanks a lot for the help ...

 On Wed, Feb 4, 2015 at 2:32 PM, Mark Reddy mark.l.re...@gmail.com wrote:

 INFO  22:17:19 Loading settings from file:/home/csduser/cassandra/
 conf/cassandra.yaml
 ERROR 22:17:20 Fatal configuration error
 org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml


 You have an malformed cassandra.yaml config file that is resulting in
 Cassandra not being able to start. You should be able to validate your yaml
 file with some online validator such as http://www.yamllint.com/


 Regards,
 Mark

 On 4 February 2015 at 22:23, Krish Donald gotomyp...@gmail.com wrote:

 Hi,

 I am getting below error:
 Not able to understand why ??

 [csduser@master bin]$ ./cassandra -f
 CompilerOracle: inline
 org/apache/cassandra/db/AbstractNativeCell.compareTo
 (Lorg/apache/cassandra/db/composites/Composite;)I
 CompilerOracle: inline
 org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned
 (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I
 CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
 (Ljava/nio/ByteBuffer;[B)I
 CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
 ([BLjava/nio/ByteBuffer;)I
 CompilerOracle: inline
 org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned
 (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
 CompilerOracle: inline
 org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
 (Ljava/lang/Object;JILjava/lang/Object;JI)I
 CompilerOracle: inline
 org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
 (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
 CompilerOracle: inline
 org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
 (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
 INFO  22:17:19 Hostname: master.my.com
 INFO  22:17:19 Loading settings from
 file:/home/csduser/cassandra/conf/cassandra.yaml
 ERROR 22:17:20 Fatal configuration error
 org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml
 at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:120)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:158)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:133)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
 [apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
 [apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
 [apache-cassandra-2.1.2.jar:2.1.2]
 Caused by: org.yaml.snakeyaml.scanner.ScannerException: while scanning a
 simple key; could not found expected ':';  in 'reader', line 33, column 1:
 # See http://wiki.apache.org/cas ...
 ^
 at
 org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:460)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:558)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105)
 ~[snakeyaml-1.11.jar:na

Re: Case Study for Learning Cassandra

[Mark Reddy]

Hey Krish,

Here is a list of use cases from the industry:
http://planetcassandra.org/apache-cassandra-use-cases/ You may find some
examples on how people/companies are using and running their C* clusters.

Also DataStax provide online training that will get you up to speed on
several aspects of C* usage: https://academy.datastax.com/


Regards,
Mark

On 5 February 2015 at 04:38, Krish Donald gotomyp...@gmail.com wrote:

 Hi,

 I am new to Cassandra and have setup 4 nodes Cassandra cluster using VMs.
 Looking for any case study which I can do to understand the Cassandra
 Administration and put in my resume as well.

 Any help is appreciated.

 Thanks
 Krish

Re: Newly added column not visible

[Mark Reddy]

Hey Saurabh,

I can't say that I have experienced this before, however if you can
reliably reproduce the issue it would be worth commenting on the JIRA issue
you linked to or alternatively creating a new JIRA with as much info
(setup, test case, debug logs, etc) as possible.

Regards,
Mark

On 5 February 2015 at 00:50, Saurabh Sethi saurabh_se...@symantec.com
wrote:

 I have a 3 node cluster running Cassandra version 2.1.2. Through my unit
 test, I am creating a column family with 3 columns, inserting a row,
 asserting that the values got inserted and then truncating the column
 family.

 After that I am adding a fourth column to the column family and inserting
 a new row with four values but this insertion is failing because it can’t
 find the newly added column.

 If I go via debug mode, it works fine but not otherwise. I also tried
 putting a Thread.sleep() for 10 seconds after adding the new column but to
 no avail.

 Anyone has any idea what might be going on here?

 I see that a jira was filed related to similar issue in May 2014 but they
 closed it in December 2014 stating not reproducible.
 https://issues.apache.org/jira/browse/CASSANDRA-7186

 Thanks,
 Saurabh

Re: Error while starting Cassandra for the first time

[Mark Reddy]

 INFO  22:17:19 Loading settings from file:/home/csduser/cassandra/
 conf/cassandra.yaml
 ERROR 22:17:20 Fatal configuration error
 org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml


You have an malformed cassandra.yaml config file that is resulting in
Cassandra not being able to start. You should be able to validate your yaml
file with some online validator such as http://www.yamllint.com/


Regards,
Mark

On 4 February 2015 at 22:23, Krish Donald gotomyp...@gmail.com wrote:

 Hi,

 I am getting below error:
 Not able to understand why ??

 [csduser@master bin]$ ./cassandra -f
 CompilerOracle: inline
 org/apache/cassandra/db/AbstractNativeCell.compareTo
 (Lorg/apache/cassandra/db/composites/Composite;)I
 CompilerOracle: inline
 org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned
 (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I
 CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
 (Ljava/nio/ByteBuffer;[B)I
 CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
 ([BLjava/nio/ByteBuffer;)I
 CompilerOracle: inline
 org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned
 (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
 CompilerOracle: inline
 org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
 (Ljava/lang/Object;JILjava/lang/Object;JI)I
 CompilerOracle: inline
 org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
 (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
 CompilerOracle: inline
 org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
 (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
 INFO  22:17:19 Hostname: master.my.com
 INFO  22:17:19 Loading settings from
 file:/home/csduser/cassandra/conf/cassandra.yaml
 ERROR 22:17:20 Fatal configuration error
 org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml
 at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:120)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:158)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:133)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
 [apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
 [apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
 [apache-cassandra-2.1.2.jar:2.1.2]
 Caused by: org.yaml.snakeyaml.scanner.ScannerException: while scanning a
 simple key; could not found expected ':';  in 'reader', line 33, column 1:
 # See http://wiki.apache.org/cas ...
 ^
 at
 org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:460)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:558)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105)
 ~[snakeyaml-1.11.jar:na]
 at
 org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)
 ~[snakeyaml-1.11.jar:na]
 at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
 ~[snakeyaml-1.11.jar:na]
 at org.yaml.snakeyaml.Yaml.load(Yaml.java:412)
 ~[snakeyaml-1.11.jar:na]
 at
 org.apache.cassandra.config.YamlConfigurationLoader.logConfig(YamlConfigurationLoader.java:126)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:104)
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 ... 6 common frames omitted
 Invalid yaml
 Fatal configuration error; unable to start. See log for stacktrace.


 Thanks
 Krish

Re: Question about use scenario with fulltext search

[Mark Reddy]

 If you have a list of usernames stored in your cassandra database,
 how could you find all usernames starting with 'Jo'?


Cassandra does not support full text search on its own, if you are looking
into DataStax enterprise Cassandra there is an integration with Slor that
gives you this functionality.

Personally for projects I work on that use Cassandra and require full text
search, the necessary data is indexed into Elasticsearch.

Or ... if this is not possible,
 what are you using cassandra for?


If you are looking for use cases here is a comprehensive set from companies
spanning many industries:
http://planetcassandra.org/apache-cassandra-use-cases/


Regards,
Mark

On 1 February 2015 at 16:05, anton anto...@gmx.de wrote:

 Hi,

 I was just reading about cassandra and playing a little
 with it (using django www.djangoproject.com on the web server).

 One thing that I realized now is that fulltext search
 as in a normal sql statement (example):

   select name from users where name like 'Jo%';

 Simply does not work because this functionality does not exist.
 After reading and googeling and reading ...
 I still do not understand how I could use a db without this
 functionality (If I do not want to restrict myself on numerical data).

 So my question is:

 If you have a list of usernames stored in your cassandra database,
 how could you find all usernames starting with 'Jo'?


 Or ... if this is not possible,
 what are you using cassandra for?

 Actually I still did not get the point of how I could use cassandra :-(

 Anton

Re: A questiion to adding a new data center

[Mark Reddy]

Hi Boying,

I'm not sure I fully understand your question here, so some clarification
may be needed. However, if you are asking what steps need to be performed
on the current datacenter or on the new datacenter:

Step 1 - Current DC
Step 2 - New DC
Step 3 - Depending on the snitch you may need to make changes on both the
current and new DCs
Step 4 - Client config
Step 5 - Client config
Step 6 - New DC
Step 7 - New DC
Step 8 - New DC


Mark

On 21 November 2014 03:27, Lu, Boying boying...@emc.com wrote:

 Hi, all,



 I read the document about how to adding a new data center to existing
 clusters posted at
 http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

 But I have a question: Are all those steps executed *only* at the new
 adding cluster or on  existing clusters also? ( Step 7 is to be executed on
 the new cluster according to the document).



 Thanks



 Boying

Re: [jira] Akhtar Hussain shared a search result with you

[Mark Reddy]

I believe you were attempting to share:
https://issues.apache.org/jira/browse/CASSANDRA-8352

Your cassandra logs outputs the following:

 DEBUG [Thrift:4] 2014-11-20 15:36:50,653 CustomTThreadPoolServer.java
 (line 204) Thrift transport error occurred during processing of message.
 org.apache.thrift.transport.TTransportException: Cannot read. Remote side
 has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often
 indicative of an internal error on the server side. Please check your
 server logs.)


Which indicates that your server is under pressure at that moment and
points you to look at your server logs for further diagnosis.


Mark

On 21 November 2014 11:15, Akhtar Hussain (JIRA) j...@apache.org wrote:

 Akhtar Hussain shared a search result with you
 -


 https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=truejqlQuery=reporter+%3D+currentUser%28%29+ORDER+BY+createdDate+DESC

 We have a Geo-red setup with 2 Data centers having 3 nodes each. When
 we bring down a single Cassandra node down in DC2 by kill -9
 Cassandra-pid, reads fail on DC1 with TimedOutException for a brief
 amount of time (15-20 sec~).




 --
 This message was sent by Atlassian JIRA
 (v6.3.4#6332)

Re: max ttl for column

[Mark Reddy]

Hi Rajanish,

Cassandra imposes a max TTL of 20 years.

public static final int MAX_TTL = 20 * 365 * 24 * 60 * 60; // 20 years in
 seconds


See:
https://github.com/apache/cassandra/blob/8d8fed52242c34b477d0384ba1d1ce3978efbbe8/src/java/org/apache/cassandra/db/ExpiringCell.java#L37



Mark

On 21 November 2014 17:29, Rajanish GJ rajan...@apigee.com wrote:


 Does hector or cassandra imposes a limit on max ttl value for a column?

 I am trying to insert record into one of the column family and seeing the
 following error..
 Cassandra version : 1.1.12
 Hector  : 1.1-4

 Any pointers appreciated.

 me.prettyprint.hector.api.exceptions.HInvalidRequestException:
 InvalidRequestException(why:ttl is too large. requested (951027277) maximum
 (63072))
 at
 me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:52)
 ~[hector-core-1.1-4.jar:na]
 at
 me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:260)
 ~[hector-core-1.1-4.jar:na]
 at
 me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113)
 ~[hector-core-1.1-4.jar:na]
 at
 me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
 ~[hector-core-1.1-4.jar:na]
 at
 me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115)
 ~[hector-core-1.1-4.jar:na]
 at
 me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163)
 ~[hector-core-1.1-4.jar:na]
 at
 me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69)
 ~[hector-core-1.1-4.jar:na]

 =
 *Also tried using cql, and it seems to hangs and not responding.. trying
 again with few combinations*

 *INSERT INTO users (key,id) values ('test6','951027277 secs') USING
 TTL 951027277 ; *



 Regards
 Rajanish GJ
 apigee | rajan...@apigee.com

Re: logging over multi-datacenter

[Mark Reddy]

Hi Adil,

What Cassandra version are you using? Are you using the default user or a
non-default user?


Mark

On 20 November 2014 08:20, Adil adil.cha...@gmail.com wrote:

 Hi,
 We have two data-center, we configured PasswordAuthenticator on each node,
 we increment the RF of system_auth to the number of nodes (each
 data-center) as recommended.
 We can logged-in via cqlsh without problem, but when i stop cassandra on
 all nodes of a data-center we can't logged in in the other
 data-center...this error is displayed as output:
 Bad credentials]
 message=org.apache.cassandra.exceptions.UnavailableException: Cannot
 achieve consistency level QUORUM'

 from what i understand we should be able to logged in even if there is
 only one node UP but it seems that has to reach QUORUM consistency level (2
 data-center).

 my question is if the java driver cql uses the same condition and if there
 is a way to set the consistency level to like LOCAL_ONE.

 Thanks
 Adil

Re: logging over multi-datacenter

[Mark Reddy]

Hi Adil,

When using the default superuser ('cassandra') a consistency level of
QUORUM is used. When using other users ONE is used.

You are not supposed to use 'cassandra' user directly, except to create
another superuser and use that one from that point on.


Mark

On 20 November 2014 14:40, Adil adil.cha...@gmail.com wrote:

 cassandra version 2.1.2
 with the default user
 we create another user and with this one we could login even if only one
 node is up

 2014-11-20 15:16 GMT+01:00 Mark Reddy mark.l.re...@gmail.com:

 Hi Adil,

 What Cassandra version are you using? Are you using the default user or a
 non-default user?


 Mark

 On 20 November 2014 08:20, Adil adil.cha...@gmail.com wrote:

 Hi,
 We have two data-center, we configured PasswordAuthenticator on each
 node, we increment the RF of system_auth to the number of nodes (each
 data-center) as recommended.
 We can logged-in via cqlsh without problem, but when i stop cassandra on
 all nodes of a data-center we can't logged in in the other
 data-center...this error is displayed as output:
 Bad credentials]
 message=org.apache.cassandra.exceptions.UnavailableException: Cannot
 achieve consistency level QUORUM'

 from what i understand we should be able to logged in even if there is
 only one node UP but it seems that has to reach QUORUM consistency level (2
 data-center).

 my question is if the java driver cql uses the same condition and if
 there is a way to set the consistency level to like LOCAL_ONE.

 Thanks
 Adil

Re: any way to get nodetool proxyhistograms data for an entire cluster?

[Mark Reddy]

 To be clear, I don't believe DSE is required to use OpsCenter. My
 understanding is that it can be used with vanilla Apache Cassandra, but I
 have never actually tried to do so.


Correct Rob, you can use Opscenter with vanilla Apache Cassandra.


Mark

On 20 November 2014 16:10, Robert Coli rc...@eventbrite.com wrote:

 On Wed, Nov 19, 2014 at 7:20 PM, Clint Kelly clint.ke...@gmail.com
 wrote:

 We have DSE so I can use opscenter.  I was just looking for something
 more precise than the graphs that I get from opscenter.


 To be clear, I don't believe DSE is required to use OpsCenter. My
 understanding is that it can be used with vanilla Apache Cassandra, but I
 have never actually tried to do so.

 =Rob

Re: What time range does nodetool cfhistograms use?

[Mark Reddy]

Hi Clint,

The values of SSTables, Write Latency and Read Latency will be reset on
node start/restart and after running the cfhistograms command itself.

The values of Row Size and Column Count are calculated at startup and
then re-evaluated during compaction.


Mark


On 16 November 2014 17:12, Clint Kelly clint.ke...@gmail.com wrote:

 Hi all,

 Over what time range does nodetool cfhistograms operate?

 I am using Cassandra 2.0.8.39.

 I am trying to debug some very high 95th and 99th percentile read
 latencies in an application that I'm working on.

 I tried running nodetool cfhistograms to get a flavor for the
 distribution of read latencies and also to see how many SSTables our
 reads are using, and I saw different results yesterday versus this
 morning, so I assume the time window is fairly tight (like an hour or
 so).   I vaguely recall Aaron Morton talking about this during the
 training course I took at Cassandra Summit, but I cannot find my notes
 and I think the behavior of the tool with regard to time windows
 changed from version to version.

 Thanks!

 -Clint

Re: Netstats 100% streaming

[Mark Reddy]

HI Eric,

It looks like you are running into CASSANDRA-7878
https://issues.apache.org/jira/browse/CASSANDRA-7878 which is fixed
in 2.0.11 / 2.1.1


Mark

On 1 November 2014 14:08, Eric Stevens migh...@gmail.com wrote:

 We've been commissioning some new nodes on a 2.0.10 community edition
 cluster, and we're seeing streams that look like they're shipping way more
 data than they ought for individual files during bootstrap.


 /var/lib/cassandra/data/x/y/x-y-jb-11748-Data.db
 3756423/3715409 bytes(101%) sent to /1.2.3.4

 /var/lib/cassandra/data/x/y/x-y-jb-11043-Data.db
 584745/570432 bytes(102%) sent to /1.2.3.4
 /var/lib/cassandra/data/x/z/x-z-jb-525-Data.db
 13020828/11141590 bytes(116%) sent to /1.2.3.4
 /var/lib/cassandra/data/x/w/x-w-jb-539-Data.db
 1044124/51404 bytes(2031%) sent to /1.2.3.4
 /var/lib/cassandra/data/x/v/x-v-jb-546-Data.db
 971447/22253 bytes(4365%) sent to /1.2.3.4

 /var/lib/cassandra/data/x/y/x-y-jb-10404-Data.db
 6225920/23215181 bytes(26%) sent to /1.2.3.4

 Has anyone else seen something like this, and this something we should be
 worried about?  I haven't been able to find any information about this
 symptom.

 -Eric

Re: EC2 Snitch load imbalance

[Mark Reddy]

Oleg,

If you are running nodetool status, be sure to specify the keyspace also.
If you don't specify the keyspace the results will be nonsense.

https://issues.apache.org/jira/browse/CASSANDRA-7173


Regards,
Mark

On 28 October 2014 10:35, Oleg Dulin oleg.du...@gmail.com wrote:

 I have a setup with 6 cassandra nodes (1.2.18), using RandomPartition, not
 using vnodes -- this is a legacy cluster.

 We went from 3 nodes to 6 in the last few days to add capacity. However,
 there appears to be an imbalance:

 Datacenter: us-east
 ==
 Replicas: 2

 Address RackStatus State   LoadOwns
Token

113427455640312821154458202477256070484
 x.x.x.73   1d  Up Normal  154.64 GB   33.33%
  85070591730234615865843651857942052863
 x.x.x.2511a  Up Normal  62.26 GB16.67%
  28356863910078205288614550619314017621
 x.x.x.238   1b  Up Normal  243.7 GB50.00%
  56713727820156410577229101238628035242
 x.x.x.25   1a  Up Normal  169.3 GB33.33%
 210
 x.x.x.162  1b  Up Normal  118.24 GB   50.00%
  141784319550391026443072753096570088105
 x.x.x.208   1d  Up Normal  226.85 GB   16.67%
  113427455640312821154458202477256070484


 What is the cause of this imbalance ? How can I rectify it ?

 Regards,
 Oleg

Re: Can't Add AWS Node due to /mnt/cassandra/data directory

[Mark Reddy]

Hi stephen,

I have never added a node via OpsCenter, so this may be a short coming of
that process. However in non OpsCenter installs you would have to create
the data directories first:

sudo mkdir -p /mnt/cassandra/commitlog
sudo mkdir -p /mnt/cassandra/data
sudo mkdir -p /mnt/cassandra/saved_caches

And then give the cassandra user ownership of those directories:

sudo chown -R cassandra:cassandra /mnt/cassandra

Once this is done Cassandra will have the correct directories and
permission to start up.


Mark


On 27 August 2014 09:50, Stephen Portanova sport...@gmail.com wrote:

 I already have a 3node m3.large DSE cluster, but I can't seem to add
 another m3.large node. I'm using the 
 ubuntu-trusty-14.04-amd64-server-20140607.1
 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7
 and the JNA, then I go into opscenter to add a node. Things look good for 3
 or 4 green circles, until I either get this error: Start Errored: Timed
 out waiting for Cassandra to start. or this error: Agent Connection
 Errored: Timed out waiting for agent to connect.

 I check the system.log and output.log, and they both say:
 INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall
 successful
 ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) 
 *Directory
 /mnt/cassandra/data doesn't exist*
 *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has
 no permission to create /mnt/cassandra/data directory*
  INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE
 shutting down...
 ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199)
 Exception in thread Thread[Thread-1,5,main]
 java.lang.AssertionError
 at
 org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263)
 at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171)
 at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478)
 at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)

 My agent.log file says:

 Node is still provisioning, not attempting to determine ip.

  INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before
 trying to determine IP over JMX again

  INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning,
 not attempting to determine ip.

  INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before
 trying to determine IP over JMX again

  INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning,
 not attempting to determine ip.

  INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before
 trying to determine IP over JMX again

  INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning,
 not attempting to determine ip.


 I feel like I'm missing something easy with the mount, so if you could
 point me in the right direction, I would really appreciate it!

 --
 Stephen Portanova
 (480) 495-2634

Re: Installing Cassandra Multinode on CentOs coming up with exception

[Mark Reddy]

It is telling you that your yaml is invalid, from looking at the snippet
you have provided it looks like the seed_provider.parameters is not
correctly indented, it should look something like:

seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
 - seeds: 192.168.1.32


Regards,
Mark


On 26 August 2014 08:12, Vineet Mishra clearmido...@gmail.com wrote:


 Hi All,

 I am installing Cassandra Multinode Setup on a 4 node CentOs Cluster, my
 cassandra.yaml looks like so

 cluster_name: 'node'
 initial_token: 0
 num_tokens: 256
 seed_provider:
 - class_name: org.apache.cassandra.locator.SimpleSeedProvider
 parameters:
 - seeds: 192.168.1.32
 listen_address: 192.168.1.32
 rpc_address: 0.0.0.0
 endpoint_snitch: RackInferringSnitch

 Similarly cassandra.yaml for second node

 cluster_name: 'node'
 initial_token: 2305843009213693952
 num_tokens: 256
 seed_provider:
 - class_name: org.apache.cassandra.locator.SimpleSeedProvider
 parameters:
 - seeds: 192.168.1.32
 listen_address: 192.168.1.36
 rpc_address: 0.0.0.0
 endpoint_snitch: RackInferringSnitch

 and so on. . .

 While trying to start the Server running on the Seed node(192.168.1.32) it
 throws this nasty exception and didn't start,


 -bash-4.1$ sudo bin/cassandra

 -bash-4.1$  INFO 12:19:46,653 Logging initialized
  INFO 12:19:46,688 Loading settings from
 file:/home/cluster/cassandra/conf/cassandra.yaml
 ERROR 12:19:46,985 Fatal configuration error
 org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml
  at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:100)
 at
 org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:135)
  at
 org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:111)
 at
 org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:156)
  at
 org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
 at
 org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
 Caused by: Can't construct a java object for 
 tag:yaml.org,2002:org.apache.cassandra.config.Config;
 exception=Cannot create property=seed_provider for
 JavaBean=org.apache.cassandra.config.Config@676c6370;
 java.lang.reflect.InvocationTargetException
  in 'reader', line 1, column 1:
 cluster_name: 'pcross'
 ^

 at
 org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:333)
  at
 org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182)
 at
 org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141)
  at
 org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127)
 at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
  at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:475)
 at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:93)
  ... 5 more
 Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create
 property=seed_provider for
 JavaBean=org.apache.cassandra.config.Config@676c6370;
 java.lang.reflect.InvocationTargetException
  at
 org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:299)
 at
 org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:189)
  at
 org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:331)
 ... 11 more
 Caused by: org.yaml.snakeyaml.error.YAMLException:
 java.lang.reflect.InvocationTargetException
 at
 org.yaml.snakeyaml.constructor.Constructor$ConstructSequence.construct(Constructor.java:542)
  at
 org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182)
 at
 org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:296)
  ... 13 more
 Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at
 org.yaml.snakeyaml.constructor.Constructor$ConstructSequence.construct(Constructor.java:540)
  ... 15 more
 Caused by: java.lang.NullPointerException
 at
 org.apache.cassandra.config.SeedProviderDef.init(SeedProviderDef.java:33)
  ... 20 more
 Invalid yaml


 I am not sure exactly whats making it to throw NullPointer and halt the
 process.

 Expert Advice would be appreciated!
 URGENT!

 Thanks!

Re: Installing Cassandra Multinode on CentOs coming up with exception

[Mark Reddy]

You are missing commitlog_sync in your cassandra.yaml.

Are you generating your own cassandra.yaml or editing the package default?
If you are generating your own there are several configuration options that
are required and if not present, Cassandra will fail to start.


Regards,
Mark


On 26 August 2014 11:14, Vineet Mishra clearmido...@gmail.com wrote:

 Thanks Mark,
 That was indeed yaml formatting issue.
 Moreover I am getting the underlying error now,

 INFO 15:33:43,770 Loading settings from
 file:/home/cluster/cassandra/conf/cassandra.yaml
  INFO 15:33:44,100 Data files directories: [/var/lib/cassandra/data]
  INFO 15:33:44,101 Commit log directory: /var/lib/cassandra/commitlog
 ERROR 15:33:44,103 Fatal configuration error
 org.apache.cassandra.exceptions.ConfigurationException: Missing required
 directive CommitLogSync
  at
 org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:147)
 at
 org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:111)
  at
 org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:156)
 at
 org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
  at
 org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
 Missing required directive CommitLogSync
 Fatal configuration error; unable to start. See log for stacktrace.

 Do you have any idea about this.

 Thanks!


 On Tue, Aug 26, 2014 at 3:07 PM, Mark Reddy mark.l.re...@gmail.com
 wrote:

 It is telling you that your yaml is invalid, from looking at the snippet
 you have provided it looks like the seed_provider.parameters is not
 correctly indented, it should look something like:

 seed_provider:
   - class_name: org.apache.cassandra.locator.SimpleSeedProvider
 parameters:
  - seeds: 192.168.1.32


 Regards,
 Mark


 On 26 August 2014 08:12, Vineet Mishra clearmido...@gmail.com wrote:


 Hi All,

 I am installing Cassandra Multinode Setup on a 4 node CentOs Cluster, my
 cassandra.yaml looks like so

 cluster_name: 'node'
 initial_token: 0
 num_tokens: 256
 seed_provider:
 - class_name: org.apache.cassandra.locator.SimpleSeedProvider
 parameters:
 - seeds: 192.168.1.32
 listen_address: 192.168.1.32
 rpc_address: 0.0.0.0
 endpoint_snitch: RackInferringSnitch

 Similarly cassandra.yaml for second node

 cluster_name: 'node'
 initial_token: 2305843009213693952
 num_tokens: 256
 seed_provider:
 - class_name: org.apache.cassandra.locator.SimpleSeedProvider
 parameters:
 - seeds: 192.168.1.32
 listen_address: 192.168.1.36
 rpc_address: 0.0.0.0
 endpoint_snitch: RackInferringSnitch

 and so on. . .

 While trying to start the Server running on the Seed node(192.168.1.32)
 it throws this nasty exception and didn't start,


 -bash-4.1$ sudo bin/cassandra

 -bash-4.1$  INFO 12:19:46,653 Logging initialized
  INFO 12:19:46,688 Loading settings from
 file:/home/cluster/cassandra/conf/cassandra.yaml
 ERROR 12:19:46,985 Fatal configuration error
 org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml
  at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:100)
 at
 org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:135)
  at
 org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:111)
 at
 org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:156)
  at
 org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
 at
 org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
 Caused by: Can't construct a java object for 
 tag:yaml.org,2002:org.apache.cassandra.config.Config;
 exception=Cannot create property=seed_provider for
 JavaBean=org.apache.cassandra.config.Config@676c6370;
 java.lang.reflect.InvocationTargetException
  in 'reader', line 1, column 1:
 cluster_name: 'pcross'
 ^

 at
 org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:333)
  at
 org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182)
 at
 org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141)
  at
 org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127)
 at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
  at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:475)
 at
 org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:93)
  ... 5 more
 Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create
 property=seed_provider for
 JavaBean=org.apache.cassandra.config.Config@676c6370;
 java.lang.reflect.InvocationTargetException
  at
 org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:299)
 at
 org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:189)
  at
 org.yaml.snakeyaml.constructor.Constructor

Re: Secondary index or dedicated CF?

[Mark Reddy]

Hi,

As a general rule of thumb I would steer clear of secondary indexes, this is 
also the official stand that DataStax take (see p5 of their best practices doc: 
http://www.datastax.com/wp-content/uploads/2014/04/WP-DataStax-Enterprise-Best-Practices.pdf).

“It is best to avoid using Cassandra's built-in secondary indexes where 
possible. Instead, it is recommended to denormalize data and manually maintain 
a dynamic table as a form of an index instead of using a secondary index. If 
and when secondary indexes are to be used, they should be created only on 
columns containing low-cardinality data (for example: fields with less than 
1000 states).“

Mark

On 22 Aug 2014, at 15:58, DuyHai Doan doanduy...@gmail.com wrote:

 Hello Eric
 
 Under the hood what is the difference of the both solutions? 
 
  1. Cassandra secondary index: distributed index, supports better high volume 
 of data, the index itself is distributed so there is no bottleneck. The 
 tradeoff is that depending on the cardinality of data having the same 
 bucketname+tenantID the performance may drop sharply. Please read this: 
 http://www.datastax.com/documentation/cql/3.1/cql/ddl/ddl_when_use_index_c.html?scroll=concept_ds_sgh_yzz_zj__when-no-index.
  There are several restrictions to secondary index
 
 2. Manual index: easy to design, but potentially wide row and not well 
 balance if  data having the same bucketname+tenantID is very large. 
 Furthermore you need to manage index consistency manually so that it is 
 synced with source data updates. 
 
  The best thing to do is to benchmark both solutions and takes the approach 
 giving you the best results. Be careful with benchmarks, it should be 
 representative of the data pattern you likely have in production.
 
 
 On Fri, Aug 22, 2014 at 7:47 AM, Leleu Eric eric.le...@worldline.com wrote:
 Hi,
 
  
 
  
 
 I’m new with Cassandra and I wondering what is the best design for my case.
 
  
 
 I have a set of buckets that contain one or thousands of contents.
 
  
 
 Here is my Content CF :
 
  
 
 CREATE TABLE IF NOT EXISTS contents (tenantID varchar,
 
  key varchar,
 
 type varchar,
 
 bucket varchar,
 
 owner varchar,
 
 workspace varchar,
 
  public_read boolean  PRIMARY KEY ((key, tenantID), type, workspace));
 
  
 
  
 
 To retrieve all contents that belong to a bucket, I have created an index on 
 the bucket column.
 
  
 
 CREATE INDEX IF NOT EXISTS bucket_to_contents ON contents (bucket);
 
  
 
 The column value “bucket” is concatenated with the tenantId (bucket = 
 bucketname+tenantID) in order to avoid filtering on the tenantID on my 
 application.
 
  
 
 Is it the rights way to do or should I create another column family to link 
 each content to the bucket ?
 
  
 
 CREATE TABLE IF NOT EXISTS bucket_to_contents (tenantID varchar,
 
  key varchar,
 
 type varchar,
 
 bucket varchar,
 
 owner varchar,
 
 workspace varchar,
 
  public_read boolean  PRIMARY KEY ((bucket, tenantID), key));
 
  
 
 Under the hood what is the difference of the both solutions?
 
  
 
 According to my understanding, the result will be the same. Both will have 
 the rowkey equals to the “bucketname”  and the “tenantID”.
 
 Excepted that the secondary index can have a replication delay…
 
  
 
 Can you help me on this point?
 
  
 
 Regards,
 
 Eric
 
  
 
 
 
 Ce message et les pièces jointes sont confidentiels et réservés à l'usage 
 exclusif de ses destinataires. Il peut également être protégé par le secret 
 professionnel. Si vous recevez ce message par erreur, merci d'en avertir 
 immédiatement l'expéditeur et de le détruire. L'intégrité du message ne 
 pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra 
 être recherchée quant au contenu de ce message. Bien que les meilleurs 
 efforts soient faits pour maintenir cette transmission exempte de tout virus, 
 l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne 
 saurait être recherchée pour tout dommage résultant d'un virus transmis.
 
 This e-mail and the documents attached are confidential and intended solely 
 for the addressee; it may also be privileged. If you receive this e-mail in 
 error, please notify the sender immediately and destroy it. As its integrity 
 cannot be secured on the Internet, the Worldline liability cannot be 
 triggered for the message content. Although the sender endeavours to maintain 
 a computer virus-free network, the sender does not warrant that this 
 transmission is virus-free and will not be liable for any damages resulting 
 from any virus transmitted.
 

Re: 答复: can not query data from cassandra

[Mark Reddy]

Hi,

It looks like you are running into a known issue:
https://issues.apache.org/jira/browse/CASSANDRA-7780 and its being worked
on here: https://issues.apache.org/jira/browse/CASSANDRA-7145


Mark


On 20 August 2014 09:06, 鄢来琼 laiqiong@gtafe.com wrote:

  Is it related to MAX_HEAP_SIZE and HEAP_NEWSIZE?

 My system memory size is 4G,but the MAX_HEAP_SIZE and HEAP_NEWSIZE are set
 to 2G and 800M, respectively.




  --

 Thanks and regards,

 GTA Data

 *Peter.YAN*

 E-Mail：laiqiong@gtafe.com

 http://www.gtafe.com/

 [image: 说明: cid:image001.png@01CF5897.E1268DE0]





 *发件人:* 鄢来琼 [mailto:laiqiong@gtafe.com]
 *发送时间:* 2014年8月20日 14:13
 *收件人:* user@cassandra.apache.org
 *主题:* can not query data from cassandra



 HI ALL,



 I setup Cassandra on a linux host.

 I have insert some data into “mykeyspace.cffex_l23” table.

 The following error are raised during query data from
 “mykeyspace.cffex_l23”.

 Could you give me any suggestion to fix it?



 According to “top” cmd, I found that most of the memory are used by
 Cassandra.



 *Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,
 0.0%st*

 *Mem:   4043660k total,  3281336k used,   762324k free,42704k buffers*

 *Swap:0k total,0k used,0k free,   741456k cached*



 *  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND*

 *7583 root  20   0 4437m 2.3g  65m S  0.0 58.7 531:32.23 java*

 *10907 root  15   0  575m  27m 2024 S  0.0  0.7   0:03.07 mongod*



 The system.log is attached.



 INFO [MemoryMeter:1] 2014-08-20 14:00:24,612 Memtable.java (line 481)
 CFS(Keyspace='mykeyspace', ColumnFamily='cffex_l23') liveRatio is
 6.294228546622461 (just-counted was 6.294228546622461).  calculation took
 2907ms for 279972 cells

 ERROR [ReadStage:871] 2014-08-20 14:01:51,197 CassandraDaemon.java (line
 199) Exception in thread Thread[ReadStage:871,5,main]

 java.lang.RuntimeException: java.lang.RuntimeException:
 java.io.FileNotFoundException:
 /opt/apache-cassandra-2.0.9/cassandraData/data/mykeyspace/cffex_l23/mykeyspace-cffex_l23-jb-799-Data.db
 (No such file or directory)

 at
 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1931)

 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

 at java.lang.Thread.run(Thread.java:745)

 Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
 /opt/apache-cassandra-2.0.9/cassandraData/data/mykeyspace/cffex_l23/mykeyspace-cffex_l23-jb-799-Data.db
 (No such file or directory)

 at
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47)

 at
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)

 at
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)

 at
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)

 at
 org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:396)

 at
 org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:366)

 at
 org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145)

 at
 org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45)

 at
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)

 at
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)

 at
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)

 at
 org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)

 at
 org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)

 at
 org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)

 at
 org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:87)

 at
 org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)

 at
 org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)

 at
 org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)

 at
 org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)

 at
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)

 at
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)

 at
 

Re: Cassandra Consistency Level

[Mark Reddy]

Hi,

As you are writing as CL.ONE and cqlsh by default reads at CL.ONE, there is
a probability that you are reading stale data i.e. the node you have
contacted for the read may not have the most recent data. If you have a
higher consistency requirement, you should look at increasing your
consistency level, for a more detailed look at this see:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html

If you want to continue using CL.ONE, you could look at increasing the
read_repair_chance for better consistency.
http://www.datastax.com/documentation/cassandra/2.0/cassandra/reference/referenceTableAttributes.html

Just to verify that this is in fact a consistency issue could you run a
nodetool repair on that table and run the same queries again?


Mark

Regards,
Mark


On 20 August 2014 00:14, Check Peck comptechge...@gmail.com wrote:

 We have cassandra cluster in three different datacenters (DC1, DC2 and
 DC3) and we have 10 machines in each datacenter. We have few tables in
 cassandra in which we have less than 100 records.

 What we are seeing - some tables are out of sync between machines in DC3
 as compared to DC1 or DC2 when we do select count(*) on it.

 As an example we did select count(*) while connecting to one cassandra
 machine in dc3 datacenter as compared to one cassandra machine in dc1
 datacenter and the results were different.

 root@machineA:/home/david/apache-cassandra/bin# python cqlsh
 dc3114.dc3.host.com
 Connected to TestCluster at dc3114.dc3.host.com:9160.
 [cqlsh 2.3.0 | Cassandra 1.2.9 | CQL spec 3.0.0 | Thrift protocol
 19.36.0]
 Use HELP for help.
 cqlsh use testingkeyspace ;
 cqlsh:testingkeyspace select count(*) from test_metadata ;

 count
 ---
 12

 cqlsh:testingkeyspace exit
 root@machineA:/home/david/apache-cassandra/bin# python cqlsh
 dc18b0c.dc1.host.com
 Connected to TestCluster at dc18b0c.dc1.host.com:9160.
 [cqlsh 2.3.0 | Cassandra 1.2.9 | CQL spec 3.0.0 | Thrift protocol
 19.36.0]
 Use HELP for help.
 cqlsh use testingkeyspace ;
 cqlsh:testingkeyspace select count(*) from test_metadata ;

 count
 ---
 16

 What could be the reason for this sync issue? Can anyone shed  some light
 on this?

 Since our java driver code and datastax c++ driver code are using these
 tables with CONSISTENCY LEVEL ONE.

Re: question about OpsCenter agent

[Mark Reddy]

Hi Clint,

You need to configure the datastax-agents so they know what machine
OpsCenter is running on. To do this you will need to edit the address.yaml
of the datastax-agent, located in /var/lib/datastax-agent/conf/. In this
file you need to add the following line:

stomp_interface: opscenter-ip-address


This will allow your agents to connect to the OpsCenter server (opscenterd)
over the stomp protocol (stomp.github.io/). For a more detailed look at the
configuration options available to the datastax-agent see this page:
datastax.com/documentation/opscenter/5.0/opsc/configure/agentAddressConfiguration.html


Mark



On Fri, Aug 15, 2014 at 3:32 AM, Clint Kelly clint.ke...@gmail.com wrote:

 Hi all,

 I just installed DataStax Enterprise 4.5.  I installed OpsCenter
 Server on one of my four machines.  The port that OpsCenter usually
 uses () was used by something else, so I modified
 /usr/share/opscenter/conf/opscenterd.conf to set the port to 8889.

 When I log into OpsCenter, it says 0 of 4 agents connected.

 I have started datastax-agent on all four of the nodes in my Cassandra
 cluster.  Is there another OpsCenter agent that I have to start?  I'm
 not sure how to fix this.

 I clicked on the Fix link and it prompted me You have nodes that
 need the OpsCenter agent installed.  How do I install the OpsCenter
 agent without using this GUI?  I don't recall getting prompted for
 that when I installed DSE.

 Best regards,
 Clint

Re: Bootstrap failures: unable to find sufficient sources for streaming range

[Mark Reddy]

Hi Peter,

At the time of the IllegalStateException, do you see the node that it
should be streaming from marked as down by the failure detector?


Mark


On Fri, Aug 15, 2014 at 5:45 AM, Peter Haggerty peter.hagge...@librato.com
wrote:

 When adding nodes via bootstrap to a 27 node 2.0.9 cluster with a
 cluster-wide phi_convict_threshold of 12 the nodes fail to bootstrap.
 This worked a half dozen times in the past few weeks as we've scaled
 this cluster from 21 to 24 and then to 27 nodes. There have been no
 configuration or Cassandra version changes since the cluster was
 booted several weeks ago. This is adding a single node at a time just
 as we've done before.

 When using a phi of 16 on the node that is bootstrapping we've had one
 failure and one success. Adjusting RING_DELAY from 30 seconds to 45
 doesn't appear to help.

 We have DEBUG level logs for a successful attempt and a failed attempt
 but they seem quite similar in behavior, even bad behavior like ERRORS
 reported by MigrationTask.java and FailureDetector.java.


 In the failed attempts the bootstrap immediately falls over with
 unable to find ...:
 java.lang.IllegalStateException: unable to find sufficient sources for
 streaming range (1445230693272554509,1449885986247309687]

 The logs clearly show that it knows where those ranges can be found:
 DEBUG [main] 2014-08-14 18:56:45,019 RangeStreamer.java (line 122)
 Bootstrap: range (1445230693272554509,1449885986247309687] exists on
 /10.11.12.13
 DEBUG [main] 2014-08-14 18:56:45,019 RangeStreamer.java (line 122)
 Bootstrap: range (1445230693272554509,1449885986247309687] exists on
 /10.11.12.14

 If you look for the various non-debug log lines for these addresses
 you'll see some odd behavior, but you see similar odd behavior in the
 logs of the node that is able to successfully bootstrap.
 $ grep '10.11.12.13' system.log | grep -v ^DEBUG
  INFO [GossipStage:1] 2014-08-14 18:56:09,870 Gossiper.java (line 910)
 Node /10.11.12.13 is now part of the cluster
  INFO [GossipStage:1] 2014-08-14 18:56:15,267 Gossiper.java (line 910)
 Node /10.11.12.13 is now part of the cluster
  INFO [HANDSHAKE-/10.11.12.13] 2014-08-14 18:56:45,571
 OutboundTcpConnection.java (line 386) Handshaking version with
 /10.11.12.13
  INFO [RequestResponseStage:5] 2014-08-14 18:56:45,579 Gossiper.java
 (line 876) InetAddress /10.11.12.13 is now UP
  INFO [RequestResponseStage:2] 2014-08-14 18:56:45,579 Gossiper.java
 (line 876) InetAddress /10.11.12.13 is now UP
  INFO [RequestResponseStage:7] 2014-08-14 18:56:45,580 Gossiper.java
 (line 876) InetAddress /10.11.12.13 is now UP
  INFO [RequestResponseStage:1] 2014-08-14 18:56:45,579 Gossiper.java
 (line 876) InetAddress /10.11.12.13 is now UP
  INFO [RequestResponseStage:8] 2014-08-14 18:56:45,579 Gossiper.java
 (line 876) InetAddress /10.11.12.13 is now UP

 $ grep '10.11.12.14' system.log | grep -v ^DEBUG
  INFO [GossipStage:1] 2014-08-14 18:56:09,865 Gossiper.java (line 910)
 Node /10.11.12.14 is now part of the cluster
  INFO [GossipStage:1] 2014-08-14 18:56:15,106 Gossiper.java (line 910)
 Node /10.11.12.14 is now part of the cluster

 The general order of events, as told by JOINING lines:
 $ grep 'JOINING' system.log
  INFO [main] 2014-08-14 18:56:10,638 StorageService.java (line 1009)
 JOINING: waiting for ring information
  INFO [main] 2014-08-14 18:56:14,640 StorageService.java (line 1009)
 JOINING: schema complete, ready to bootstrap
  INFO [main] 2014-08-14 18:56:14,640 StorageService.java (line 1009)
 JOINING: waiting for pending range calculation
  INFO [main] 2014-08-14 18:56:14,641 StorageService.java (line 1009)
 JOINING: calculation complete, ready to bootstrap
  INFO [main] 2014-08-14 18:56:14,641 StorageService.java (line 1009)
 JOINING: getting bootstrap token
  INFO [main] 2014-08-14 18:56:14,712 StorageService.java (line 1009)
 JOINING: sleeping 3 ms for pending range setup
  INFO [main] 2014-08-14 18:56:44,713 StorageService.java (line 1009)
 JOINING: Starting to bootstrap...


 Thanks in advance for any suggestions.


 Peter

Re: How to prevent the removed DC comes back automactically?

[Mark Reddy]

 How can we prevent a disconnected DC from coming back automatically?


You could use firewall rules to prevent the disconnected DC from contacting
your live DCs when it becomes live again


Mark


On Thu, Aug 14, 2014 at 6:48 AM, Lu, Boying boying...@emc.com wrote:

 Hi, All,



 We are using Cassandra 2.0.7 in a multi DCs environments.  If a connected
 DC is powered off, we use the ‘nodetool removenode’ command to remove it
 from the connected DCs.

 But we found that once the disconnected DC is powered on, it will connect
 to other DCs automatically.



 How can we prevent a disconnected DC from coming back automatically?



 Thanks a lot



 Boying

Re: A question to nodetool removenode command

[Mark Reddy]

Hi,

Gossip will propagate to all nodes in a cluster. So if you have a cluster
spanning DC1, DC2 and DC3 and you then remove all nodes in DC3 via nodetool
removenode from a node in DC1, all nodes in both DC1 and DC2 will be
informed of the nodes removal so no need to run it from a node in DC2.


Mark


On Thu, Aug 14, 2014 at 10:20 AM, Lu, Boying boying...@emc.com wrote:

 Hi, All,



 We have a Cassandra 2.0.7 running in three connected DCs, say DC1, DC2 and
 DC3.



 DC3 is powered off, so we run  ‘nodetool removenode’ command on DC1 to
 remove all nodes of DC3.

 Do we need to run the same command on DC2?



 Thanks



 Boying

Re: Cassandra corrupt column family

[Mark Reddy]

Hi,

Without more information (Cassandra version, setup, topology, schema,
queries performed) this list won't be able to assist you. If you can
provide a more detailed explanation of the steps you took to reach your
current state that would be great.


Mark


On Tue, Aug 12, 2014 at 12:21 PM, Batranut Bogdan batra...@yahoo.com
wrote:

 Hello all,

 I have altered a table in cassandra and on one node it somehow got
 corrupted. I the changes did not propagate ok. Ran repair keyspace
 columnfamily... noting changed...

 Is there a way to repair this?

Re: Nodetool Repair questions

[Mark Reddy]

Hi Vish,

1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?


Even if you never delete data you should run repairs occasionally to ensure
overall consistency. While hinted handoffs and read repairs do lead to
better consistency, they are only helpers/optimization and are not regarded
as operations that ensure consistency.

2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ?


Its difficult to calculate the performance of a repair, I've seen the time
to completion fluctuate between 4hrs to 10hrs+ on the same node. However in
theory adding more nodes would spread the data and free up machine
resources, thus resulting in more performant repairs.

The essence of this question is: If I have a keyspace with x number of
 replicas in each data center, do I have to deal with an upper limit on the
 number of data centers/nodes?


Could you expand on why you believe there would be an upper limit of
dc/nodes due to running repairs?


Mark


On Tue, Aug 12, 2014 at 10:06 PM, Viswanathan Ramachandran 
vish.ramachand...@gmail.com wrote:

 Some questions on nodetool repair.

 1. This tool repairs inconsistencies across replicas of the row. Since
 latest update always wins, I dont see inconsistencies other than ones
 resulting from the combination of deletes, tombstones, and crashed nodes.
 Technically, if data is never deleted from cassandra, then nodetool repair
 does not need to be run at all. Is this understanding correct? If wrong,
 can anyone provide other ways inconsistencies could occur?

 2. Want to understand the performance of 'nodetool repair' in a Cassandra
 multi data center setup. As we add nodes to the cluster in various data
 centers, does the performance of nodetool repair on each node increase
 linearly, or is it quadratic ? The essence of this question is: If I have a
 keyspace with x number of replicas in each data center, do I have to deal
 with an upper limit on the number of data centers/nodes?


 Thanks

 Vish

Re: Cassandra bootstrap error: java.lang.IllegalStateException: No sources found

[Mark Reddy]

Hi,

When you configured the new node, did you correctly configure your seeds
and listen address?


Mark


On Mon, Aug 11, 2014 at 6:12 PM, Rahul Gupta rgu...@dekaresearch.com
wrote:

  I have a 3 node Cassandra cluster and I am adding a new node when I get
 below error during bootstrap.

 Running DataStax ver 4.5 (Cassandra 2.0.8.39) on all the nodes. Nodes are
 VMs running on VMWare .



 Any ideas what might be wrong here?



 INFO [main] 2014-08-11 13:01:09,427 StorageService.java (line 1007)
 JOINING: Starting to bootstrap...

 ERROR [main] 2014-08-11 13:01:09,443 CassandraDaemon.java (line 513)
 Exception encountered during startup

 java.lang.IllegalStateException: No sources found for
 (85070591730234615865843651857942052864,134599085099335834412322738856718602703]

 at
 org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:159)

 at
 org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:117)

 at
 org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:72)

 at
 org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1035)

 at
 org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:797)

 at
 org.apache.cassandra.service.StorageService.initServer(StorageService.java:614)

 at
 org.apache.cassandra.service.StorageService.initServer(StorageService.java:504)

 at
 org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)

 at
 com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:394)

 at
 org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)

 at
 com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:574)

 INFO [Thread-1] 2014-08-11 13:01:09,451 DseDaemon.java (line 477) DSE
 shutting down...

 INFO [StorageServiceShutdownHook] 2014-08-11 13:01:09,476 Gossiper.java
 (line 1279) Announcing shutdown

 INFO [Thread-1] 2014-08-11 13:01:09,521 PluginManager.java (line 317) All
 plugins are stopped.

 INFO [Thread-1] 2014-08-11 13:01:09,522 CassandraDaemon.java (line 463)
 Cassandra shutting down...

 ERROR [Thread-1] 2014-08-11 13:01:09,522 CassandraDaemon.java (line 199)
 Exception in thread Thread[Thread-1,5,main]

 java.lang.NullPointerException

 at
 org.apache.cassandra.service.CassandraDaemon.stop(CassandraDaemon.java:464)

 at
 com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:480)

 at
 com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)

 INFO [StorageServiceShutdownHook] 2014-08-11 13:01:11,477
 MessagingService.java (line 683) Waiting for messaging service to quiesce

 INFO [ACCEPT-/172.17.0.122] 2014-08-11 13:01:11,478 MessagingService.java
 (line 923) MessagingService has terminated the accept() thread



 *Rahul Gupta*
 *DEKA* *Research  Development* http://www.dekaresearch.com/

 340 Commercial St  Manchester, NH  03101

 P: 603.666.3908 extn. 6504 | C: 603.718.9676



 This e-mail and the information, including any attachments, it contains
 are intended to be a confidential communication only to the person or
 entity to whom it is addressed and may contain information that is
 privileged. If the reader of this message is not the intended recipient,
 you are hereby notified that any dissemination, distribution or copying of
 this communication is strictly prohibited. If you have received this
 communication in error, please immediately notify the sender and destroy
 the original message.



 --
 This e-mail and the information, including any attachments it contains,
 are intended to be a confidential communication only to the person or
 entity to whom it is addressed and may contain information that is
 privileged. If the reader of this message is not the intended recipient,
 you are hereby notified that any dissemination, distribution or copying of
 this communication is strictly prohibited. If you have received this
 communication in error, please immediately notify the sender and destroy
 the original message.

 Thank you.

 Please consider the environment before printing this email.

Re: Unable to connect to cassandra server on EC2?

[Mark Reddy]

Hi,

While I have no direct experience with the Python driver itself I took a
quick and it uses Cassandra's native transport protocol, so setting the
port to 9160 (Thrift protocol) won't work. You will need to set it to the
native transport port, which is 9042.

Also make sure that you have the rpc_address (and if needed
broadcast_rpc_address) set correctly.

On a side note, if you are trying out 2.1.0, 2.1.0-rc5 is out and with many
fixes and improvements.


Mark


On Sun, Aug 10, 2014 at 1:04 PM, Subodh Nijsure subodh.nijs...@gmail.com
wrote:

 Hello,


 I have setup a new EC2 instance to run cassandra on EC2, have gone
 through bunch of questions that don't seem to help. I am running
 apache-cassandra-2.1.0-rc3

 I have opened port 9160, 9042 on my EC2 instance say its IP address is
 1.2.3.4

 Since this is single node system I haven't opened other ports as pointed in


 http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMISecurityGroup.html

 When I try to connect to this host using the python driver with
 cluster config like this:

 cluster = Cluster(contact_points=['1.2.3.4',], port=9160)

 I am getting error:

  File cluster.py, line 1649, in _reconnect_internal
 raise NoHostAvailable(Unable to connect to any servers, errors)
 cassandra.cluster.NoHostAvailable: ('Unable to connect to any
 servers', {'1.2.3.4': error(111, 'Connection refused')})

 Any ideas on what incantation I need to have in cassandra.yml file to
 allow external clients to connect cassandra instance running on EC2 ?

 -Subodh

Re: clarification on 100k tombstone limit in indexes

[Mark Reddy]

Hi Ian,

The issues here, which relates to normal and index column families, is
scanning over a large number of tombstones can cause Cassandra to fall over
due to increased GC pressure. This pressure is caused because tombstones
will create DeletedColumn objects which consume heap. Also
these DeletedColumn objects will have to be serialized and sent back to the
coordinator, thus increasing your response times. Take for example a row
that does deletes and you query it with a limit of 100. In a worst case
scenario you could end up reading say 50k tombstones to reach the 100
'live' column limit, all of which has to be put on heap and then sent over
the wire to the coordinator. This would be considered a Cassandra
anti-pattern.[1]

With that in mind there was a debug warning added to 1.2 to inform the user
when they were querying a row with 1000 tombstones [2]. Then in 2.0 the
action was taken to drop requests reaching 100k tombstones[3] rather than
just printing out a warning. This is a safety measure, as it is not advised
to perform such a query and is a result of most people 'doing it wrong'.

For those people who understand the risk of scanning over large numbers of
tombstones there is a configuration option in the cassandra.yaml to
increase this threshold, tombstone_failure_threshold.[4]


Mark

[1]
http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
[2] https://issues.apache.org/jira/browse/CASSANDRA-6042
[3] https://issues.apache.org/jira/browse/CASSANDRA-6117
[4]
https://github.com/jbellis/cassandra/blob/4ac18ae805d28d8f4cb44b42e2244bfa6d2875e1/conf/cassandra.yaml#L407-L417



On Sun, Aug 10, 2014 at 7:19 PM, Ian Rose ianr...@fullstory.com wrote:

 Hi -

 On this page (
 http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html),
 the docs state:

 Do not use an index [...] On a frequently updated or deleted column


 and


 *Problems using an index on a frequently updated or deleted column*¶
 http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html?scroll=concept_ds_sgh_yzz_zj__upDatIndx

 Cassandra stores tombstones in the index until the tombstone limit reaches
 100K cells. After exceeding the tombstone limit, the query that uses the
 indexed value will fail.



 I'm afraid I don't really understand this limit from its (brief)
 description.  I also saw this recent thread
 http://mail-archives.apache.org/mod_mbox/cassandra-user/201403.mbox/%3CCABNXB2Bf4aeoDVpMNOxJ_e7aDez2EuZswMJx=jWfb8=oyo4...@mail.gmail.com%3E
  but
 I'm afraid it didn't help me much...


 *SHORT VERSION*

 If I have tens or hundreds of thousands of rows in a keyspace, where every
 row has an indexed column that is updated O(10) times during the lifetime
 of each row, is that going to cause problems for me?  If that 100k limit is 
 *per
 row* then I should be fine but if that 100k limit is *per keyspace* then
 I'd definitely exceed it quickly.


 *FULL EXPLANATION*

 In our system, items are created at a rate of ~10/sec.  Each item is
 updated ~10 times over the next few minutes (although in rare cases the
 number of updates, and the duration, might be several times as long).  Once
 the last update is received for an item, we select it from Cassandra,
 process the data, then delete the entire row.

 The tricky bit is that sometimes (maybe 30-40% of the time) we don't
 actually know when the last update has been received so we use a timeout:
 if an item hasn't been updated for 30 minutes, then we assume it is done
 and should process it as before (select, then delete).  So I am trying to
 design a schema that will allow for efficient queries of the form find me
 all items that have not been updated in the past 30 minutes.  We plan to
 call this query once a minute.

 Here is my tentative schema:

 CREATE TABLE items (
   item_id ascii,
   last_updated timestamp,
   item_data listblob,
   PRIMARY KEY (item_id)
 )
 plus an index on last_updated.

 So updates to an existing item would just be lookup by item_id, append
 new data to item_data, and set last_updated to now.  And queries to find
 items that have timed out would use the index on last_updated: find all
 items where last_updated  [now - 30 minutes].

 Assuming, that is, that the aforementioned 100k tombstone limit won't
 bring this index crashing to a halt...

 Any clarification on this limit and/or suggestions on a better way to
 model/implement this system would be greatly appreciated!

 Cheers,
 Ian

Re: clarification on 100k tombstone limit in indexes

[Mark Reddy]

Hi Ian

Are these tombstones ever GCed out of the index?  How frequently?


Yes, tombstones are removed after the time specified by gc_grace_seconds
has elapsed, which by default is 10 days and is configurable. Knowing and
understanding how Cassandra handles distributed deletes is key to designing
an efficient schema if you plan to delete often. There are lots of
resources on how deletes are handled in Cassandra, take a look at these
links for example:
http://wiki.apache.org/cassandra/DistributedDeletes
http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_about_deletes_c.html

Mark


On Sun, Aug 10, 2014 at 9:02 PM, Ian Rose ianr...@fullstory.com wrote:

 Hi Mark -

 Thanks for the clarification but as I'm not too familiar with the nuts 
 bolts of Cassandra I'm not sure how to apply that info to my current
 situation.  It sounds like this 100k limit is, indeed, a global limit as
 opposed to a per-row limit.  Are these tombstones ever GCed out of the
 index?  How frequently?  If not, then it seems like *any* index is at risk
 of reaching this tipping point; it's just that indexes on frequently
 updated columns will reach this pointer faster the indexes on rarely
 updated columns.

 Basically I'm trying to get some kind of sense for what frequently
 updated
 http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html
 means quantitatively.  As written, the docs make it sound dangerous to
 create an index on a column that is *ever* deleted or updated since there
 is no sense of how frequent is too frequent.

 Cheers,
 Ian







 On Sun, Aug 10, 2014 at 3:02 PM, Mark Reddy mark.re...@boxever.com
 wrote:

 Hi Ian,

 The issues here, which relates to normal and index column families, is
 scanning over a large number of tombstones can cause Cassandra to fall over
 due to increased GC pressure. This pressure is caused because tombstones
 will create DeletedColumn objects which consume heap. Also
 these DeletedColumn objects will have to be serialized and sent back to the
 coordinator, thus increasing your response times. Take for example a row
 that does deletes and you query it with a limit of 100. In a worst case
 scenario you could end up reading say 50k tombstones to reach the 100
 'live' column limit, all of which has to be put on heap and then sent over
 the wire to the coordinator. This would be considered a Cassandra
 anti-pattern.[1]

 With that in mind there was a debug warning added to 1.2 to inform the
 user when they were querying a row with 1000 tombstones [2]. Then in 2.0
 the action was taken to drop requests reaching 100k tombstones[3] rather
 than just printing out a warning. This is a safety measure, as it is not
 advised to perform such a query and is a result of most people 'doing it
 wrong'.

 For those people who understand the risk of scanning over large numbers
 of tombstones there is a configuration option in the cassandra.yaml to
 increase this threshold, tombstone_failure_threshold.[4]


 Mark

 [1]
 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
 [2] https://issues.apache.org/jira/browse/CASSANDRA-6042
 [3] https://issues.apache.org/jira/browse/CASSANDRA-6117
 [4]
 https://github.com/jbellis/cassandra/blob/4ac18ae805d28d8f4cb44b42e2244bfa6d2875e1/conf/cassandra.yaml#L407-L417



 On Sun, Aug 10, 2014 at 7:19 PM, Ian Rose ianr...@fullstory.com wrote:

 Hi -

 On this page (
 http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html),
 the docs state:

 Do not use an index [...] On a frequently updated or deleted column


 and


 *Problems using an index on a frequently updated or deleted column*¶
 http://www.datastax.com/documentation/cql/3.0/cql/ddl/ddl_when_use_index_c.html?scroll=concept_ds_sgh_yzz_zj__upDatIndx

 Cassandra stores tombstones in the index until the tombstone limit
 reaches 100K cells. After exceeding the tombstone limit, the query that
 uses the indexed value will fail.



 I'm afraid I don't really understand this limit from its (brief)
 description.  I also saw this recent thread
 http://mail-archives.apache.org/mod_mbox/cassandra-user/201403.mbox/%3CCABNXB2Bf4aeoDVpMNOxJ_e7aDez2EuZswMJx=jWfb8=oyo4...@mail.gmail.com%3E
  but
 I'm afraid it didn't help me much...


 *SHORT VERSION*

 If I have tens or hundreds of thousands of rows in a keyspace, where
 every row has an indexed column that is updated O(10) times during the
 lifetime of each row, is that going to cause problems for me?  If that 100k
 limit is *per row* then I should be fine but if that 100k limit is *per
 keyspace* then I'd definitely exceed it quickly.


 *FULL EXPLANATION*

 In our system, items are created at a rate of ~10/sec.  Each item is
 updated ~10 times over the next few minutes (although in rare cases the
 number of updates, and the duration, might be several times as long).  Once
 the last update is received for an item, we select it from Cassandra,
 process the data, then delete the entire

Re: Reasonable range for the max number of tables?

[Mark Reddy]

Hi Phil,

In theory, the max number of column families would be in the low number of
hundreds. In practice the limit is related the amount of heap you have, as
each column family will consume 1 MB of heap due to arena allocation.

To segregate customer data, you could:
- Use customer specific column families under a single keyspace
- Use a keyspace per customer
- Use the same column families and have a column that identifies the
customer. On the application layer ensure that there are sufficient checks
so one customer can't read another customers data


Mark


On Tue, Aug 5, 2014 at 9:09 AM, Phil Luckhurst 
phil.luckhu...@powerassure.com wrote:

 Is there any mention of this limitation anywhere in the Cassandra
 documentation? I don't see it mentioned in the 'Anti-patterns in Cassandra'
 section of the DataStax 2.0 documentation or anywhere else.

 When starting out with Cassandra as a store for a multi-tenant application
 it seems very attractive to segregate data for each tenant using a tenant
 specific keyspace each with their own set of tables. It's not until you
 start browsing through forums such as this that you find out that it isn't
 going to scale above a few tenants.

 If you want to be able to segregate customer data in Cassandra is it the
 accepted practice to have multiple Cassandra installations?



 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Reasonable-range-for-the-max-number-of-tables-tp7596094p7596106.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: Make an existing cluster multi data-center compatible.

[Mark Reddy]

Yes, you must run a full repair for the reasons stated in the yaml file.


Mark


On Tue, Aug 5, 2014 at 11:52 AM, Rene Kochen rene.koc...@schange.com
wrote:

 Hi all,

 I want to add a data-center to an existing single data-center cluster.
 First I have to make the existing cluster multi data-center compatible.

 The existing cluster is a 12 node cluster with:
 - Replication factor = 3
 - Placement strategy = SimpleStrategy
 - Endpoint snitch = SimpleSnitch

 If I change the following:
 - Placement strategy = NetworkTopologyStrategy
 - Endpoint snitch = PropertyFileSnitch - all 12 nodes in this file belong
 to the same data-center and rack.

 Do I have to run full repairs after this change? Because the yaml file
 states: IF YOU CHANGE THE SNITCH AFTER DATA IS INSERTED INTO THE CLUSTER,
 YOU MUST RUN A FULL REPAIR, SINCE THE SNITCH AFFECTS WHERE REPLICAS ARE
 PLACED.

 Thanks!

 Rene

Re: Node bootstrap

[Mark Reddy]

 Yes num_tokens is set to 256. initial_token is blank on all nodes
 including the new one.


Ok so you have num_tokens set to 256 for all nodes with initial_token
commented out, this means you are using vnodes and the new node will
automatically grab a list of tokens to take over responsibility for.

Pool NameActive   Pending  Completed   Blocked  All
 time blocked
 FlushWriter   0 0   1136 0
   512

 Looks like about 50% of flushes are blocked.


This is a problem as it indicates that the IO system cannot keep up.

Just ran this on the new node:
 nodetool netstats | grep Streaming from | wc -l
 10


This is normal as the new node will most likely take tokens from all nodes
in the cluster.

Sorry for the multiple updates, but another thing I found was all the other
 existing nodes have themselves in the seeds list, but the new node does not
 have itself in the seeds list. Can that cause this issue?


Seeds are only used when a new node is bootstrapping into the cluster and
needs a set of ips to contact and discover the cluster, so this would have
no impact on data sizes or streaming. In general it would be considered
best practice to have a set of 2-3 seeds from each data center, with all
nodes having the same seed list.


What is the current output of 'nodetool compactionstats'? Could you also
paste the output of nodetool status keyspace?

Mark



On Tue, Aug 5, 2014 at 3:59 PM, Ruchir Jha ruchir@gmail.com wrote:

 Sorry for the multiple updates, but another thing I found was all the
 other existing nodes have themselves in the seeds list, but the new node
 does not have itself in the seeds list. Can that cause this issue?


 On Tue, Aug 5, 2014 at 10:30 AM, Ruchir Jha ruchir@gmail.com wrote:

 Just ran this on the new node:

 nodetool netstats | grep Streaming from | wc -l
 10

 Seems like the new node is receiving data from 10 other nodes. Is that
 expected in a vnodes enabled environment?

 Ruchir.



 On Tue, Aug 5, 2014 at 10:21 AM, Ruchir Jha ruchir@gmail.com wrote:

 Also not sure if this is relevant but just noticed the nodetool tpstats
 output:

 Pool NameActive   Pending  Completed   Blocked
  All time blocked
 FlushWriter   0 0   1136 0
 512

 Looks like about 50% of flushes are blocked.


 On Tue, Aug 5, 2014 at 10:14 AM, Ruchir Jha ruchir@gmail.com
 wrote:

 Yes num_tokens is set to 256. initial_token is blank on all nodes
 including the new one.


 On Tue, Aug 5, 2014 at 10:03 AM, Mark Reddy mark.re...@boxever.com
 wrote:

 My understanding was that if initial_token is left empty on the new
 node, it just contacts the heaviest node and bisects its token range.


 If you are using vnodes and you have num_tokens set to 256 the new
 node will take token ranges dynamically. What is the configuration of your
 other nodes, are you setting num_tokens or initial_token on those?


 Mark


 On Tue, Aug 5, 2014 at 2:57 PM, Ruchir Jha ruchir@gmail.com
 wrote:

 Thanks Patricia for your response!

 On the new node, I just see a lot of the following:

 INFO [FlushWriter:75] 2014-08-05 09:53:04,394 Memtable.java (line
 400) Writing Memtable
 INFO [CompactionExecutor:3] 2014-08-05 09:53:11,132
 CompactionTask.java (line 262) Compacted 12 sstables to

 so basically it is just busy flushing, and compacting. Would you have
 any ideas on why the 2x disk space blow up. My understanding was that if
 initial_token is left empty on the new node, it just contacts the 
 heaviest
 node and bisects its token range. And the heaviest node is around 2.1 TB,
 and the new node is already at 4 TB. Could this be because compaction is
 falling behind?

 Ruchir


 On Mon, Aug 4, 2014 at 7:23 PM, Patricia Gorla 
 patri...@thelastpickle.com wrote:

 Ruchir,

 What exactly are you seeing in the logs? Are you running major
 compactions on the new bootstrapping node?

 With respect to the seed list, it is generally advisable to use 3
 seed nodes per AZ / DC.

 Cheers,


 On Mon, Aug 4, 2014 at 11:41 AM, Ruchir Jha ruchir@gmail.com
 wrote:

 I am trying to bootstrap the thirteenth node in a 12 node cluster
 where the average data size per node is about 2.1 TB. The bootstrap
 streaming has been going on for 2 days now, and the disk size on the 
 new
 node is already above 4 TB and still going. Is this because the new 
 node is
 running major compactions while the streaming is going on?

 One thing that I noticed that seemed off was the seeds property in
 the yaml of the 13th node comprises of 1..12. Where as the seeds 
 property
 on the existing 12 nodes consists of all the other nodes except the
 thirteenth node. Is this an issue?

 Any other insight is appreciated?

 Ruchir.





 --
 Patricia Gorla
 @patriciagorla

 Consultant
 Apache Cassandra Consulting
 http://www.thelastpickle.com http://thelastpickle.com

Re: A question about using 'update keyspace with strategyoptions' command

[Mark Reddy]

 So the ‘strategy’ change may not be seen by all nodes when the ‘upgrade
 keyspace …’ command returns and I can use ’describe cluster’ to check if
 the change has taken effect on all nodes right?


Correct, the change may take time to propagate to all nodes. As Rahul said
you can check describe cluster in cli to be sure.


Mark


On Tue, Aug 5, 2014 at 3:06 PM, Lu, Boying boying...@emc.com wrote:

 Thanks a lot.



 So the ‘strategy’ change may not be seen by all nodes when the ‘upgrade
 keyspace …’ command returns and I can use ’describe cluster’ to check if

 the change has taken effect on all nodes right?



 *From:* Rahul Neelakantan [mailto:ra...@rahul.be]
 *Sent:* 2014年8月5日 18:46
 *To:* user@cassandra.apache.org
 *Subject:* Re: A question about using 'update keyspace with
 strategyoptions' command



 Try running describe cluster from Cassandra-CLI to see if all nodes have
 the same schema version.

 Rahul Neelakantan


 On Aug 5, 2014, at 6:13 AM, Sylvain Lebresne sylv...@datastax.com wrote:

 On Tue, Aug 5, 2014 at 11:40 AM, Lu, Boying boying...@emc.com wrote:

 What I want to know is “are the *strategy* changed ?’ after the ‘udpate
 keyspace with strategy_options…’ command returns successfully



 Like all schema changes, not necessarily on all nodes. You will have to
 check for schema agreement between nodes.




 Not the *data* change.



 e.g. say I run the command ‘update keyspace with strategy_opitons [dc1: 3,
 dc2:3]’ , when this command returns,

 are the *strategy* options already changed? Or I need to wait some time
 for the strategy to be changed?





 *From:* Sylvain Lebresne [mailto:sylv...@datastax.com]
 *Sent:* 2014年8月5日 16:59
 *To:* user@cassandra.apache.org


 *Subject:* Re: A question about using 'update keyspace with
 strategyoptions' command



 Changing the strategy options, and in particular the replication factor,
 does not perform any data replication by itself. You need to run a repair
 to ensure data is replicated following the new replication.



 On Tue, Aug 5, 2014 at 10:52 AM, Lu, Boying boying...@emc.com wrote:

 Thanks. yes. I can use the ‘show keyspace’ command to check and see the
 strategy does changed.



 But what I want to know is if the ‘update keyspace with strategy_options
 …’ command is

 a ‘sync’ operation or a ‘async’ operation.







 *From:* Rahul Menon [mailto:ra...@apigee.com]
 *Sent:* 2014年8月5日 16:38
 *To:* user
 *Subject:* Re: A question about using 'update keyspace with
 strategyoptions' command



 Try the show keyspaces command and look for Options under each keyspace.



 Thanks

 Rahul



 On Tue, Aug 5, 2014 at 2:01 PM, Lu, Boying boying...@emc.com wrote:

 Hi, All,



 I want to run ‘update keyspace with strategy_options={dc1:3, dc2:3}’ from
 cassandra-cli to update the strategy options of some keyspace

 in a multi-DC environment.



 When the command returns successfully, does it mean that the strategy
 options have been updated successfully or I need to wait

 some time for the change to be propagated  to all DCs?



 Thanks



 Boying

Re: Node bootstrap

[Mark Reddy]

Hi Ruchir,

With the large number of blocked flushes and the number of pending
compactions would still indicate IO contention. Can you post the output of
'iostat -x 5 5'

If you do in fact have spare IO, there are several configuration options
you can tune such as increasing the number of flush writers and
compaction_throughput_mb_per_sec

Mark


On Tue, Aug 5, 2014 at 5:22 PM, Ruchir Jha ruchir@gmail.com wrote:

 Also Mark to your comment on my tpstats output, below is my iostat output,
 and the iowait is at 4.59%, which means no IO pressure, but we are still
 seeing the bad flush performance. Should we try increasing the flush
 writers?


 Linux 2.6.32-358.el6.x86_64 (ny4lpcas13.fusionts.corp)  08/05/2014
  _x86_64_(24 CPU)

 avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   5.80   10.250.654.590.00   78.72

 Device:tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
 sda 103.83  9630.62 11982.60 3231174328 4020290310
 dm-0 13.57   160.1781.12   53739546   27217432
 dm-1  7.5916.9443.775682200   14686784
 dm-2   5792.76 32242.66 45427.12 10817753530 15241278360
 sdb 206.09 22789.19 33569.27 7646015080 11262843224



 On Tue, Aug 5, 2014 at 12:13 PM, Ruchir Jha ruchir@gmail.com wrote:

 nodetool status:

 Datacenter: datacenter1
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  Address  Load   Tokens  Owns (effective)  Host ID
   Rack
 UN  10.10.20.27  1.89 TB256 25.4%
 76023cdd-c42d-4068-8b53-ae94584b8b04  rack1
 UN  10.10.20.62  1.83 TB256 25.5%
 84b47313-da75-4519-94f3-3951d554a3e5  rack1
 UN  10.10.20.47  1.87 TB256 24.7%
 bcd51a92-3150-41ae-9c51-104ea154f6fa  rack1
 UN  10.10.20.45  1.7 TB 256 22.6%
 8d6bce33-8179-4660-8443-2cf822074ca4  rack1
 UN  10.10.20.15  1.86 TB256 24.5%
 01a01f07-4df2-4c87-98e9-8dd38b3e4aee  rack1
 UN  10.10.20.31  1.87 TB256 24.9%
 1435acf9-c64d-4bcd-b6a4-abcec209815e  rack1
 UN  10.10.20.35  1.86 TB256 25.8%
 17cb8772-2444-46ff-8525-33746514727d  rack1
 UN  10.10.20.51  1.89 TB256 25.0%
 0343cd58-3686-465f-8280-56fb72d161e2  rack1
 UN  10.10.20.19  1.91 TB256 25.5%
 30ddf003-4d59-4a3e-85fa-e94e4adba1cb  rack1
 UN  10.10.20.39  1.93 TB256 26.0%
 b7d44c26-4d75-4d36-a779-b7e7bdaecbc9  rack1
 UN  10.10.20.52  1.81 TB256 25.4%
 6b5aca07-1b14-4bc2-a7ba-96f026fa0e4e  rack1
 UN  10.10.20.22  1.89 TB256 24.8%
 46af9664-8975-4c91-847f-3f7b8f8d5ce2  rack1


 Note: The new node is not part of the above list.

 nodetool compactionstats:

 pending tasks: 1649
   compaction typekeyspace   column family   completed
   total  unit  progress
Compaction   iprod   customerorder  1682804084
 17956558077 bytes 9.37%
Compactionprodgatecustomerorder
  1664239271  1693502275 bytes98.27%
Compaction  qa_config_bkupfixsessionconfig_hist
  2443   27253 bytes 8.96%
Compactionprodgatecustomerorder_hist
  1770577280  5026699390 bytes35.22%
Compaction   iprodgatecustomerorder_hist
  2959560205312350192622 bytes 0.95%




 On Tue, Aug 5, 2014 at 11:37 AM, Mark Reddy mark.re...@boxever.com
 wrote:

 Yes num_tokens is set to 256. initial_token is blank on all nodes
 including the new one.


 Ok so you have num_tokens set to 256 for all nodes with initial_token
 commented out, this means you are using vnodes and the new node will
 automatically grab a list of tokens to take over responsibility for.

 Pool NameActive   Pending  Completed   Blocked
  All time blocked
 FlushWriter   0 0   1136 0
   512

 Looks like about 50% of flushes are blocked.


 This is a problem as it indicates that the IO system cannot keep up.

 Just ran this on the new node:
 nodetool netstats | grep Streaming from | wc -l
 10


 This is normal as the new node will most likely take tokens from all
 nodes in the cluster.

 Sorry for the multiple updates, but another thing I found was all the
 other existing nodes have themselves in the seeds list, but the new node
 does not have itself in the seeds list. Can that cause this issue?


 Seeds are only used when a new node is bootstrapping into the cluster
 and needs a set of ips to contact and discover the cluster, so this would
 have no impact on data sizes or streaming. In general it would be
 considered best practice to have a set of 2-3 seeds from each data center,
 with all nodes having the same seed list.


 What is the current output of 'nodetool compactionstats'? Could you also
 paste the output of nodetool status keyspace?

 Mark



 On Tue, Aug 5, 2014 at 3

Re: Node stuck during nodetool rebuild

[Mark Reddy]

Hi Vasilis,

To further on what Rob said

I believe you might be able to tune the phi detector threshold to help this
 operation complete, hopefully someone with direct experience of same will
 chime in.


I have been through this operation where streams break due to a node
falsely being marked down (flapping). In an attempt to  mitigate this I
increase the phi_convict_threshold in cassandra.yaml from 8 to 10, after
which the rebuild was able to successfully complete. The default value for
phi_convict_threshold is 8 with 12 being the maximum recommended value.


Mark


On Tue, Aug 5, 2014 at 7:22 PM, Robert Coli rc...@eventbrite.com wrote:

 On Tue, Aug 5, 2014 at 1:28 AM, Vasileios Vlachos 
 vasileiosvlac...@gmail.com wrote:

 The problem is that the nodetool seems to be stuck, and nodetool netstats
 on node1 of DC2 appears to be stuck at 10% streaming a 5G file from node2
 at DC1. This doesn't tally with nodetool netstats when running it against
 either of the DC1 nodes. The DC1 nodes don't think they stream anything to
 DC2.


 Yes, streaming is fragile and breaks and hangs forever and your only
 option in most cases is to stop the rebuilding node, nuke its data, and
 start again.

 I believe you might be able to tune the phi detector threshold to help
 this operation complete, hopefully someone with direct experience of same
 will chime in.

 =Rob

Re: `system` keyspace replication

[Mark Reddy]

Hi Jens,

The system keyspace is configured with LocalStrategy, this strategy only
stores data on the the local node. This strategy is reserved for internal
use only and is used for other things such as SecondaryIndexes.

You cannot change the replication factor of the 'system' keyspace. If you
attempt to do so, the command will not succeed and will return the
following message: Bad Request: Cannot alter system keyspace


Mark


On Thu, Jul 31, 2014 at 10:58 AM, Jens Rantil jens.ran...@tink.se wrote:

 Hi,

 Datastax has a documentation page about configuring replication[1]. It
 mentions a couple of system keyspaces that they recommend increasing
 replication for. However, it does not mention the `system` keyspace.

 Question: Is it recommended to increase replication factor for the
 `system` keyspace for production system?

 [1]
 http://www.datastax.com/docs/datastax_enterprise3.1/deploy/configuring_replication

 Thanks,
 Jens

Re: bootstrapping new nodes on 1.2.12

[Mark Reddy]

Hi Parag,

1)  Would anyone be able to help me interrupt this information from
 OpsCenter?


At a high level bootstrapping a new node has two phases, streaming and
secondary index builds. I believe OpsCenter will only report active
streams, the pending stream will be listed as such in OpsCenter as well. In
OpsCenter rather than looking at the Data Size check the used space on
the Storage Capacity pie chart, this will show how much data is on disk but
not necessarily live on the node yet.

Personally I would check 'nodetool netstats' to see what streams are
remaining, this will list all active / pending stream and what files are to
be streamed, at the moment you might just be streaming some very large
files and once complete you will see a dramatic increase in data size.

If streaming is complete and you use secondary indexes, check 'nodetool
compcationstats' for any secondary index builds that may be taking place.


2)  Is there anything I can do to speed this up?


If you have the capacity you could
increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml

If you don't have the capacity you could add more nodes to spread the data
so you stream less in future.

Finally you could upgrade to 2.0.x as it contains a complete refactor of
streaming and should make your streaming sessions more robust and
transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286


Mark


On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel ppa...@clearpoolgroup.com
wrote:

  Hi,



 It’s taking a while to boostrap a 13th node into a 12 node cluster.  The
 average node size is about 1.7TB.  At the beginning of today we were close
 to .9TB on the new node and 12 hours later we’re at 1.1TB.  I figured it
 would have finished by now because when I was looking on OpsCenter, there
 were 2 transfers remaining.  1 was at 0% and the other was at 2%.  I look
 again now and those same nodes haven’t progressed all day.  Instead I see 9
 more transfers (some of which are progressing).



 1)  Would anyone be able to help me interrupt this information from
 OpsCenter?

 2)  Is there anything I can do to speed this up?



 Thanks,

 Parag

Re: bootstrapping new nodes on 1.2.12

[Mark Reddy]

 Thanks for the detailed response.  I checked ‘nodetool netstats’ and I see
 there are pending streams, all of which are stuck at 0%.  I was expecting
 to see at least one output that was more than 0%.  Have you seen this
 before?


This could indicate that the bootstrap process is hung due to a failed
streaming session. Can you check your logs for the following line:

AbstractStreamSession.java (line 110) Stream failed because
/xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active
in background, but further streams won't be started)

If that is the case you will need to wipe the node and begin the
bootstrapping process again


Mark



On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel ppa...@clearpoolgroup.com
wrote:

  Thanks for the detailed response.  I checked ‘nodetool netstats’ and I
 see there are pending streams, all of which are stuck at 0%.  I was
 expecting to see at least one output that was more than 0%.  Have you seen
 this before?



 Side question – does a new node stream from other nodes in any particular
 order?  Perhaps this is a coincidence, but if I were to sort my hostnames
 in alphabetical order, it’s currently streaming from the last 2.



 *From:* Mark Reddy [mailto:mark.re...@boxever.com]
 *Sent:* Wednesday, July 30, 2014 4:42 AM
 *To:* user@cassandra.apache.org
 *Subject:* Re: bootstrapping new nodes on 1.2.12



 Hi Parag,



 1)  Would anyone be able to help me interrupt this information from
 OpsCenter?



 At a high level bootstrapping a new node has two phases, streaming and
 secondary index builds. I believe OpsCenter will only report active
 streams, the pending stream will be listed as such in OpsCenter as well. In
 OpsCenter rather than looking at the Data Size check the used space on
 the Storage Capacity pie chart, this will show how much data is on disk but
 not necessarily live on the node yet.



 Personally I would check 'nodetool netstats' to see what streams are
 remaining, this will list all active / pending stream and what files are to
 be streamed, at the moment you might just be streaming some very large
 files and once complete you will see a dramatic increase in data size.



 If streaming is complete and you use secondary indexes, check 'nodetool
 compcationstats' for any secondary index builds that may be taking place.





 2)  Is there anything I can do to speed this up?



 If you have the capacity you could
 increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml



 If you don't have the capacity you could add more nodes to spread the data
 so you stream less in future.



 Finally you could upgrade to 2.0.x as it contains a complete refactor of
 streaming and should make your streaming sessions more robust and
 transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286





 Mark



 On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel ppa...@clearpoolgroup.com
 wrote:

  Hi,



 It’s taking a while to boostrap a 13th node into a 12 node cluster.  The
 average node size is about 1.7TB.  At the beginning of today we were close
 to .9TB on the new node and 12 hours later we’re at 1.1TB.  I figured it
 would have finished by now because when I was looking on OpsCenter, there
 were 2 transfers remaining.  1 was at 0% and the other was at 2%.  I look
 again now and those same nodes haven’t progressed all day.  Instead I see 9
 more transfers (some of which are progressing).



 1)  Would anyone be able to help me interrupt this information from
 OpsCenter?

 2)  Is there anything I can do to speed this up?



 Thanks,

 Parag

Re: bootstrapping new nodes on 1.2.12

[Mark Reddy]

HI Parag,

I see this output my log many times over for 2 nodes.  We have a cron entry
 across all clusters that force a full GC at 2 AM.  node1 is due to Full
 GC that was scheduled (I can disable this).  Node2 was due to a Full GC
 that occurred during our peak operation (these happen occasionally, we’ve
 been working to reduce them).


Firstly, why are you forcing a GC? Do you have sufficient evidence that
Cassandra is not managing the heap in the way in which your application
requires?

Also how are you accomplishing this full GC? Are you removing the node from
the ring, forcing a GC and then adding it back in? Or are you forcing a GC
while it is in the ring?

1)  Will any node leaving the cluster while streaming force us to
 bootsrap all over again?  If so, is this addressed in future versions?


If the node that is leaving the ring is streaming data to  the
bootstrapping node, yes, this will break the streaming session and no further
streams will be started from that node. To my knowledge, there is nothing
in newer / future versions that will prevent this.


2)  We have too much data to migrate to run on non-production hours.
 How do we make it such that full GC’s don’t impact bootstrapping?  Should
 we increase phi_convict_threshold ?


Again I'll need some more information around these manual GC's. But yes,
increasing the phi value would reduce the chance of a node in the ring
being marked down during a heavy gc cycle.


Mark


On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel ppa...@clearpoolgroup.com
wrote:

  Mark,



 I see this output my log many times over for 2 nodes.  We have a cron
 entry across all clusters that force a full GC at 2 AM.  node1 is due to
 Full GC that was scheduled (I can disable this).  Node2 was due to a Full
 GC that occurred during our peak operation (these happen occasionally,
 we’ve been working to reduce them).  Few Questions



 1)  Will any node leaving the cluster while streaming force us to
 bootsrap all over again?  If so, is this addressed in future versions?

 2)  We have too much data to migrate to run on non-production hours.
 How do we make it such that full GC’s don’t impact bootstrapping?  Should
 we increase phi_convict_threshold ?



 Parag





 *From:* Mark Reddy [mailto:mark.re...@boxever.com]
 *Sent:* Wednesday, July 30, 2014 7:58 AM

 *To:* user@cassandra.apache.org
 *Subject:* Re: bootstrapping new nodes on 1.2.12



 Thanks for the detailed response.  I checked ‘nodetool netstats’ and I see
 there are pending streams, all of which are stuck at 0%.  I was expecting
 to see at least one output that was more than 0%.  Have you seen this
 before?



 This could indicate that the bootstrap process is hung due to a failed
 streaming session. Can you check your logs for the following line:



 AbstractStreamSession.java (line 110) Stream failed because
 /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active
 in background, but further streams won't be started)



 If that is the case you will need to wipe the node and begin the
 bootstrapping process again





 Mark





 On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel ppa...@clearpoolgroup.com
 wrote:

  Thanks for the detailed response.  I checked ‘nodetool netstats’ and I
 see there are pending streams, all of which are stuck at 0%.  I was
 expecting to see at least one output that was more than 0%.  Have you seen
 this before?



 Side question – does a new node stream from other nodes in any particular
 order?  Perhaps this is a coincidence, but if I were to sort my hostnames
 in alphabetical order, it’s currently streaming from the last 2.



 *From:* Mark Reddy [mailto:mark.re...@boxever.com]
 *Sent:* Wednesday, July 30, 2014 4:42 AM
 *To:* user@cassandra.apache.org
 *Subject:* Re: bootstrapping new nodes on 1.2.12



 Hi Parag,



 1)  Would anyone be able to help me interrupt this information from
 OpsCenter?



 At a high level bootstrapping a new node has two phases, streaming and
 secondary index builds. I believe OpsCenter will only report active
 streams, the pending stream will be listed as such in OpsCenter as well. In
 OpsCenter rather than looking at the Data Size check the used space on
 the Storage Capacity pie chart, this will show how much data is on disk but
 not necessarily live on the node yet.



 Personally I would check 'nodetool netstats' to see what streams are
 remaining, this will list all active / pending stream and what files are to
 be streamed, at the moment you might just be streaming some very large
 files and once complete you will see a dramatic increase in data size.



 If streaming is complete and you use secondary indexes, check 'nodetool
 compcationstats' for any secondary index builds that may be taking place.





 2)  Is there anything I can do to speed this up?



 If you have the capacity you could
 increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml



 If you don't have the capacity you

Re: bootstrapping new nodes on 1.2.12

[Mark Reddy]

 Our Full GC’s take about 9 seconds.  If we were to increase the
 phi_convict_threshold to not take a node offline for a 9 second
 unavailability, what negative side effects can there be?


When you observe these GC's do you also see the node being marked down and
then back up ~9 seconds later? GC's can often happen and have no effect on
gossip marking a node as down, in which case the streaming session will
remain intact. The side effect of long GC's is increased latency from that
node during that period.


Mark

On Wed, Jul 30, 2014 at 2:24 PM, Parag Patel ppa...@clearpoolgroup.com
wrote:

  As to why we do it, we need to reevaluate because the GC optimizations
 we’ve made recently probably don’t require it anymore.  However, prior to
 our optimizations we observed a benefit at our peak time. When we force a
 GC, we don’t remove it from the ring.  This seems like a fundamental flaw
 in our approach.  Thanks for pointing this out.  For the purposes of
 bootstrapping, we will disable the manual GC’s to make sure we don’t
 interrupt the joining process.  However, one unpredictable problem can
 always remain – a Full GC happens causing the node to go offline and
 causing the bootstrap to fail.  To solve this, we’ll try increasing the
 phi_convict_threshold.



 Our Full GC’s take about 9 seconds.  If we were to increase the
 phi_convict_threshold to not take a node offline for a 9 second
 unavailability, what negative side effects can there be?



 Parag





 *From:* Mark Reddy [mailto:mark.re...@boxever.com]
 *Sent:* Wednesday, July 30, 2014 9:06 AM

 *To:* user@cassandra.apache.org
 *Subject:* Re: bootstrapping new nodes on 1.2.12



 HI Parag,



 I see this output my log many times over for 2 nodes.  We have a cron
 entry across all clusters that force a full GC at 2 AM.  node1 is due to
 Full GC that was scheduled (I can disable this).  Node2 was due to a Full
 GC that occurred during our peak operation (these happen occasionally,
 we’ve been working to reduce them).



 Firstly, why are you forcing a GC? Do you have sufficient evidence that
 Cassandra is not managing the heap in the way in which your application
 requires?



 Also how are you accomplishing this full GC? Are you removing the node
 from the ring, forcing a GC and then adding it back in? Or are you forcing
 a GC while it is in the ring?



 1)  Will any node leaving the cluster while streaming force us to
 bootsrap all over again?  If so, is this addressed in future versions?



 If the node that is leaving the ring is streaming data to  the
 bootstrapping node, yes, this will break the streaming session and no further
 streams will be started from that node. To my knowledge, there is nothing
 in newer / future versions that will prevent this.





 2)  We have too much data to migrate to run on non-production hours.
 How do we make it such that full GC’s don’t impact bootstrapping?  Should
 we increase phi_convict_threshold ?



 Again I'll need some more information around these manual GC's. But yes,
 increasing the phi value would reduce the chance of a node in the ring
 being marked down during a heavy gc cycle.





 Mark



 On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel ppa...@clearpoolgroup.com
 wrote:

  Mark,



 I see this output my log many times over for 2 nodes.  We have a cron
 entry across all clusters that force a full GC at 2 AM.  node1 is due to
 Full GC that was scheduled (I can disable this).  Node2 was due to a Full
 GC that occurred during our peak operation (these happen occasionally,
 we’ve been working to reduce them).  Few Questions



 1)  Will any node leaving the cluster while streaming force us to
 bootsrap all over again?  If so, is this addressed in future versions?

 2)  We have too much data to migrate to run on non-production hours.
 How do we make it such that full GC’s don’t impact bootstrapping?  Should
 we increase phi_convict_threshold ?



 Parag





 *From:* Mark Reddy [mailto:mark.re...@boxever.com]
 *Sent:* Wednesday, July 30, 2014 7:58 AM


 *To:* user@cassandra.apache.org
 *Subject:* Re: bootstrapping new nodes on 1.2.12



 Thanks for the detailed response.  I checked ‘nodetool netstats’ and I see
 there are pending streams, all of which are stuck at 0%.  I was expecting
 to see at least one output that was more than 0%.  Have you seen this
 before?



 This could indicate that the bootstrap process is hung due to a failed
 streaming session. Can you check your logs for the following line:



 AbstractStreamSession.java (line 110) Stream failed because
 /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active
 in background, but further streams won't be started)



 If that is the case you will need to wipe the node and begin the
 bootstrapping process again





 Mark





 On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel ppa...@clearpoolgroup.com
 wrote:

  Thanks for the detailed response.  I checked ‘nodetool netstats’ and I
 see there are pending

Re: nodetool status returning negative load values on 2.1.0-rc4

[Mark Reddy]

Looks like you are running into this issue:
https://issues.apache.org/jira/browse/CASSANDRA-7239


Mark




On Tue, Jul 29, 2014 at 5:32 PM, Jeronimo de A. Barros 
jeronimo.bar...@gmail.com wrote:

 Hi everyone,

 After upgrade and clean install on a 5 nodes cluster:

 Datacenter: datacenter1
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  Address   Load   Tokens  Owns (effective)  Host ID
   Rack
 UN  192.168.3.50  -123645 bytes  256 61.1%
 8ca12c95-9f20-4121-b6c2-0d4e09f255d9  rack1
 UN  192.168.3.20  -142445 bytes  256 59.3%
 a1e2db2b-53a2-4cf7-8246-ea86104276d4  rack1
 UN  192.168.3.40  -180088 bytes  256 56.7%
 842107c0-91c6-4dbc-b6e4-81d04f3f492b  rack1
 UN  192.168.3.25  -123678 bytes  256 63.5%
 6a50a6e7-d3bb-4b3f-a3a7-cb442b26d188  rack1
 UN  192.168.3.24  -106578 bytes  256 59.3%
 71713907-b403-4590-8079-5b750ce0b8d4  rack1

 Did I do something wrong ?

 Thanks for any help,
 Jero

Re: Full GC in cassandra

[Mark Reddy]

What is your data size and number of columns in Cassandra. Do you do many
deletions?


On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha ruchir@gmail.com wrote:

 Really curious to know what's causing the spike in Columns and
 DeletedColums below :


 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class
 Histogram:
  num #instances #bytes  class name
 --
1: 132626060 6366050880  java.nio.HeapByteBuffer
2:  28194918 3920045528  [B
3:  78124737 3749987376
  edu.stanford.ppl.concurrent.SnapTreeMap$Node
 *   4:  67650128 2164804096
 2164804096  org.apache.cassandra.db.Column*
 *   5:  16315310  522089920  org.apache.cassandra.db.DeletedColumn*
6:  6818  392489608  [I
7:   2844374  273059904
  edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
8:   5727000  22908  java.util.TreeMap$Entry
9:767742  182921376  [J
   10:   2932832  140775936
  edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
   11:   2844375   9102
  edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
   12:   4145131   66322096
  java.util.concurrent.atomic.AtomicReference
   13:437874   64072392  [C
   14:   2660844   63860256
  java.util.concurrent.ConcurrentSkipListMap$Node
   15:  4920   62849864  [[B
   16:   1632063   52226016  edu.stanford.ppl.concurrent.SnapTreeMap

Re: Cassandra version upgrade path

[Mark Reddy]

Check https://github.com/apache/cassandra/blob/trunk/NEWS.txt to see the
steps required, if any, when upgrading.

As a general practice you should also reviw the changes at
https://github.com/apache/cassandra/blob/trunk/CHANGES.txt If upgrading
major versions it is often recommended to be on the latest version of your
current major version first before attempting an upgrade and test in a non
production environment beforehand.


Mark


On Sat, Jul 26, 2014 at 8:01 AM, Rahul Neelakantan ra...@rahul.be wrote:

 Is there a tool/guide/document that shows you upgrade paths for different
 versions of Cassandra?

 For example if you are on version X and want to go to version Y, here are
 the intermediate versions you need to upgrade to and here are some special
 precautions/Steps you need to take for so and so version while upgrading

 Rahul Neelakantan
 678-451-4545

Re: How to get rid of stale info in gossip

[Mark Reddy]

After removing a node, it's information can persist in the Gossiper for up
to 3 days, after which time it should be removed.

Are you having issues with a removed node state persisting for longer?


Mark


On Fri, Jul 25, 2014 at 11:33 AM, Rahul Neelakantan ra...@rahul.be wrote:

 Is there a way to get rid of stale information that shows up for
 removed/dead nodes in gossip, without a complete cluster bounce?

 Rahul Neelakantan

Re: do all nodes actually send the data to the coordinator when doing a read?

[Mark Reddy]

 However what's about timestamp checking ? You're saying that the
 coordinator checks for the digest of data (cell value) from both nodes but
 if the cell name have different timestamp would it still request a full
 data read to the node having the most recent time ?


When generating the hash to be returned to the coordinator, the possible
cell values that are used are name, value, timestamp, serialisationFlag and
depending on the cell type possibly other values. From there the hashes are
compared and if there is a mismatch the data is requested from the
replicas. At this stage the RowDataResolver will compute the most recent
version of each column, and send diffs to out-of-date replicas.




On Fri, Jul 25, 2014 at 11:32 PM, Jaydeep Chovatia 
chovatia.jayd...@gmail.com wrote:

 Yes. Digest includes following: {name, value, timestamp, flags(deleted,
 expired, etc.)}


 On Fri, Jul 25, 2014 at 2:33 PM, DuyHai Doan doanduy...@gmail.com wrote:

 Thanks Mark for the very detailed explanation.

  However what's about timestamp checking ? You're saying that the
 coordinator checks for the digest of data (cell value) from both nodes but
 if the cell name have different timestamp would it still request a full
 data read to the node having the most recent time ?


 On Fri, Jul 25, 2014 at 11:25 PM, Mark Reddy mark.re...@boxever.com
 wrote:

 Hi Brian,

 A read request will be handled in the following manner:

 Once the coordinator receives a read request it will firstly determine
 the replicas responsible for the data. From there those replicas are sorted
 by proximity to the coordinator. The closest node as determined by
 proximity sorting will be sent a command to perform an actual data read
 i.e. return the data to the coordinator

 If you have a Replication Factor (RF) of 3 and are reading at CL.QUORUM,
 one additional node will be sent a digest query. A digest query is like a
 read query except that instead of the receiving node actually returning the
 data, it only returns a digest (hash) of the would-be data. The reason for
 this is to discover whether the two nodes contacted agree on what the
 current data is, without sending the data over the network. Obviously for
 large data sets this is an effective bandwidth saver.

 Back on the coordinator node if the data and the digest match the data
 is returned to the client. If the data and digest do not match, a full data
 read is performed against the contacted replicas in order to guarantee that
 the most recent data is returned.

 Asynchronously in the background, the third replica is checked for
 consistency with the first two, and if needed, a read repair is initiated
 for that node.


 Mark



 On Fri, Jul 25, 2014 at 9:12 PM, Brian Tarbox briantar...@gmail.com
 wrote:

 We're considering a C* setup with very large columns and I have a
 question about the details of read.

 I understand that a read request gets handled by the coordinator which
 sends read requests to quorum of the nodes holding replicas of the data,
 and once quorum nodes have replied with consistent data it is returned to
 the client.

 My understanding is that each of the nodes actually sends the full data
 being requested to the coordinator (which in the case of very large columns
 would involve lots of network traffic).  Is that right?

 The alternative (which I don't think is the case but I've been asked to
 verify) is that the replicas first send meta-data to the coordinator which
 then asks one replica to send the actual data.  Again, I don't think this
 is the case but was asked to confirm.

 Thanks.

 --
 http://about.me/BrianTarbox

Re: What is C*?

[Mark Reddy]

Yes you are correct, Cassandra is often abbreviated as C*. With most
languages and applications being referenced by their acronym /
abbreviation, I guess you just have to pick one that is available. I assume
if someone creates a new language and wants to name it C*, they will see
that it is taken and avoid the conflict.


Mark


On Wed, Jul 23, 2014 at 11:50 PM, jcllings jclli...@gmail.com wrote:

 Keep seeing refs to C*.

 I assume that C* == Cassandra?  IMHO not a good ref to use what with C,
 C++, C#.  A language called C* can't be far behind assuming it doesn't
 already exist.
 ;-)

 Jim C.

Re: Case Study from Migrating from RDBMS to Cassandra

[Mark Reddy]

PlanetCassandra has a collection of migration use cases:

http://planetcassandra.org/mysql-to-cassandra-migration/
http://planetcassandra.org/oracle-to-cassandra-migration/

If they don't contain the information you need I'm sure you could reach out
to the companies involved or DataStax itself to get what you require.


Mark


On Wed, Jul 23, 2014 at 5:28 AM, Surbhi Gupta surbhi.gupt...@gmail.com
wrote:

 Thansk Shane, Howover i am looking for any Proof of Concepts kind of
 document .
 Does anybody has complete end to end document which contains the
 application overview,

 How they have migrated from RDBMS to Cassandra?
 What are the things to consider?
 How they have converted data model and after the new data model?
 How they have loaded the data into cassadnra ?
 Performance test after and before migartion etc.

 Thanks
 Surbhi

 On 23 July 2014 08:51, Shane Hansen shanemhan...@gmail.com wrote:

 There's lots of info on migrating from a relational database to Cassandra
 here:
 http://www.datastax.com/relational-database-to-nosql



 On Tue, Jul 22, 2014 at 7:45 PM, Surbhi Gupta surbhi.gupt...@gmail.com
 wrote:

 Hi,

 Does anybody has the case study for Migrating from RDBMS to Cassandra ?

 Thanks

Re: Why is the cassandra documentation such poor quality?

[Mark Reddy]

Hi Kevin,

The difference here is that the Apache Cassandra site is maintained by the
community whereas the DataStax site is maintained by paid employees with a
vested interest in producing documentation.

With DataStax having some comprehensive docs, I guess the desire for people
to maintain the Apache site has dwindled. However, if you are interested in
contributing to it and bringing it back up to standard you can, thus is the
freedom of open source.


Mark


On Wed, Jul 23, 2014 at 2:54 AM, Kevin Burton bur...@spinn3r.com wrote:

 This document:

 https://wiki.apache.org/cassandra/Operations

 … for example.  Is extremely out dated… does NOT reflect 2.x releases
 certainly.  Mentions commands that are long since removed/deprecated.

 Instead of giving bad documentation, maybe remove this and mark it as
 obsolete.

 The datastax documentation… is … acceptable I guess.  My main criticism
 there is that a lot of it it is in their blog.

 Kevin

 --

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog: http://burtonator.wordpress.com
 … or check out my Google+ profile
 https://plus.google.com/102718274791889610666/posts
 http://spinn3r.com

Re: When will a node's host ID change?

[Mark Reddy]

It won't. Cassandra stores a node's host ID in an end-point to host ID
mapping, each ID must be unique and cannot be changed after the fact.


Mark


On Mon, Jul 21, 2014 at 4:44 PM, John Sanda john.sa...@gmail.com wrote:

 Under what circumstances, if any, will a node's host ID change?

 - John

Re: Only one node in Cassandra AMI cluster

[Mark Reddy]

Have you setup your security group correctly so that all nodes can
communicate. Is there anything in the logs to suggest nodes could not
communicate to each other?

When you log into the other instances, is Cassandra running correctly?


On Tue, Jul 15, 2014 at 6:28 PM, Lars Schouw schou...@yahoo.com wrote:

 When I start Cassandra AMI cluster with 4 nodes on AWS and open the
 OpsCenter there is only one node shown. I followed the doc
 http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMILaunch.html
 I choose

 Region  AMI
 us-east-1   ami-ada2b6c4

 I set the number of Ec2 instances to 4 and advanced details to
 --clustername myDSCcluster --totalnodes 4 --version community
 [image: enter image description here]
 Then after launching and waiting a bit I can only see one node in the
 cluster

 ubuntu@ip-:~$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: us-east
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  XXX  59.24 KB   256 100.0%    1c

Re: Only one node in Cassandra AMI cluster

[Mark Reddy]

Good to hear.


Mark


On Tue, Jul 15, 2014 at 8:16 PM, Lars Schouw schou...@yahoo.com wrote:

 Yes, it works when setting the security group so all traffic is allowed
 from anywhere.

 I also found the doc that tells me how to add the more restrictive
 security
 http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMISecurityGroup.html





   On Tuesday, July 15, 2014 6:43 PM, Mark Reddy mark.re...@boxever.com
 wrote:


 Have you setup your security group correctly so that all nodes can
 communicate. Is there anything in the logs to suggest nodes could not
 communicate to each other?

 When you log into the other instances, is Cassandra running correctly?


 On Tue, Jul 15, 2014 at 6:28 PM, Lars Schouw schou...@yahoo.com wrote:

  When I start Cassandra AMI cluster with 4 nodes on AWS and open the
 OpsCenter there is only one node shown. I followed the doc
 http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installAMILaunch.html
  I choose

 Region  AMI
 us-east-1   ami-ada2b6c4

 I set the number of Ec2 instances to 4 and advanced details to
 --clustername myDSCcluster --totalnodes 4 --version community
 [image: enter image description here]
 Then after launching and waiting a bit I can only see one node in the
 cluster

 ubuntu@ip-:~$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: us-east
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  XXX  59.24 KB   256 100.0%    1c

Re: Small bug on Cassandra 2.1.0-rc3 and fix

[Mark Reddy]

Hi Carlos,

I have just downloaded 2.1.0-rc3 to validate this and the cassandra.in.sh
contains the correct reference to the jamm-0.2.6.jar

This can also been seen on the trunk:
https://github.com/apache/cassandra/blob/trunk/bin/cassandra.in.sh

I am unsure why you are seeing jamm-0.2.5.jar in your cassandra.in.sh file
if you have taken down a fresh 2.1.0-rc, can you give any further
information?


Mark


On Fri, Jul 11, 2014 at 9:49 PM, Carlos Scheidecker nando@gmail.com
wrote:

 Hello all,

 Probably not here that I should submit it, but I have been testing
 Cassandra 2.1.0 for quite a while.

 I have tried to subscribe to the dev list as well.

 Even now on rc3, there is still a small bug on cassandra.in.sh used,
 among others, for nodetool settings.

 The error is that it is still loading jamm-0.2.5.jar while the package has
 been updated to 0.2.6 on this version.

 Where it says:

 then
 JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar
 fi

 Should be inside cassandra.in.sh

 then
 JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.6.jar
 fi

 Otherwise, of course, nodetool will not start unless you have version
 0.2.5 from somewhere else in your classpath.

Re: UnavailableException

[Mark Reddy]

Can you post the output of nodetool status and your Astyanax connection
settings?


On Fri, Jul 11, 2014 at 9:06 PM, Ruchir Jha ruchir@gmail.com wrote:

 This is how we create our keyspace. We just ran this command once through
 a cqlsh session on one of the nodes, so don't quite understand what you
 mean by check that your DC names match up

 CREATE KEYSPACE prod WITH replication = {
   'class': 'NetworkTopologyStrategy',
   'datacenter1': '3'
 };



 On Fri, Jul 11, 2014 at 3:48 PM, Chris Lohfink clohf...@blackbirdit.com
 wrote:

 What replication strategy are you using? if using NetworkTopolgyStrategy
 double check that your DC names match up (case sensitive)

 Chris

 On Jul 11, 2014, at 9:38 AM, Ruchir Jha ruchir@gmail.com wrote:

 Here's the complete stack trace:

 com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException:
 TokenRangeOfflineException:
 [host=ny4lpcas5.fusionts.corp(10.10.20.47):9160, latency=22784(42874),
 attempts=3]UnavailableException()
 at
 com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165)
 at
 com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
 at
 com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
 at
 com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
 at
 com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
 at
 com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
 at
 com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:485)
 at
 com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:79)
 at
 com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:123)
 Caused by: UnavailableException()
 at
 org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20841)
 at
 org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
 at
 org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
 at
 org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
 at
 com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:129)
 at
 com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:126)
 at
 com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
 ... 12 more



 On Fri, Jul 11, 2014 at 9:11 AM, Prem Yadav ipremya...@gmail.com wrote:

 Please post the full exception.


 On Fri, Jul 11, 2014 at 1:50 PM, Ruchir Jha ruchir@gmail.com
 wrote:

 We have a 12 node cluster and we are consistently seeing this exception
 being thrown during peak write traffic. We have a replication factor of 3
 and a write consistency level of QUORUM. Also note there is no unusual Or
 Full GC activity during this time. Appreciate any help.

 Sent from my iPhone

Re: Small bug on Cassandra 2.1.0-rc3 and fix

[Mark Reddy]

This looks to have been fixed only yesterday:
https://github.com/apache/cassandra/commit/420c68511417e917925898e8c0637c9df3e317b0


On Fri, Jul 11, 2014 at 10:49 PM, Carlos Scheidecker nando@gmail.com
wrote:

 Mark,

 Sure!

 Actually it might be a package problem. Not installing Cassandra by hand
 as I need to test it across a bigger dev cluster.

 As I am using Ubuntu 14.04 on the test nodes using the following
 repository:

 ### Cassandra
 deb http://www.apache.org/dist/cassandra/debian 21x main # disabled on
 upgrade to saucy
 deb-src http://www.apache.org/dist/cassandra/debian 21x main # disabled
 on upgrade to saucy
 ### Cassandra

 Therefore, it seems to be a packaging problem as the
 file /usr/share/cassandra/cassandra.in.sh is the culprid.

 Here is its entire content after the fix:

 root@hp-dv7-7243cl:/etc/apt# cat /usr/share/cassandra/cassandra.in.sh

 # The directory where Cassandra's configs live (required)
 CASSANDRA_CONF=/etc/cassandra

 CASSANDRA_HOME=/usr/share/cassandra

 # The java classpath (required)
 if [ -n $CLASSPATH ]; then
 CLASSPATH=$CLASSPATH:$CASSANDRA_CONF
 else
 CLASSPATH=$CASSANDRA_CONF
 fi

 for jar in /usr/share/cassandra/lib/*.jar; do
 CLASSPATH=$CLASSPATH:$jar
 done

 for jar in /usr/share/cassandra/*.jar; do
 CLASSPATH=$CLASSPATH:$jar
 done

 CLASSPATH=$CLASSPATH:$EXTRA_CLASSPATH


 # set JVM javaagent opts to avoid warnings/errors
 if [ $JVM_VENDOR != OpenJDK -o $JVM_VERSION \ 1.6.0 ] \
   || [ $JVM_VERSION = 1.6.0 -a $JVM_PATCH_VERSION -ge 23 ]
 then
 JAVA_AGENT=$JAVA_AGENT -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.6.jar
 fi




 On Fri, Jul 11, 2014 at 3:02 PM, Mark Reddy mark.re...@boxever.com
 wrote:

 Hi Carlos,

 I have just downloaded 2.1.0-rc3 to validate this and the cassandra.in.sh
 contains the correct reference to the jamm-0.2.6.jar

 This can also been seen on the trunk:
 https://github.com/apache/cassandra/blob/trunk/bin/cassandra.in.sh

 I am unsure why you are seeing jamm-0.2.5.jar in your cassandra.in.sh
 file if you have taken down a fresh 2.1.0-rc, can you give any further
 information?


 Mark


 On Fri, Jul 11, 2014 at 9:49 PM, Carlos Scheidecker nando@gmail.com
 wrote:

 Hello all,

 Probably not here that I should submit it, but I have been testing
 Cassandra 2.1.0 for quite a while.

 I have tried to subscribe to the dev list as well.

 Even now on rc3, there is still a small bug on cassandra.in.sh used,
 among others, for nodetool settings.

 The error is that it is still loading jamm-0.2.5.jar while the package
 has been updated to 0.2.6 on this version.

 Where it says:

 then
 JAVA_AGENT=$JAVA_AGENT
 -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar
 fi

 Should be inside cassandra.in.sh

 then
 JAVA_AGENT=$JAVA_AGENT
 -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.6.jar
 fi

 Otherwise, of course, nodetool will not start unless you have version
 0.2.5 from somewhere else in your classpath.

Re: TTransportException (java.net.SocketException: Broken pipe)

[Mark Reddy]

Hi Bhaskar,

Can you check your limits using 'ulimit -a'? The default is 1024, which
needs to be increased if you have not done so already.

Here you will find a list of recommended production settings:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html


Mark

On Tue, Jul 8, 2014 at 5:30 AM, Bhaskar Singhal bhaskarsing...@yahoo.com
wrote:

 Hi,

 I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad
 core ubuntu server with 32gb ram) and trying to ingest 1MB values using
 cassandra-stress. It works fine for a while(1600secs) but after ingesting
 around 120GB data, I start getting the following error:
 Operation [70668] retried 10 times - error inserting key 0070668
 ((TTransportException): java.net.SocketException: Broken pipe)

 The cassandra server is still running but in the system.log I see the
 below mentioned errors.

 ERROR [COMMIT-LOG-ALLOCATOR] 2014-07-07 22:39:23,617 CassandraDaemon.java
 (line 198) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
 java.lang.NoClassDefFoundError:
 org/apache/cassandra/db/commitlog/CommitLog$4
 at
 org.apache.cassandra.db.commitlog.CommitLog.handleCommitError(CommitLog.java:374)
 at
 org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116)
 at
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at java.lang.Thread.run(Thread.java:744)
 Caused by: java.lang.ClassNotFoundException:
 org.apache.cassandra.db.commitlog.CommitLog$4
 at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
 ... 4 more
 Caused by: java.io.FileNotFoundException:
 /path/2.0.7/cassandra/build/classes/main/org/apache/cassandra/db/commitlog/CommitLog$4.class
 (Too many open files)
 at java.io.FileInputStream.open(Native Method)
 at java.io.FileInputStream.init(FileInputStream.java:146)
 at
 sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1086)
 at sun.misc.Resource.cachedInputStream(Resource.java:77)
 at sun.misc.Resource.getByteBuffer(Resource.java:160)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
 at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 ... 10 more
 ERROR [FlushWriter:7] 2014-07-07 22:39:24,924 CassandraDaemon.java (line
 198) Exception in thread Thread[FlushWriter:7,5,main]
 FSWriteError in
 /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db
 at
 org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
 at
 org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
 at
 org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
 at
 org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417)
 at
 org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350)
 at
 org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
 at
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
 Caused by: java.io.FileNotFoundException:
 /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db
 (Too many open files)
 at java.io.FileOutputStream.open(Native Method)
 at java.io.FileOutputStream.init(FileOutputStream.java:221)
 at java.io.FileOutputStream.init(FileOutputStream.java:110)
 at
 org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466)
 ... 9 more

 There are around 9685 open files by the Cassandra server process (using
 lsof), 3938 commit log segments in /cassandra/commitlog and around 572
 commit log segments deleted during the course of the test.

 I am wondering what is causing Cassandra to open so many files, is the
 flushing slow? or something else?

 I tried increasing the flush writers, but that didn't help.


 Regards,
 Bhaskar


 CREATE KEYSPACE Keyspace1 WITH replication = {
   'class': 'SimpleStrategy',
   'replication_factor': '1'
 };

 CREATE TABLE Standard1 (
   key blob,
   C0 blob,
   PRIMARY KEY (key)
 ) WITH COMPACT STORAGE AND
   

Re: Disable vnode

[Mark Reddy]

Hi Rameez,

I have never done a migration from vnodes to non-vnodes however I would
imagine that the procedure would be the same as its counterpart. As always
testing in dev should be done first.

To move from vnodes to non-vodes I would add a new datacenter to the
cluster with vnodes disabled and rebuild from your vnode cluster.

You can find some more details about adding a data center to your cluster
here:
http://datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html?scroll=task_ds_hmp_54q_gk__task_ds_hmp_54q_gk_unique_1



Mark



On Fri, Jul 4, 2014 at 2:43 PM, Rameez Thonnakkal ssram...@gmail.com
wrote:

 hello Team,

 I am looking for standard operating procedure to disable vnode in a
 production cluster.
 This is to enable solr which doesn't work with a cassandra cluster having
 vnode enabled.

 Any suggestions/

 Thanks,
 Rameez

Re: why aren't my nodes show same percentage of ownership

[Mark Reddy]

You should run nodetool ring and specify a keyspace, otherwise the
ownership information will be nonsense.

https://issues.apache.org/jira/browse/CASSANDRA-7173


Mark

On Mon, Jun 30, 2014 at 6:13 PM, S C as...@outlook.com wrote:

 When I run nodetoool ring I see the ownership with different
 percentages. However, LOAD column shows not a huge deviation. Why is that?
 I am using Datastax 3.0.

 http://pastebin.com/EcWbZn26

Re: Index with same Name but different keyspace

[Mark Reddy]

Can you share your schema and the commands you are running?


On Thu, May 15, 2014 at 7:54 PM, mahesh rajamani
rajamani.mah...@gmail.comwrote:

 Hi,

 I am using Cassandra 2.0.5 version. I trying to setup 2 keyspace with same
 tables for different testing. While creating index on the tables, I
 realized I am not able to use the same index name  though the tables are in
 different keyspaces. Is maintaining unique index name across keyspace is
 must/feature?

 --
 Regards,
 Mahesh Rajamani

Re: Mutation messages dropped

[Mark Reddy]

Yes, please see http://wiki.apache.org/cassandra/FAQ#dropped_messages for
further details.


Mark


On Fri, May 9, 2014 at 12:52 PM, Raveendran, Varsha IN BLR STS 
varsha.raveend...@siemens.com wrote:

  Hello,

 I am writing around 10Million records continuously into a single node
 Cassandra (2.0.5) .
 In the Cassandra log file I see an entry “*272 MUTATION messages dropped
 in last 5000ms*” . Does this mean that 272 records were not written
 successfully?

 Thanks,
 Varsha

Re: Can the seeds list be changed at runtime?

[Mark Reddy]

Hi Boying,

From Datastax documentation:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/architecture/architectureGossipAbout_c.html

 The seed node designation has no purpose other than bootstrapping the
 gossip process for new nodes joining the cluster. Seed nodes are not a
 single point of failure, nor do they have any other special purpose in
 cluster operations beyond the bootstrapping of nodes.


For this reason you can change the seed list on existing node at any time,
as the node itself will already be aware of the cluster and would not need
to rely on the seed list to join. For new nodes that you want to bootstrap
into the cluster you can specify any nodes you wish.


Mark


On Tue, Apr 29, 2014 at 2:57 AM, Lu, Boying boying...@emc.com wrote:

 Hi, All,



 I wonder if I can change the seeds list at runtime. i.e. without change
 the yaml file and restart DB service?



 Thanks



 Boying

Re: JDK 8

[Mark Reddy]

 Datastax recommended during a long time (still now ?) to use Java 6


Java 6 is recommended for version 1.2
Java 7 is required for version 2.0


Mark


On Tue, Apr 29, 2014 at 10:19 AM, Alain RODRIGUEZ arodr...@gmail.comwrote:

 Looks like it will be like with the version 7... Cassandra has been
 compatible with this version for a long time, but there were no official
 validations and Datastax recommended during a long time (still now ?) to
 use Java 6.

 The best thing would be to use older versions. If for some reason you use
 Java 8, run some tests and let us know how things goes :).

 Good luck with this.


 2014-04-29 1:09 GMT+02:00 Colin co...@clark.ws:

 It seems to run ok, but I havent seen it yet in production on 8.

 --
 *Colin Clark*
 +1-320-221-9531


 On Apr 28, 2014, at 4:01 PM, Ackerman, Mitchell 
 mitchell.acker...@pgi.com wrote:

  I’ve been searching around, but cannot find any information as to
 whether Cassandra runs on JRE 8.  Any information on that?



 Thanks, Mitchell

Re: Bootstrap Timing

[Mark Reddy]

If all streams have completed, the node could be still rebuilding secondary
indexes? Try looking at 'nodetool compactionstats' for this.


On Fri, Apr 18, 2014 at 1:05 PM, Phil Burress philburress...@gmail.comwrote:

 nodetool netstats shows 84 files. They are all at 100%. Nothing showing in
 Pending or Active for Read Repair Stats.

 I'm assuming this means it's done. But it still shows JOINING. Is there
 an undocumented step I'm missing here? This whole process seems broken to
 me.


 On Thu, Apr 17, 2014 at 4:32 PM, Robert Coli rc...@eventbrite.com wrote:

 On Wed, Apr 16, 2014 at 1:56 PM, Phil Burress 
 philburress...@gmail.comwrote:

 I've shut down two of the nodes and am bootstrapping one right now. Is
 there any way to tell when it will finish bootstrapping?


 nodetool netstats will show the progress of the streams involved, which
 could help you estimate.

 =Rob

Re: How to clear all data using CQL?

[Mark Reddy]

To get a list of all keyspaces via CQL you can run:

*describe keyspaces *or *select keyspace_name from system.schema_keyspaces;*


On Wed, Apr 16, 2014 at 9:23 AM, Sebastian Schmidt isib...@gmail.comwrote:

 Hey

 I want to do unit tests for software that uses Cassandra as storage
 backend. Using the Hector client, I could just iterate over all
 keyspaces and delete them. In CQL, there is no query to list all
 keyspaces. How can I drop all Keyspaces from Cassandra, without knowing
 every single Keyspace name?

 Regards,
 Sebastian

Re: Unable to complete request: one or more nodes were unavailable.

[Mark Reddy]

 create keyspace twitter with replication = {'class':'SimpleStrategy',
 'replication_factor' : 3}


Your replication factor is your issue here, you have a single node and a
RF=3. For a single node setup your RF should be 1. You can find more info
about replication here:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html


On Wed, Apr 16, 2014 at 1:44 PM, Vivek Mishra mishra.v...@gmail.com wrote:

 Hi,
 Mine is a simple case. Running on single node only. Keyspace is:

 create keyspace twitter with replication = {'class':'SimpleStrategy',
 'replication_factor' : 3}

 -Vivek


 On Wed, Apr 16, 2014 at 1:27 AM, Tupshin Harper tups...@tupshin.comwrote:

 Please provide your keyspace definition,  and the output of nodetool
 ring

 -Tupshin
 On Apr 15, 2014 3:52 PM, Vivek Mishra mishra.v...@gmail.com wrote:

 Hi,
 I am trying Cassandra light weight transaction support with Cassandra
 2.0.4

 cqlsh:twitter create table user(user_id text primary key, namef text);
 cqlsh:twitter insert into user(user_id,namef) values('v','ff') if not
 exists;

 *Unable to complete request: one or more nodes were unavailable.*

 Any suggestions?

 -Vivek

Re: Unable to complete request: one or more nodes were unavailable.

[Mark Reddy]

The Paxos protocol used for CAS operations will always use at least a
consistency level effectively equivalent to QUORUM (called SERIAL) when
writing, even if you explicitly specify a lower level, e.g. ANY or ONE.
Setting consistency level to ALL will make the write execute on all
replicas if the condition is met, but the comparison itself is executed
against a QUORUM number of nodes. As a result, a write operation with ALL
consistency level that fails to meet the specified check may not throw an
Exception, even if some replica nodes are not accessible.



On Wed, Apr 16, 2014 at 2:00 PM, Tupshin Harper tups...@tupshin.com wrote:

 No, but you do need a quorum of nodes.


 http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html

 SERIAL
 A write must be written conditionally to the commit log and memory table
 on a quorum of replica nodes.

 Used to achievelinearizable 
 consistencyhttp://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_tunable_consistency_c.html#concept_ds_f4h_hwx_zjfor
 lightweight transactions by preventing unconditional updates.
 On Apr 16, 2014 5:56 AM, Vivek Mishra mishra.v...@gmail.com wrote:

 Thanks Mark. does this mean with RF=3, all 3 nodes must be up and running
 for CAS updates?

 -Vivek


 On Wed, Apr 16, 2014 at 6:22 PM, Mark Reddy mark.re...@boxever.comwrote:

 create keyspace twitter with replication = {'class':'SimpleStrategy',
 'replication_factor' : 3}


 Your replication factor is your issue here, you have a single node and a
 RF=3. For a single node setup your RF should be 1. You can find more info
 about replication here:
 http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html


 On Wed, Apr 16, 2014 at 1:44 PM, Vivek Mishra mishra.v...@gmail.comwrote:

 Hi,
 Mine is a simple case. Running on single node only. Keyspace is:

 create keyspace twitter with replication = {'class':'SimpleStrategy',
 'replication_factor' : 3}

 -Vivek


 On Wed, Apr 16, 2014 at 1:27 AM, Tupshin Harper tups...@tupshin.comwrote:

 Please provide your keyspace definition,  and the output of nodetool
 ring

 -Tupshin
 On Apr 15, 2014 3:52 PM, Vivek Mishra mishra.v...@gmail.com wrote:

 Hi,
 I am trying Cassandra light weight transaction support with Cassandra
 2.0.4

 cqlsh:twitter create table user(user_id text primary key, namef
 text);
 cqlsh:twitter insert into user(user_id,namef) values('v','ff') if
 not exists;

 *Unable to complete request: one or more nodes were unavailable.*

 Any suggestions?

 -Vivek

Re: GC histogram analysis

[Mark Reddy]

Do you delete and/or set TTLs on your data?


On Wed, Apr 16, 2014 at 2:14 PM, Ruchir Jha ruchir@gmail.com wrote:

 Hi,

 I am trying to investigate ParNew promotion failures happening routinely
 in production. As part of this exercise, I enabled
 -XX:PrintHistogramBeforeFullGC and saw the following output. As you can see
 there are a ton of Columns, ExpiringColumns and DeletedColumns before GC
 ran and these numbers go down significantly right after GC. Why are there
 so many expiring and deleted columns?



 *Before GC:* num #instances #bytes  class name
 --
1: 113539896 5449915008  java.nio.*HeapByteBuffer*
2:  15979061 2681431488  [B
3:  36364545 1745498160
 edu.stanford.ppl.concurrent.SnapTreeMap$Node
4:  23583282  754665024  org.apache.cassandra.db.*Column*
5:   8745428  209890272
 java.util.concurrent.ConcurrentSkipListMap$Node
6:   5062619  202504760  org.apache.cassandra.db.
 *ExpiringColumn*
7: 45261  198998216  [I
8:   1801535  172947360
 edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
9:   1473677  169570040  [J
   10:   4713304  113119296  java.lang.Double
   11:   3246729  103895328  org.apache.cassandra.db.
 *DeletedColumn*

 *After GC:*
 num #instances #bytes  class name
 --
 1:  11807204 1505962728  [B
 2:  12525536  601225728  java.nio.*HeapByteBuffer*
 3:   8839073  424275504
 edu.stanford.ppl.concurrent.SnapTreeMap$Node
 4:   8194496  262223872  org.apache.cassandra.db.*Column*
 cache.KeyCacheKey
 17:432119   17284760  org.apache.cassandra.db.*ExpiringColumn*
 21:351096   11235072  org.apache.cassandra.db.*DeletedColumn*

Re: Datastax OPSCENTER

[Mark Reddy]

Datastax have a comprehensive installation and configuration guide here:
http://www.datastax.com/documentation/opscenter/4.1/opsc/install/opscInstallation_g.html


On Tue, Apr 15, 2014 at 9:13 AM, Aravindan T aravinda...@tcs.com wrote:

 Hi,

 Could anyone please guide me on installation procedure of Cassandra using
 Datastax OPSCENTER? I couldnt find any documentation in regards to the same.

 Your help is much appreciated.


 Thanks,
 AT

 =-=-=
 Notice: The information contained in this e-mail
 message and/or attachments to it may contain
 confidential or privileged information. If you are
 not the intended recipient, any dissemination, use,
 review, distribution, printing or copying of the
 information contained in this e-mail message
 and/or attachments to it are strictly prohibited. If
 you have received this communication in error,
 please notify us by reply e-mail or telephone and
 immediately and permanently delete the message
 and any attachments. Thank you

Re: bloom filter + suddenly smaller CF

[Mark Reddy]

Hi Will,

You can run 'nodetool upgradesstables', this will rewrite the SSTables and
regenerate the bloom filters for those tables, This will reduce their
usage.


Mark


On Mon, Apr 14, 2014 at 3:16 PM, William Oberman
ober...@civicscience.comwrote:

 Ah, so I could change the chance value to poke it.  Good to know!



 On Mon, Apr 14, 2014 at 10:12 AM, Michal Michalski 
 michal.michal...@boxever.com wrote:

 Sorry, I misread the question - I thought you've also changed FP chance
 value, not only removed the data.

 Kind regards,
 Michał Michalski,
 michal.michal...@boxever.com


 On 14 April 2014 15:07, Michal Michalski michal.michal...@boxever.comwrote:

 Did you set Bloom Filter's FP chance before or after the step 3) above?
 If you did it before, C* should build Bloom Filters properly. If not -
 that's the reason.

 Kind regards,
 Michał Michalski,
 michal.michal...@boxever.com


 On 14 April 2014 15:04, William Oberman ober...@civicscience.comwrote:

 I didn't cross link my thread, but the basic idea is I've done:

 1.) Process that deleted ~900M of ~1G rows from a CF
 2.) Set GCGraceSeconds to 0 on CF
 3.) Run nodetool compact on all N nodes

 And I checked, and all N nodes have bloom filters using 1.5 +/- .2 GB
 of RAM (I didn't explicitly write down the before numbers, but they seem
 about the same) .  So, compaction didn't change the BF's (unless cassandra
 needs a 2nd compaction to see all of the data cleared by the 1st
 compaction).

 will


 On Mon, Apr 14, 2014 at 9:52 AM, Michal Michalski 
 michal.michal...@boxever.com wrote:

 Bloom filters are built on creation / rebuild of SSTable. If you
 removed the data, but the old SSTables weren't compacted or you didn't
 rebuild them manually, bloom filters will stay the same size.

 M.

 Kind regards,
 Michał Michalski,
 michal.michal...@boxever.com


 On 14 April 2014 14:44, William Oberman ober...@civicscience.comwrote:

 I had a thread on this forum about clearing junk from a CF.  In my
 case, it's ~90% of ~1 billion rows.

 One side effect I had hoped for was a reduction in the size of the
 bloom filter.  But, according to nodetool cfstats, it's still fairly 
 large
 (~1.5GB of RAM).

 Do bloom filters ever resize themselves when the CF suddenly gets
 smaller?

 My next test will be restarting one of the instances, though I'll
 have to wait on that operation so I thought I'd ask in the meantime.

 will

Re: Cassandra disk usage

[Mark Reddy]

What are you storing these 15 chars as; string, int, double, etc.? 15 chars
does not translate to 15 bytes.

You may be mixing up replication factor and quorum when you say Cassandra
cluster has 3 servers, and data is stored in quorum ( 2 servers ). You
read and write at quorum (N/2)+1 where N=total_number_of_nodes and your
data is replicated to the number of nodes you specify in your replication
factor. Could you clarify?

Also if you are concerned about disk usage, why are you storing the same 15
char value in both the column name and value? You could just store it as
the name and half your data usage :)




On Sun, Apr 13, 2014 at 4:26 PM, Yulian Oifa oifa.yul...@gmail.com wrote:

 I have column family with 2 raws.
 2 raws have overall 100 million columns.
 Each columns have name of 15 chars ( digits ) and same 15 chars in value (
 also digits ).
 Each column should have 30 bytes.
 Therefore all data should contain approximately 3GB.
 Cassandra cluster has 3 servers , and data is stored in quorum ( 2 servers
 ).
 Therefore each server should have 3GB*2/3=2GB of data for this column
 family.
 Table is almost never changed , data is only removed from this table ,
 which possibly created tombstones , but it should not increase the usage.
 However when i check the data i see that each server has more then 4GB of
 data ( more then twice of what should be).

 server 1:
 -rw-r--r-- 1 root root 3506446057 Dec 26 12:02 freeNumbers-g-264-Data.db
 -rw-r--r-- 1 root root  814699666 Dec 26 12:24 freeNumbers-g-281-Data.db
 -rw-r--r-- 1 root root  198432466 Dec 26 12:27 freeNumbers-g-284-Data.db
 -rw-r--r-- 1 root root   35883918 Apr 12 20:07 freeNumbers-g-336-Data.db

 server 2:
 -rw-r--r-- 1 root root 3448432307 Dec 26 11:57 freeNumbers-g-285-Data.db
 -rw-r--r-- 1 root root  762399716 Dec 26 12:22 freeNumbers-g-301-Data.db
 -rw-r--r-- 1 root root  220887062 Dec 26 12:23 freeNumbers-g-304-Data.db
 -rw-r--r-- 1 root root   54914466 Dec 26 12:26 freeNumbers-g-306-Data.db
 -rw-r--r-- 1 root root   53639516 Dec 26 12:26 freeNumbers-g-305-Data.db
 -rw-r--r-- 1 root root   53007967 Jan  8 15:45 freeNumbers-g-314-Data.db
 -rw-r--r-- 1 root root 413717 Apr 12 18:33 freeNumbers-g-359-Data.db


 server 3:
 -rw-r--r-- 1 root root 4490657264 Apr 11 18:20 freeNumbers-g-358-Data.db
 -rw-r--r-- 1 root root 389171 Apr 12 20:58 freeNumbers-g-360-Data.db
 -rw-r--r-- 1 root root   4276 Apr 11 18:20
 freeNumbers-g-358-Statistics.db
 -rw-r--r-- 1 root root   4276 Apr 11 18:24
 freeNumbers-g-359-Statistics.db
 -rw-r--r-- 1 root root   4276 Apr 12 20:58
 freeNumbers-g-360-Statistics.db
 -rw-r--r-- 1 root root976 Apr 11 18:20 freeNumbers-g-358-Filter.db
 -rw-r--r-- 1 root root208 Apr 11 18:24 freeNumbers-g-359-Data.db
 -rw-r--r-- 1 root root 78 Apr 11 18:20 freeNumbers-g-358-Index.db
 -rw-r--r-- 1 root root 52 Apr 11 18:24 freeNumbers-g-359-Index.db
 -rw-r--r-- 1 root root 52 Apr 12 20:58 freeNumbers-g-360-Index.db
 -rw-r--r-- 1 root root 16 Apr 11 18:24 freeNumbers-g-359-Filter.db
 -rw-r--r-- 1 root root 16 Apr 12 20:58 freeNumbers-g-360-Filter.db

 When i try to compact i get the following notification from first server :
 INFO [CompactionExecutor:1604] 2014-04-13 18:23:07,260
 CompactionController.java (line 146) Compacting large row
 USER_DATA/freeNumbers:8bdf9678-6d70-11e3-85ab-80e385abf85d (4555076689
 bytes) incrementally

 Which confirms that there is around 4.5GB of data on that server only.
 Why does cassandra takes so much data???

 Best regards
 Yulian Oifa

Re: Cassandra disk usage

[Mark Reddy]

 i I will change the data i am storing to decrease the usage , in value i
 will find some small value to store.Previously i used same value since this
 table is index only for search purposed and does not really has value.


If you don't need a value, you don't have to store anything. You can store
the column name and leave the value empty, this is a common practice.

1) What should be recommended read and write consistency and replication
 factor for 3 nodes with option of future increase server numbers?


Both consistency level and replication factor are tuneable depending on
your application constraints. I'd say a CL or quorum and RF of 3 is the
general practice.

Still it has 1.5X of overall data how can this be resolved and what is
 reason for that?


As Michał pointed out there is a 15 byte column overhead to consider here,
where:

total_column_size = column_name_size + column_value_size + 15


This link might shed some light on this:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/architecture/architecturePlanningUserData_t.html

Also i see that data is in different size on all nodes , does that means
 that servers are out of sync


How much is it out by? Data size may differ due to deletes, as you
mentioned you do deletes. What is the output of 'nodetool ring'?


On Sun, Apr 13, 2014 at 6:42 PM, Michal Michalski 
michal.michal...@boxever.com wrote:

  Each columns have name of 15 chars ( digits ) and same 15 chars in
 value ( also digits ).
  Each column should have 30 bytes.

 Remember about the standard Cassandra's column overhead which is, as far
 as I remember, 15 bytes, so it's 45 bytes in total - 50% more than you
 estimated, which kind of matches your 3 GB vs 4.5 GB case.

 There's also a per-row overhead, but I'm not sure about its size in
 current C* versions - I remember it was about 25 bytes or so some time ago,
 but it's not important in your case.

 Kind regards,
 Michał Michalski,
 michal.michal...@boxever.com


 On 13 April 2014 17:48, Yulian Oifa oifa.yul...@gmail.com wrote:

 Hello Mark and thanks for you reply.
 1) i store is as UTF8String.All digits are from 0x30 to 0x39 and should
 take 1 byte each digit. Since all characters are digits it should have 15
 bytes.
 2) I will change the data i am storing to decrease the usage , in value i
 will find some small value to store.Previously i used same value since this
 table is index only for search purposed and does not really has value.
 3) You are right i read and write in quorum and it was my mistake ( i
 though that if i write in quorum then data will be written to 2 nodes only).
 If i check the keyspace
 create keyspace USER_DATA
   with placement_strategy = 'NetworkTopologyStrategy'
   and strategy_options = [{19 : 3}]
   and durable_writes = true;

 it has replication factor of 3.
 Therefore i have several questions
 1) What should be recommended read and write consistency and replication
 factor for 3 nodes with option of future increase server numbers?
 2) Still it has 1.5X of overall data how can this be resolved and what is
 reason for that?
 3) Also i see that data is in different size on all nodes , does that
 means that servers are out of sync???

 Thanks and best regards
 Yulian Oifa


 On Sun, Apr 13, 2014 at 7:03 PM, Mark Reddy mark.re...@boxever.comwrote:

 What are you storing these 15 chars as; string, int, double, etc.? 15
 chars does not translate to 15 bytes.

 You may be mixing up replication factor and quorum when you say Cassandra
 cluster has 3 servers, and data is stored in quorum ( 2 servers ). You
 read and write at quorum (N/2)+1 where N=total_number_of_nodes and your
 data is replicated to the number of nodes you specify in your replication
 factor. Could you clarify?

 Also if you are concerned about disk usage, why are you storing the same
 15 char value in both the column name and value? You could just store it as
 the name and half your data usage :)




 On Sun, Apr 13, 2014 at 4:26 PM, Yulian Oifa oifa.yul...@gmail.comwrote:

 I have column family with 2 raws.
 2 raws have overall 100 million columns.
 Each columns have name of 15 chars ( digits ) and same 15 chars in
 value ( also digits ).
 Each column should have 30 bytes.
 Therefore all data should contain approximately 3GB.
 Cassandra cluster has 3 servers , and data is stored in quorum ( 2
 servers ).
 Therefore each server should have 3GB*2/3=2GB of data for this column
 family.
 Table is almost never changed , data is only removed from this table ,
 which possibly created tombstones , but it should not increase the usage.
 However when i check the data i see that each server has more then 4GB
 of data ( more then twice of what should be).

 server 1:
 -rw-r--r-- 1 root root 3506446057 Dec 26 12:02 freeNumbers-g-264-Data.db
 -rw-r--r-- 1 root root  814699666 Dec 26 12:24 freeNumbers-g-281-Data.db
 -rw-r--r-- 1 root root  198432466 Dec 26 12:27 freeNumbers-g-284-Data.db
 -rw-r--r-- 1 root root   35883918 Apr 12 20:07

Re: clearing tombstones?

[Mark Reddy]

Correct, a tombstone will only be removed after gc_grace period has
elapsed. The default value is set to 10 days which allows a great deal of
time for consistency to be achieved prior to deletion. If you are
operationally confident that you can achieve consistency via anti-entropy
repairs within a shorter period you can always reduce that 10 day interval.


Mark


On Fri, Apr 11, 2014 at 3:16 PM, William Oberman
ober...@civicscience.comwrote:

 I'm seeing a lot of articles about a dependency between removing
 tombstones and GCGraceSeconds, which might be my problem (I just checked,
 and this CF has GCGraceSeconds of 10 days).


 On Fri, Apr 11, 2014 at 10:10 AM, tommaso barbugli tbarbu...@gmail.comwrote:

 compaction should take care of it; for me it never worked so I run
 nodetool compaction on every node; that does it.


 2014-04-11 16:05 GMT+02:00 William Oberman ober...@civicscience.com:

 I'm wondering what will clear tombstoned rows?  nodetool cleanup,
 nodetool repair, or time (as in just wait)?

 I had a CF that was more or less storing session information.  After
 some time, we decided that one piece of this information was pointless to
 track (and was 90%+ of the columns, and in 99% of those cases was ALL
 columns for a row).   I wrote a process to remove all of those columns
 (which again in a vast majority of cases had the effect of removing the
 whole row).

 This CF had ~1 billion rows, so I expect to be left with ~100m rows.
  After I did this mass delete, everything was the same size on disk (which
 I expected, knowing how tombstoning works).  It wasn't 100% clear to me
 what to poke to cause compactions to clear the tombstones.  First I tried
 nodetool cleanup on a candidate node.  But, afterwards the disk usage was
 the same.  Then I tried nodetool repair on that same node.  But again, disk
 usage is still the same.  The CF has no snapshots.

 So, am I misunderstanding something?  Is there another operation to try?
  Do I have to just wait?  I've only done cleanup/repair on one node.  Do
 I have to run one or the other over all nodes to clear tombstones?

 Cassandra 1.2.15 if it matters,

 Thanks!

 will

Re: clearing tombstones?

[Mark Reddy]

Yes, running nodetool compact (major compaction) creates one large SSTable.
This will mess up the heuristics of the SizeTiered strategy (is this the
compaction strategy you are using?) leading to multiple 'small' SSTables
alongside the single large SSTable, which results in increased read
latency. You will incur the operational overhead of having to manage
compactions if you wish to compact these smaller SSTables. For all these
reasons it is generally advised to stay away from running compactions
manually.

Assuming that this is a production environment and you want to keep
everything running as smoothly as possible I would reduce the gc_grace on
the CF, allow automatic minor compactions to kick in and then increase the
gc_grace once again after the tombstones have been removed.


On Fri, Apr 11, 2014 at 3:44 PM, William Oberman
ober...@civicscience.comwrote:

 So, if I was impatient and just wanted to make this happen now, I could:

 1.) Change GCGraceSeconds of the CF to 0
 2.) run nodetool compact (*)
 3.) Change GCGraceSeconds of the CF back to 10 days

 Since I have ~900M tombstones, even if I miss a few due to impatience, I
 don't care *that* much as I could re-run my clean up tool against the now
 much smaller CF.

 (*) A long long time ago I seem to recall reading advice about don't ever
 run nodetool compact, but I can't remember why.  Is there any bad long
 term consequence?  Short term there are several:
 -a heavy operation
 -temporary 2x disk space
 -one big SSTable afterwards
 But moving forward, everything is ok right?  CommitLog/MemTable-SStables,
 minor compactions that merge SSTables, etc...  The only flaw I can think of
 is it will take forever until the SSTable minor compactions build up enough
 to consider including the big SSTable in a compaction, making it likely
 I'll have to self manage compactions.



 On Fri, Apr 11, 2014 at 10:31 AM, Mark Reddy mark.re...@boxever.comwrote:

 Correct, a tombstone will only be removed after gc_grace period has
 elapsed. The default value is set to 10 days which allows a great deal of
 time for consistency to be achieved prior to deletion. If you are
 operationally confident that you can achieve consistency via anti-entropy
 repairs within a shorter period you can always reduce that 10 day interval.


 Mark


 On Fri, Apr 11, 2014 at 3:16 PM, William Oberman 
 ober...@civicscience.com wrote:

 I'm seeing a lot of articles about a dependency between removing
 tombstones and GCGraceSeconds, which might be my problem (I just checked,
 and this CF has GCGraceSeconds of 10 days).


 On Fri, Apr 11, 2014 at 10:10 AM, tommaso barbugli 
 tbarbu...@gmail.comwrote:

 compaction should take care of it; for me it never worked so I run
 nodetool compaction on every node; that does it.


 2014-04-11 16:05 GMT+02:00 William Oberman ober...@civicscience.com:

 I'm wondering what will clear tombstoned rows?  nodetool cleanup,
 nodetool repair, or time (as in just wait)?

 I had a CF that was more or less storing session information.  After
 some time, we decided that one piece of this information was pointless to
 track (and was 90%+ of the columns, and in 99% of those cases was ALL
 columns for a row).   I wrote a process to remove all of those columns
 (which again in a vast majority of cases had the effect of removing the
 whole row).

 This CF had ~1 billion rows, so I expect to be left with ~100m rows.
  After I did this mass delete, everything was the same size on disk (which
 I expected, knowing how tombstoning works).  It wasn't 100% clear to me
 what to poke to cause compactions to clear the tombstones.  First I tried
 nodetool cleanup on a candidate node.  But, afterwards the disk usage was
 the same.  Then I tried nodetool repair on that same node.  But again, 
 disk
 usage is still the same.  The CF has no snapshots.

 So, am I misunderstanding something?  Is there another operation to
 try?  Do I have to just wait?  I've only done cleanup/repair on one 
 node.
  Do I have to run one or the other over all nodes to clear tombstones?

 Cassandra 1.2.15 if it matters,

 Thanks!

 will

Re: clearing tombstones?

[Mark Reddy]

To clarify, you would want to manage compactions only if you were concerned
about read latency. If you update rows, those rows may become spread across
an increasing number of SSTables leading to increased read latency.

Thanks for providing some insight into your use case as it does differ from
the norm. If you would consider 50GB a small CF and your data ingestion
sufficient enough to result in more SSTables of similar size soon, yes you
could run a major compaction will little operational overhead and the
compaction strategies heuristics would level out after some time.


On Fri, Apr 11, 2014 at 4:52 PM, Laing, Michael
michael.la...@nytimes.comwrote:

 I have played with this quite a bit and recommend you set gc_grace_seconds
 to 0 and use 'nodetool compact [keyspace] [cfname]' on your table.

 A caveat I have is that we use C* 2.0.6 - but the space we expect to
 recover is in fact recovered.

 Actually, since we never delete explicitly (just ttl) we always have
 gc_grace_seconds set to 0.

 Another important caveat is to be careful with repair: having set gc to 0
 and compacted on a node, if you then repair it, data may come streaming in
 from the other nodes. We don't run into this, as our gc is always 0, but
 others may be able to comment.

 ml


 On Fri, Apr 11, 2014 at 11:26 AM, William Oberman 
 ober...@civicscience.com wrote:

 Yes, I'm using SizeTiered.

 I totally understand the mess up the heuristics issue.  But, I don't
 understand You will incur the operational overhead of having to manage
 compactions if you wish to compact these smaller SSTables.  My
 understanding is the small tables will still compact.  The problem is that
 until I have 3 other (by default) tables of the same size as the big
 table, it won't be compacted.

 In my case, this might not be terrible though, right?  To get into the
 trees, I have 9 nodes with RF=3 and this CF is ~500GB/node.  I deleted like
 90-95% of the data, so I expect the data to be 25-50GB after the tombstones
 are cleared, but call it 50GB.  That means I won't compact this 50GB file
 until I gather another 150GB (50,50,50,50-200).   But, that's not
 *horrible*.  Now, if I only deleted 10% of the data, waiting to compact
 450GB until I had another 1.3TB would be rough...

 I think your advice is great for people looking for normal answers in
 the forum, but I don't think my use case is very normal :-)

 will

 On Fri, Apr 11, 2014 at 11:12 AM, Mark Reddy mark.re...@boxever.comwrote:

 Yes, running nodetool compact (major compaction) creates one large
 SSTable. This will mess up the heuristics of the SizeTiered strategy (is
 this the compaction strategy you are using?) leading to multiple 'small'
 SSTables alongside the single large SSTable, which results in increased
 read latency. You will incur the operational overhead of having to manage
 compactions if you wish to compact these smaller SSTables. For all these
 reasons it is generally advised to stay away from running compactions
 manually.

 Assuming that this is a production environment and you want to keep
 everything running as smoothly as possible I would reduce the gc_grace on
 the CF, allow automatic minor compactions to kick in and then increase the
 gc_grace once again after the tombstones have been removed.


 On Fri, Apr 11, 2014 at 3:44 PM, William Oberman 
 ober...@civicscience.com wrote:

 So, if I was impatient and just wanted to make this happen now, I
 could:

 1.) Change GCGraceSeconds of the CF to 0
 2.) run nodetool compact (*)
 3.) Change GCGraceSeconds of the CF back to 10 days

 Since I have ~900M tombstones, even if I miss a few due to impatience,
 I don't care *that* much as I could re-run my clean up tool against the now
 much smaller CF.

 (*) A long long time ago I seem to recall reading advice about don't
 ever run nodetool compact, but I can't remember why.  Is there any bad
 long term consequence?  Short term there are several:
 -a heavy operation
 -temporary 2x disk space
 -one big SSTable afterwards
 But moving forward, everything is ok right?
  CommitLog/MemTable-SStables, minor compactions that merge SSTables,
 etc...  The only flaw I can think of is it will take forever until the
 SSTable minor compactions build up enough to consider including the big
 SSTable in a compaction, making it likely I'll have to self manage
 compactions.



 On Fri, Apr 11, 2014 at 10:31 AM, Mark Reddy mark.re...@boxever.comwrote:

 Correct, a tombstone will only be removed after gc_grace period has
 elapsed. The default value is set to 10 days which allows a great deal of
 time for consistency to be achieved prior to deletion. If you are
 operationally confident that you can achieve consistency via anti-entropy
 repairs within a shorter period you can always reduce that 10 day 
 interval.


 Mark


 On Fri, Apr 11, 2014 at 3:16 PM, William Oberman 
 ober...@civicscience.com wrote:

 I'm seeing a lot of articles about a dependency between removing
 tombstones and GCGraceSeconds, which

Re: clearing tombstones?

[Mark Reddy]

Thats great Will, if you could update the thread with the actions you
decide to take and the results that would be great.


Mark


On Fri, Apr 11, 2014 at 5:53 PM, William Oberman
ober...@civicscience.comwrote:

 I've learned a *lot* from this thread.  My thanks to all of the
 contributors!

 Paulo: Good luck with LCS.  I wish I could help there, but all of my CF's
 are SizeTiered (mostly as I'm on the same schema/same settings since 0.7...)

 will



 On Fri, Apr 11, 2014 at 12:14 PM, Mina Naguib mina.nag...@adgear.comwrote:


 Levelled Compaction is a wholly different beast when it comes to
 tombstones.

 The tombstones are inserted, like any other write really, at the lower
 levels in the leveldb hierarchy.

 They are only removed after they have had the chance to naturally
 migrate upwards in the leveldb hierarchy to the highest level in your data
 store.  How long that takes depends on:
  1. The amount of data in your store and the number of levels your LCS
 strategy has
 2. The amount of new writes entering the bottom funnel of your leveldb,
 forcing upwards compaction and combining

 To give you an idea, I had a similar scenario and ran a (slow, throttled)
 delete job on my cluster around December-January.  Here's a graph of the
 disk space usage on one node.  Notice the still-diclining usage long after
 the cleanup job has finished (sometime in January).  I tend to think of
 tombstones in LCS as little bombs that get to explode much later in time:

 http://mina.naguib.ca/images/tombstones-cassandra-LCS.jpg



 On 2014-04-11, at 11:20 AM, Paulo Ricardo Motta Gomes 
 paulo.mo...@chaordicsystems.com wrote:

 I have a similar problem here, I deleted about 30% of a very large CF
 using LCS (about 80GB per node), but still my data hasn't shrinked, even if
 I used 1 day for gc_grace_seconds. Would nodetool scrub help? Does nodetool
 scrub forces a minor compaction?

 Cheers,

 Paulo


 On Fri, Apr 11, 2014 at 12:12 PM, Mark Reddy mark.re...@boxever.comwrote:

 Yes, running nodetool compact (major compaction) creates one large
 SSTable. This will mess up the heuristics of the SizeTiered strategy (is
 this the compaction strategy you are using?) leading to multiple 'small'
 SSTables alongside the single large SSTable, which results in increased
 read latency. You will incur the operational overhead of having to manage
 compactions if you wish to compact these smaller SSTables. For all these
 reasons it is generally advised to stay away from running compactions
 manually.

 Assuming that this is a production environment and you want to keep
 everything running as smoothly as possible I would reduce the gc_grace on
 the CF, allow automatic minor compactions to kick in and then increase the
 gc_grace once again after the tombstones have been removed.


 On Fri, Apr 11, 2014 at 3:44 PM, William Oberman 
 ober...@civicscience.com wrote:

 So, if I was impatient and just wanted to make this happen now, I
 could:

 1.) Change GCGraceSeconds of the CF to 0
 2.) run nodetool compact (*)
 3.) Change GCGraceSeconds of the CF back to 10 days

 Since I have ~900M tombstones, even if I miss a few due to impatience,
 I don't care *that* much as I could re-run my clean up tool against the now
 much smaller CF.

 (*) A long long time ago I seem to recall reading advice about don't
 ever run nodetool compact, but I can't remember why.  Is there any bad
 long term consequence?  Short term there are several:
 -a heavy operation
 -temporary 2x disk space
 -one big SSTable afterwards
 But moving forward, everything is ok right?
  CommitLog/MemTable-SStables, minor compactions that merge SSTables,
 etc...  The only flaw I can think of is it will take forever until the
 SSTable minor compactions build up enough to consider including the big
 SSTable in a compaction, making it likely I'll have to self manage
 compactions.



 On Fri, Apr 11, 2014 at 10:31 AM, Mark Reddy mark.re...@boxever.comwrote:

 Correct, a tombstone will only be removed after gc_grace period has
 elapsed. The default value is set to 10 days which allows a great deal of
 time for consistency to be achieved prior to deletion. If you are
 operationally confident that you can achieve consistency via anti-entropy
 repairs within a shorter period you can always reduce that 10 day 
 interval.


 Mark


 On Fri, Apr 11, 2014 at 3:16 PM, William Oberman 
 ober...@civicscience.com wrote:

 I'm seeing a lot of articles about a dependency between removing
 tombstones and GCGraceSeconds, which might be my problem (I just checked,
 and this CF has GCGraceSeconds of 10 days).


 On Fri, Apr 11, 2014 at 10:10 AM, tommaso barbugli 
 tbarbu...@gmail.com wrote:

 compaction should take care of it; for me it never worked so I run
 nodetool compaction on every node; that does it.


 2014-04-11 16:05 GMT+02:00 William Oberman ober...@civicscience.com
 :

 I'm wondering what will clear tombstoned rows?  nodetool cleanup,
 nodetool repair, or time (as in just wait)?

 I had a CF

Re: How to replace cluster name without any impact?

[Mark Reddy]

What version are you running? As of 1.2.x you can do the following:

1. Start the cqlsh connected locally to the node.
2. Run:
update system.local set cluster_name='$CLUSTER_NAME' where key='local';
3. Run nodetool flush on the node.
4. Update the cassandra.yaml file on the node, changing the cluster_name to
the same as you set in step 2.
5. Restart the node.


Please be aware that you will have two partial clusters until you complete
your rolling restart. Also considering that the cluster name is only a
cosmetic value my opinion would be to leave it, as the risk far outweighs
the benefits of changing it.


Mark


On Thu, Apr 10, 2014 at 2:49 AM, Check Peck comptechge...@gmail.com wrote:

 We have around 36 node Cassandra cluster and we have three Datacenters.
 Each datacenter have 12 node.

 We already have data flowing in Cassandra now and we cannot wipe out all
 our data now.

 Considering this - what is the right way to rename the cluster name
 without any or minimal impact?

Re: Migrating to new datacenter

[Mark Reddy]

I would go with option 1.

I think it is the safer of the two options, involves less work and if
something were to go wrong mid migration you can remove the second DC from
your keyspace replication and have a clean break. SimpleStrategy will work
across DCs. It is generally advised to not use it across DCs as it is hard
to maintain and can induce latency on writes depending on DC distance,
however as you are just migrating and then killing the original DC you
won't face these issues.


Mark


On Mon, Apr 7, 2014 at 5:25 PM, Brandon McCauslin bm3...@gmail.com wrote:

 We're currently running a small 5 node 2.0.5 cluster in a single
 datacenter using the SimpleStrategy replication strategy with replication
 factor of 3.  We want to migrate our data from our current datacenter to a
 new datacenter, without incurring any downtime or data loss.  There is no
 plan to maintain or keep the old (current) datacenter after the migration
 is complete.  I'm wondering how best to go about this migration.

 *Shoud We:*

1. add a 2nd datacenter as described here - Adding a datacenter to

 clusterhttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html,
then decommission the 1st 
 datacenterhttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_decomission_dc_t.html.
 Can we even do this given we're currently setup with SimpleStrategy
replication?
2. or should we just start adding new nodes from the new datacenter to
our existing cluster as described 
 herehttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html?
 Then once we've added all the new nodes, start decommissioning all the old
nodes from the old/current datacenter as described 
 herehttp://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html
.

 Thanks for your help and time!

Re: Securing Cassandra database

[Mark Reddy]

Hi,

If you want to just secure OpsCenter itself take a look here:
http://www.datastax.com/documentation/opscenter/4.1/opsc/configure/opscAssigningAccessRoles_t.html


If you want to enable internal authentication and still allow OpsCenter
access, you can create an OpsCenter user and once you have auth turned
within the cluster update the cluster config with the user name and
password for the OpsCenter user.

Depending on your installation type you will find the cluster config in one
of the following locations:
Packaged installs: /etc/opscenter/clusters/cluster_specific.conf
Binary installs: install_location/conf/clusters/cluster_specific.conf
Windows installs: Program Files (x86)\DataStax
Community\opscenter\conf\clusters\cluster_specific.conf

Open the file and update the username and password values under the
[cassandra] section:

[cassandra]
username =
seed_hosts =
api_port =
password =

After changing properties in this file, restart OpsCenter for the changes
to take effect.


Mark


On Sat, Apr 5, 2014 at 6:54 AM, Check Peck comptechge...@gmail.com wrote:

 Hi All,

 We would like to secure our Cassandra database. We don't want anybody to
 read/write on our Cassandra database leaving our team members only.



 We are using Cassandra 1.2.9 in Production and we have 36 node Cassandra
 cluster. 12 in each colo as we have three datacenters.


 But we would like to have OPSCENTER working as it is working currently.



 Is this possible to do anyhow? Is there any settings in yaml file which we
 can enforce?





 *Raihan Jamal*

Re: Securing Cassandra database

[Mark Reddy]

Ok so you want to enable auth on Cassandra itself. You will want to look
into the authentication and authorisation functionality then.

Here is a quick overview:
http://www.datastax.com/dev/blog/a-quick-tour-of-internal-authentication-and-authorization-security-in-datastax-enterprise-and-apache-cassandra

This section of the docs should give you the technical details needed to
move forward on this:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/security/securityTOC.html


Mark


On Sat, Apr 5, 2014 at 7:31 AM, Check Peck comptechge...@gmail.com wrote:

 Just to add, nobody should be able to read and write into our Cassandra
 database through any API *or any CQL client as well *only our team should
 be able to do that.


 On Fri, Apr 4, 2014 at 11:29 PM, Check Peck comptechge...@gmail.comwrote:

 Thanks Mark. But what about Cassandra database? I don't want anybody to
 read and write into our Cassandra database through any API only just our
 team should be able to do that.

 We are using CQL based tables so data doesn't get shown on the OPSCENTER.

 In our case, we would like to secure database itself. Is this possible to
 do as well anyhow?





 On Fri, Apr 4, 2014 at 11:24 PM, Mark Reddy mark.re...@boxever.comwrote:

 Hi,

 If you want to just secure OpsCenter itself take a look here:
 http://www.datastax.com/documentation/opscenter/4.1/opsc/configure/opscAssigningAccessRoles_t.html


 If you want to enable internal authentication and still allow OpsCenter
 access, you can create an OpsCenter user and once you have auth turned
 within the cluster update the cluster config with the user name and
 password for the OpsCenter user.

 Depending on your installation type you will find the cluster config in
 one of the following locations:
 Packaged installs: /etc/opscenter/clusters/cluster_specific.conf
 Binary installs: install_location/conf/clusters/cluster_specific.conf
 Windows installs: Program Files (x86)\DataStax
 Community\opscenter\conf\clusters\cluster_specific.conf

 Open the file and update the username and password values under the
 [cassandra] section:

 [cassandra]
 username =
 seed_hosts =
 api_port =
 password =

 After changing properties in this file, restart OpsCenter for the
 changes to take effect.


 Mark


 On Sat, Apr 5, 2014 at 6:54 AM, Check Peck comptechge...@gmail.comwrote:

 Hi All,

 We would like to secure our Cassandra database. We don't want anybody
 to read/write on our Cassandra database leaving our team members only.



 We are using Cassandra 1.2.9 in Production and we have 36 node
 Cassandra cluster. 12 in each colo as we have three datacenters.


 But we would like to have OPSCENTER working as it is working currently.



 Is this possible to do anyhow? Is there any settings in yaml file which
 we can enforce?