Re: vnodes: high availability

[Rahul Neelakantan]

Not necessarily. It depends on how the token ranges for the vNodes are
assigned to them. For example take a look at this diagram
http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeDistribute_c.html

In the vNode part of the diagram, you will see that Loss of Node 3 and Node
6, will still not have any effect on Token Range A. But yes if you lose two
nodes that both have Token Range A assigned to them (Say Node 1 and Node
2), you will have unavailability with your specified configuration.

You can sort of circumvent this by using the DataStax Java Driver and
having the client recognize a degraded cluster and operate temporarily in
downgraded consistency mode

http://docs.datastax.com/en/latest-java-driver-api/com/datastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html

- Rahul

On Mon, Jan 15, 2018 at 10:04 AM, Kyrylo Lebediev 
wrote:

> Hi,
>
>
> Let's say we have a C* cluster with following parameters:
>
>  - 50 nodes in the cluster
>
>  - RF=3
>
>  - vnodes=256 per node
>
>  - CL for some queries = QUORUM
>
>  - endpoint_snitch = SimpleSnitch
>
>
> Is it correct that 2 any nodes down will cause unavailability of a
> keyrange at CL=QUORUM?
>
>
> Regards,
>
> Kyrill
>

Re: Data sync between 2 clusters in single DC

[Rahul Neelakantan]

Why wouldnt you set it up as a single cluster that spans 2 DCs?

On Mon, Oct 30, 2017 at 4:09 PM, Vincent Lee  wrote:

> For high availability in a single DC region, I would like to install one
> Cassandra cluster on one AZ and a second cluster on a different AZ.
> The data between them needs to be synchronized. Is this possible?
>
> Note that this is for a single DC (region).
> Currently I am using GossipingPropertyFileSnitch.
>
> I look forward for your input.
>

Re: calculation of disk size

[Rahul Neelakantan]

Here is a calculator someone has put together 

http://btoddb-cass-storage.blogspot.com/2012/01/ive-been-asked-for-spreadsheet-to-help.html?m=1

Rahul

 On Apr 30, 2015, at 12:53 AM, arun sirimalla arunsi...@gmail.com wrote:
 
 Hi Rahul,
 
 If you are expecting 15 GB of data per day, here is the calculation.
 
 1 Day = 15 GB, 1 Month = 450 GB, 1 Year = 5.4 TB, so your raw data size for 
 one year is 5.4 TB with replication factor of 3 it would be around 16.2 TB of 
 data for one year.
 
 Taking compaction into consideration and your use case being write heavy, if 
 you go with size tiered compaction. you would need twice the space of your 
 raw data. 
 
 So you would need around 32-34 TB of disk space.
 
 Reference: 
 http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architecturePlanningDiskCapacity_t.html
 
 Thanks
 
 On Wed, Apr 29, 2015 at 9:20 PM, Rahul Bhardwaj 
 rahul.bhard...@indiamart.com wrote:
 Hi All,
 
 
 We are planning to set up a cluster of 5 nodes with RF 3 for write heavy 
 project, our current database size is around 500 GB. And it is growing at 
 rate of 15 GB every day. We learnt that cassandra consumes space for 
 compaction processes, So how can we calculate the amount of disk space we 
 would require. 
 
 Kindly suggest.
 
 
 
 Regards:
 Rahul Bhardwaj
 
 
 Follow IndiaMART.com for latest updates on this and more:Mobile Channel: 
   
 
 Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki 
 Kaam Yahin Banta Hai!!!
 
 
 
 -- 
 Arun 
 Senior Hadoop/Cassandra Engineer
 Cloudwick
 
 Champion of Big Data (Cloudera)
 http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html
 
 2014 Data Impact Award Winner (Cloudera)
 http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
 

Re: How to use nodetool ring only for one data center

[Rahul Neelakantan]

Do you want this for some sort of reporting requirement? If so you may be able 
to write a quick she'll script using grep to remove the unwanted data

Rahul

 On Apr 28, 2015, at 7:24 PM, Surbhi Gupta surbhi.gupt...@gmail.com wrote:
 
 Hi,
 
 I wanted to know, how can we get the information of the token rings only for 
 one data centers when using vnodes and multiple data center.
 
 Thanks
 Surbhi

Re: Drawbacks of Major Compaction now that Automatic Tombstone Compaction Exists

[Rahul Neelakantan]

Rob,
Does that mean once you split it back into small ones, automatic compaction a 
will continue to happen on a more frequent basis now that it's no longer a 
single large monolith?

Rahul

 On Apr 13, 2015, at 3:23 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Mon, Apr 13, 2015 at 10:52 AM, Anuj Wadehra anujw_2...@yahoo.co.in 
 wrote:
 
 Any comments on side effects of Major compaction especially when sstable 
 generated is 100+ GB? 
 
 I have no idea how this interacts with the automatic compaction stuff; if you 
 find out, let us know?
 
 But if you want to do a major and don't want to deal with One Big SSTable 
 afterwards, stop the node and then run sstable_split utility. 
 
 =Rob
 

Re: Adding new node to Cassandra cluster is too slow

[Rahul Neelakantan]

You won't lose data unless you have run nodetool cleanup on the existing nodes.

Rahul

 On Mar 19, 2015, at 9:16 PM, Pranay Agarwal agarwalpran...@gmail.com wrote:
 
 Also, the new nodes (3 of them, in UJ state) are showing some data size 
 (~10g). Is there any data loss chances with stopping the cassandra on them? 
 
 On Thu, Mar 19, 2015 at 6:02 PM, Pranay Agarwal agarwalpran...@gmail.com 
 wrote:
 Thanks Rob, You are right. I am using ReleaseVersion: 2.1.0
  
 What do you mean by point 3? Also, by doing one at a time, does it mean wait 
 till nodetool status of the new node is UN from UJ?
 
 On Thu, Mar 19, 2015 at 5:44 PM, Robert Coli rc...@eventbrite.com wrote:
 On Thu, Mar 19, 2015 at 5:32 PM, Pranay Agarwal agarwalpran...@gmail.com 
 wrote:
 I have 14 nodes cassandra cluster, each node as around 50gb of data. I 
 added 3 new nodes to the cluster and I can see the status as UJ for the 
 new nodes. They have been in that for almost a day now and their data size 
 seems to be same as well. There is almost no CPU or disk usage either on 
 them.
  
 It is not supported to add multiple nodes to a cluster simultaneously until 
 2.1.1 [1]. Usually what happens is one or more of the bootstraps fails and 
 hangs forever. This seems to be what has happened to you.
 
 To resolve :
 
 1) stop each of the bootstrapping nodes
 2) wipe their data directories completey
 3) verify that they do not show up in gossip on the other nodes
 4) bootstrap them again, one at a time
 
 =Rob
 [1] https://issues.apache.org/jira/browse/CASSANDRA-7069
 

Re: best way to measure repair times?

[Rahul Neelakantan]

Wouldn't GC Grace set to 34 days increase the bloat in the DB?

Rahul

 On Mar 19, 2015, at 3:02 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Thu, Mar 19, 2015 at 10:30 AM, Ian Rose ianr...@fullstory.com wrote:
 I'd like to (a) monitor how long my repairs are taking, and (b) know when a 
 repair is finished so that I can take some kind of followup action.  What's 
 the best way to tackle either or both of these?
 
 https://issues.apache.org/jira/browse/CASSANDRA-5483
 
 Also consider increasing your gc_grace_seconds to 34 days by default 
 (CASSANDRA-5850) to decrease the frequency of repair.
 
 =Rob
  

Re: 2 Cassandra nodes slow

[Rahul Neelakantan]

Is the data distribution OK? Have you tried running repairs?

Rahul

 On Jan 13, 2015, at 5:01 AM, Batranut Bogdan batra...@yahoo.com wrote:
 
 Hello,
 
 I have a cluster of 6 C* nodes. All machines have the same hardware. I have 
 noticed in opscenter that when I start reading a lot from the cluster 2 nodes 
 have read latencies, but the rest do not have such high values. The 
 replication factor for the keyspace is 3. Also those 2 nodes have latencies 
 when measuring the CF Local Read Latency so I know that the reads on the main 
 col family are slow on these 2 nodes. Partitioning is good since in the 
 column family there are a few million possible values.
 Another thing is that I have stopped these 2 nodes and ran a few selections 
 for 6000 rows  those were executed in about 15 seconds. I have changed 
 the keys so that I avoid caching as much as possible.
 Also I have done the same test when running with all nodes and the execution 
 of the selection was abou 22 seconds
 
 Any ideeas? 
 
 I am thinking on rebuilding the data on the nodes... Stop the node , delete 
 the data folder and restart... but I don't know the consequences

Re: problem in exporting large table

[Rahul Neelakantan]

Why can't you use sstable2json?


Rahul

 On Jan 12, 2015, at 11:24 PM, Rahul Bhardwaj rahul.bhard...@indiamart.com 
 wrote:
 
 Hi All,
 
 We are using C* 2.1. we need to export data of one table (consist 10 lacs 
 records) using COPY command. After executing copy command cqlsh hangs and get 
 stuck . Please help in resolving the same or provide any alternative for the 
 same. pfb table stats:
 
 Keyspace: clickstream
 Read Count: 3567
 Read Latency: 8.109851135407906 ms.
 Write Count: 923452
 Write Latency: 2.8382575358545976 ms.
 Pending Flushes: 0
 Table: business_feed_new
 SSTable count: 15
 Space used (live): 446908108
 Space used (total): 446908108
 Space used by snapshots (total): 0
 SSTable Compression Ratio: 0.21311411274805014
 Memtable cell count: 249458
 Memtable data size: 14938837
 Memtable switch count: 37
 Local read count: 3567
 Local read latency: 8.110 ms
 Local write count: 923452
 Local write latency: 2.839 ms
 Pending flushes: 0
 Bloom filter false positives: 0
 Bloom filter false ratio: 0.0
 Bloom filter space used: 560
 Compacted partition minimum bytes: 18
 Compacted partition maximum bytes: 557074610
 Compacted partition mean bytes: 102846983
 Average live cells per slice (last five minutes): 
 96.81356882534342
 Maximum live cells per slice (last five minutes): 102.0
 Average tombstones per slice (last five minutes): 0.0
 Maximum tombstones per slice (last five minutes): 0.0
 
 
 
 regards
 
 Rahul Bhardwaj
 
 
 
 
 Follow IndiaMART.com for latest updates on this and more:Mobile Channel:  
  
 
 Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam 
 Yahin Banta Hai!!!

Re: nodetool compact cannot remove tombstone in system keyspace

[Rahul Neelakantan]

I am not sure about the tombstone_failure_threshold, but the tombstones will 
only get removed during compaction if they are older than GC_Grace_Seconds for 
that CF. How old are these tombstones?

Rahul

 On Jan 12, 2015, at 11:27 PM, Xu Zhongxing xu_zhong_x...@163.com wrote:
 
 Hi,
 
 When I connect to C* with driver, I found some warnings in the log (I 
 increased tombstone_failure_threshold to 15 to see the warning)
 
 WARN [ReadStage:5] 2015-01-13 12:21:14,595 SliceQueryFilter.java (line 225) 
 Read 34188 live and 104186 tombstoned cells in system.schema_columns (see 
 tombstone_warn_threshold). 2147483387 columns was requested, slices=[-], 
 delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
  WARN [ReadStage:5] 2015-01-13 12:21:15,562 SliceQueryFilter.java (line 225) 
 Read 34209 live and 104247 tombstoned cells in system.schema_columns (see 
 tombstone_warn_threshold). 2147449199 columns was requested, slices=[-], 
 delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
 
 I run the command:
 nodetool compact system 
 
 But the tombstone number does not decrease. I still see the warnings with the 
 exact number of tombstones.
 Why is this happening? What should I do to remove the tombstones in the 
 system keyspace?

Re: Hinted handoff not working

[Rahul Neelakantan]

http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html?scroll=reference_ds_qfg_n1r_1k__hinted_handoff_enabled

Rahul

 On Dec 14, 2014, at 9:46 AM, Robert Wille rwi...@fold3.com wrote:
 
 I have a cluster with RF=3. If I shut down one node, add a bunch of data to 
 the cluster, I don’t see a bunch of records added to system.hints. Also, du 
 of /var/lib/cassandra/data/system/hints of the nodes that are up shows that 
 hints aren’t being stored. When I start the down node, its data doesn’t grow 
 until I run repair, which then takes a really long time because it is 
 significantly out of date. Is there some magic setting I cannot find in the 
 documentation to enable hinted handoff? I’m running 2.0.11. Any insights 
 would be greatly appreciated. 
 
 Thanks
 
 Robert
 

Data not replicating consistently

[Rahul Neelakantan]

I have a setup that looks like this

Dc1: 9 nodes
Dc2: 9 nodes
Dc3: 9 nodes
C* version: 2.0.10
RF: 2 in each DC
Empty CF with no data at the beginning of the test

Scenario 1 (happy path): I connect to a node in DC1 using CQLsh, validate that 
I am using CL=1, insert 10 rows.
Then using CQLsh connect to one node in each of the 3 DCs and with CL=1, select 
* on the table, each DC shows all 10 rows.

Scenario 1: using a program based on datastax drivers, write 10 rows to DC1. 
The program uses CL=1, also does a read after write.
Then using CQLsh connect to one node in each of the 3 DCs and with CL=1 or 
LocalQuorum, select * on the table,
DC1 shows all 10 rows.
DC2 shows 8 or 9 rows
DC3 shows 8 or 9 rows 
The missing rows never show up in DC2 and DC3 unless I do a CQLsh lookup with 
CL=all

Why is there a difference in the replication between writes performed using the 
datastax drivers and while using CQLsh?


Rahul

Re: Force purging of tombstones

[Rahul Neelakantan]

Is this page incorrect then and needs to be updated or am I interpreting it 
incorrectly ?

http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_about_deletes_c.html

Particularly this sentence 

After data is marked with a tombstone, the data is automatically removed 
during the normal compaction and repair processes

Rahul

 On Nov 17, 2014, at 6:57 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Mon, Nov 17, 2014 at 3:10 PM, Rahul Neelakantan ra...@rahul.be wrote:
 Doesn't repair also get rid of tombstones?
 
 Repair is a non-destructive activity, and therefore cannot purge tombstones.
 
 =Rob
  

Re: Force purging of tombstones

[Rahul Neelakantan]

Doesn't repair also get rid of tombstones?

Rahul Neelakantan

 On Nov 17, 2014, at 5:53 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Mon, Nov 17, 2014 at 1:40 PM, Ken Hancock ken.hanc...@schange.com wrote:
 You can use the JMX forceUserDefinedCompaction operation to compact each 
 SSTable individually.
 
 https://github.com/hancockks/cassandra-compact-cf
 
 I don't recall why I think this, but I think cleanup now also discards 
 expired tombstones, and is easier to use from nodetool than 
 UserDefinedCompaction.
 
 =Rob
  

Re: Did not get positive replies from all endpoints error on incremental repair

[Rahul Neelakantan]

It appears to come from the ActiveRepairService.prepareForRepair portion of the 
Code.

Are you sure all nodes are reachable from the node you are initiating repair 
on, at the same time?

Any Node up/down/died messages?

Rahul Neelakantan

 On Oct 30, 2014, at 6:37 AM, Juho Mäkinen juho.maki...@gmail.com wrote:
 
 I'm having problems running nodetool repair -inc -par -pr on my 2.1.1 cluster 
 due to Did not get positive replies from all endpoints error.
 
 Here's an example output:
 root@db08-3:~# nodetool repair -par -inc -pr  

 [2014-10-30 10:33:02,396] Nothing to repair for keyspace 'system'
 [2014-10-30 10:33:02,420] Starting repair command #10, repairing 256 ranges 
 for keyspace profiles (seq=false, full=false)
 [2014-10-30 10:33:17,240] Repair failed with error Did not get positive 
 replies from all endpoints.
 [2014-10-30 10:33:17,263] Starting repair command #11, repairing 256 ranges 
 for keyspace OpsCenter (seq=false, full=false)
 [2014-10-30 10:33:32,242] Repair failed with error Did not get positive 
 replies from all endpoints.
 [2014-10-30 10:33:32,249] Starting repair command #12, repairing 256 ranges 
 for keyspace system_traces (seq=false, full=false)
 [2014-10-30 10:33:44,243] Repair failed with error Did not get positive 
 replies from all endpoints.
 
 The local system log shows that the repair commands got started, but it seems 
 that they immediately get cancelled due to that error, which btw can't be 
 seen in the cassandra log.
 
 I tried monitoring all logs from all machines in case another machine would 
 show up with some useful error, but so far I haven't found nothing.
 
 Any ideas where this error comes from?
 
  - Garo
 

Re: opscenter with community cassandra

[Rahul Neelakantan]

You can turn off the phone home function in the opscenterd.conf file

[stat_reporter]
interval = 0


Rahul Neelakantan

 On Oct 28, 2014, at 11:08 AM, Colin colpcl...@gmail.com wrote:
 
 I cant run opscenter in a secure environment for a couple of reasons, one - 
 it phones home, two - lack of role based security.
 
 It is a mistake to call a proprietary piece of software community when you 
 cant use it in production.
 
 It is easy enough to automate what opscenter does rather than relying in a 
 third party in my enterprise,.
 
 
 
 On Oct 28, 2014, at 10:04 AM, Josh Smith josh.sm...@careerbuilder.com 
 wrote:
 
 Yes Opscenter does work with the opensource version of Cassandra. I am 
 currently running both in the cloud and our private datacenter with no 
 problems. I have not tried 2.1.1 yet but I do not see why it wouldn’t work 
 also.
  
 Josh
  
 From: Tim Dunphy [mailto:bluethu...@gmail.com] 
 Sent: Tuesday, October 28, 2014 10:43 AM
 To: user@cassandra.apache.org
 Subject: opscenter with community cassandra
  
 Hey all,
  
  I'd like to setup datastax opscenter to monitor my cassandra ring. However 
 I'm using the open source version of 2.1.1. And before I expend any time and 
 effort in setting this up, I'm wondering if it will work with the open 
 source version? Or would I need to be running datastax cassandra in order to 
 get this going?
  
 Thanks
 Tim
  
 -- 
 GPG me!!
 
 gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: Looking for a cassandra web interface

[Rahul Neelakantan]

It is not web based, you can try Helenos, but I don't think it is ready for 
newer versions.

Rahul Neelakantan

 On Oct 21, 2014, at 12:26 AM, Vishanth Balasubramaniam vishan...@wso2.com 
 wrote:
 
 I believe DevCenter is not a 'web' interface, isn't it?
 
 On Mon, Oct 20, 2014 at 11:40 PM, Jared Biel jared.b...@bolderthinking.com 
 wrote:
 Very nice, I had forgotten about Cyclop. We used to use Cassandra Cluster 
 Admin, but it doesn't support CQL so it's not very useful anymore. If you're 
 looking for a developer UI, I recommend trying DataStax DevCenter 
 (http://www.datastax.com/what-we-offer/products-services/devcenter). It's 
 completely free.
  
 
 On 20 October 2014 18:05, DuyHai Doan doanduy...@gmail.com wrote:
 I remember someone mentioning Cyclop, a web interface based on Apache 
 Wicket for Cassandra: https://github.com/maciejmiklas/cyclop
 
 If you want a module which exposes Cassandra operations as REST services, 
 look at Virgil https://github.com/hmsonline/virgil
 
 On Mon, Oct 20, 2014 at 7:40 PM, Vishanth Balasubramaniam 
 vishan...@wso2.com wrote:
 Hi,
 
 I am very new to cassandra. I have started cassandra inside an instance in 
 my VM and I want to expose a cassandra web interface. What is the most 
 stable web interface for Cassandra with a proper guide to set up?
 
 Thanks and Regards,
 Vishanth
 
 
 
 -- 
 Vishanth Balasubramaniam
 Software Engineer
 WSO2 Inc.; http://wso2.com
 lean.enterprise.middleware
 
 mobile: +94771737718
 about me: http://about.me/vishanth
 

Deleting data in 1.2.8 based on time stamp

[Rahul Neelakantan]

Does anyone know of a way to delete rows from C* 1.2.8 based on the timestamp 
(time from epoch) that is present on each column in the triplet of name, value 
and timestamp? (I do not have a separate date/timestamp column that I insert)

Rahul Neelakantan

Re: Deleting data in 1.2.8 based on time stamp

[Rahul Neelakantan]

So this would need me to know the partition keys, what if I simply wanted to 
say delete all rows where the timestamp was older than 123456789? 


Rahul Neelakantan

 On Oct 16, 2014, at 6:27 PM, Tyler Hobbs ty...@datastax.com wrote:
 
 For each partition in the table, run:
 
 DELETE FROM mytable WHERE partitionkey=? USING TIMESTAMP 123456789
 
 And it will delete everything older than or equal to 123456789 (in 
 microseconds since the epoch, if you're using standard timestamps).
 
 On Thu, Oct 16, 2014 at 5:09 PM, Rahul Neelakantan ra...@rahul.be wrote:
 Does anyone know of a way to delete rows from C* 1.2.8 based on the 
 timestamp (time from epoch) that is present on each column in the triplet of 
 name, value and timestamp? (I do not have a separate date/timestamp column 
 that I insert)
 
 Rahul Neelakantan
 
 
 
 -- 
 Tyler Hobbs
 DataStax

Re: Repair taking long time

[Rahul Neelakantan]

What is the recommendation on the number of tokens value? I am asking because 
of the issue with sequential repairs on token range after token range.

Rahul Neelakantan

 On Sep 29, 2014, at 2:29 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Fri, Sep 26, 2014 at 9:52 AM, Gene Robichaux gene.robich...@match.com 
 wrote:
 I am fairly new to Cassandra. We have a 9 node cluster, 5 in one DC and 4 in 
 another.
 
  
 
 Running a repair on a large column family seems to be moving much slower 
 than I expect.
 
 
 Unfortunately, as others have mentioned, the slowness/broken-ness of repair 
 is a long running (groan!) issue and therefore currently expected. 
 
 At this time, I do not recommend upgrading to 2.1 in production to attempt to 
 fix it. I am also broadly skeptical that it as fixed in 2.1 as all that.
 
 Once can increase gc_grace_seconds to 34 days [1] and repair once a month, 
 which should help make repair slightly more tractable.
 
 For now you should probably evaluate which of your column families you 
 *absolutely must* repair (because you do DELETE like operations in them, 
 etc.) and only repair those.
 
 As an aside, you just lose with vnodes and clusters of the size. I presume 
 you plan to grow over appx 9 nodes per DC, in which case you probably do want 
 vnodes enabled.
 
 One note :
  Looking at nodetool compaction stats it indicates the Validation phase is 
 running that the total bytes is 4.5T (4505336278756).
 
 This is the uncompressed size, I'm betting your actual on disk size is closer 
 to 2T? Even though 2.0 has improved performance for nodes with lots of data, 
 2T per node is still relatively fat for a Cassandra node.
 
 
 =Rob
 [1] https://issues.apache.org/jira/browse/CASSANDRA-5850

Re: ava.lang.OutOfMemoryError: unable to create new native thread

[Rahul Neelakantan]

What is your sstable size set to for each of the sstables, using LCS? Are you 
at the default of 5 MB?

Rahul Neelakantan

 On Sep 17, 2014, at 10:58 AM, Yatong Zhang bluefl...@gmail.com wrote:
 
 sorry, about 300k+
 
 On Wed, Sep 17, 2014 at 10:56 PM, Yatong Zhang bluefl...@gmail.com wrote:
 no, I am running 64 bit JVM。 But I have many sstable files, about 30k+
 
 On Wed, Sep 17, 2014 at 10:50 PM, graham sanderson gra...@vast.com wrote:
 Are you running on a 32 bit JVM?
 
 On Sep 17, 2014, at 9:43 AM, Yatong Zhang bluefl...@gmail.com wrote:
 
 Hi there,
 
 I am using leveled compaction strategy and have many sstable files. The 
 error was during the startup, so any idea about this?
  
 ERROR [FlushWriter:4] 2014-09-17 22:36:59,383 CassandraDaemon.java (line 
 199) Exception in thread Thread[FlushWriter:4,5,main]
 java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:693)
 at 
 java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
 at 
 java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:724)
 ERROR [FlushWriter:2] 2014-09-17 22:36:59,472 CassandraDaemon.java (line 
 199) Exception in thread Thread[FlushWriter:2,5,main]
 FSReadError in 
 /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-23-Index.db
 at 
 org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
 at 
 org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:324)
 at 
 org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:394)
 at 
 org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342)
 at 
 org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:724)
 Caused by: java.io.IOException: Map failed
 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
 at 
 org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
 ... 10 more
 Caused by: java.lang.OutOfMemoryError: Map failed
 at sun.nio.ch.FileChannelImpl.map0(Native Method)
 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846)
 ... 11 more
 

Re: Questions about cleaning up/purging Hinted Handoffs

[Rahul Neelakantan]

Will try... Thank you 

Rahul Neelakantan

 On Sep 10, 2014, at 12:01 AM, Rahul Menon ra...@apigee.com wrote:
 
 I use jmxterm. http://wiki.cyclopsgroup.org/jmxterm/ attach it to your c* 
 process and then use the org.apache.cassandra.db:HintedHandoffManager bean 
 and run deleteHintsforEndpoint ip to drop hints for each ip. 
 
 
 
 
 On Wed, Sep 10, 2014 at 3:37 AM, Rahul Neelakantan ra...@rahul.be wrote:
 RF=3, two DCs. (Going to 1.2.x in a few weeks)
 
 What's the procedure to drop via JMX?
 
 - Rahul
 1-678-451-4545 (US)
 +91 99018-06625 (India)
 
 
 On Sep 9, 2014, at 9:23 AM, Rahul Menon ra...@apigee.com wrote:
 
 Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your 
 replication facter, i have had huge hints pile up, where i had to drop the 
 entire coloumn family and then run a repair. Either that or you can use the 
 JMX HintedHandoffManager and delete hints per endpoint. Also it maybe 
 worthwhile to investigate why you have hints pile up
 
 Rahul
 
 On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli rc...@eventbrite.com wrote:
 On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan ra...@rahul.be wrote:
 The reason I asked about he hints is because I see hints being replayed 
 but the large compacted hints stable still sticks around, perhaps it is a 
 bug with that version .
 
 I've seen this behavior with HH in older versions, so probably.
 
 =Rob
 

Re: hardware sizing for cassandra

[Rahul Neelakantan]

Why not more than 32gb of RAM/node?

Rahul Neelakantan

 On Sep 9, 2014, at 3:52 PM, Paolo Crosato paolo.cros...@targaubiest.com 
 wrote:
 
 Every node should have at least 4 cores, with a maximum of 8. Memory 
 shouldn't be higher than 32g, 16gb is good for a start. Every node should be 
 a phisical machine, not a virtual one, or at least a virtual machine with an 
 ssd hd subsystem. The disk subsystem should be directly connected to the 
 machine, no sans or fiber channel between. Cassandra is cpu and io bounded, 
 so you should get the maximum io speed and a reasonable number of cores.
 
 Number of nodes should be 3 at least with replication factor of 2. You should 
 prefer more less powerful nodes then fewer more powerful nodes.
 
 Disk size depends on your workload, although you should always keep 50% of 
 the disk free in the case repair sessions requires space, or perform sub 
 range repairs.
 
 In my experience a 1GB link between nodes is ok, but the less lag the better.
 
 Summing up if you need to save some money, get 4 cores and 16 gb or ram, 32 
 is rarely needed and 64 a waste. 8 cores would probably be too much with 1000 
 writes a second.
 
 Paolo
 
 
 
 Paolo Crosato
 Software engineer/Custom Solutions
 
 
 
 Da: Chris Lohfink clohf...@blackbirdit.com
 Inviato: martedì 9 settembre 2014 21.26
 A: user@cassandra.apache.org
 Oggetto: Re: hardware sizing for cassandra
 
 It depends.  Ultimately your load is low enough a single node can probably 
 handle it so you kinda want a minimum cluster.  Different people have 
 different thoughts on what this means - I would recommend 5-6 nodes with a 3 
 replication factor.  (say m1.xlarge, or c3.2xlarge striped ephemerals, I like 
 i2's but kinda overkill here).  Nodes with less then 16gb of ram wont last 
 long so should really start around there.
 
 Chris
 
 On Sep 9, 2014, at 11:02 AM, Oleg Ruchovets oruchov...@gmail.com wrote:
 
 Hi ,
   Where can I find the document with best practices about sizing for 
 cassandra deployment?
   We have 1000 writes / reads per second. record size 1k.
 
 Questions:
   1) how many machines do we need?
   2) how many ram ,disc size / type?
   3) What should be network?
 
 I understand that hardware is very depends on data distribution and access 
 pattern and other criteria, but I still want to believe that there is a best 
 practice :-)
 
 Thanks
 Oleg.
 

Re: Questions about cleaning up/purging Hinted Handoffs

[Rahul Neelakantan]

RF=3, two DCs. (Going to 1.2.x in a few weeks)

What's the procedure to drop via JMX?

- Rahul
1-678-451-4545 (US)
+91 99018-06625 (India)


On Sep 9, 2014, at 9:23 AM, Rahul Menon ra...@apigee.com wrote:

 Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your replication 
 facter, i have had huge hints pile up, where i had to drop the entire coloumn 
 family and then run a repair. Either that or you can use the JMX 
 HintedHandoffManager and delete hints per endpoint. Also it maybe worthwhile 
 to investigate why you have hints pile up
 
 Rahul
 
 On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli rc...@eventbrite.com wrote:
 On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan ra...@rahul.be wrote:
 The reason I asked about he hints is because I see hints being replayed but 
 the large compacted hints stable still sticks around, perhaps it is a bug 
 with that version .
 
 I've seen this behavior with HH in older versions, so probably.
 
 =Rob
 

Questions about cleaning up/purging Hinted Handoffs

[Rahul Neelakantan]

The questions are for Cassandra 1.0.8 but may apply to later versions too.

1) When the coordinator notices that a node it is collecting hints for is
down for more than the max hint window, it stops collecting hints for that
node
- Does it automatically purge the hints it collected so far, or does
removing them require manual intervention?
- What is the recommended way to remove hints if they are not
automatically removed? Stop node and delete hints CF db files? or use
truncate CF?

2) When the coordinator replays hints to a node that has come back within
the max hint window, does it purge those hints once they are replayed or
does cleanup of those hints require manual intervention?

3) If a node is flapping because of network issues and gossip keeps
thinking the node is down, causing the coordinator to reset the max hint
window for that node and start collecting hints again for the node
 - Does it automatically purge the hints collected in the previous windows?
 - Or Does it replay all hints collected in all windows that the node was
down?

- Rahul

Re: Questions about cleaning up/purging Hinted Handoffs

[Rahul Neelakantan]

Yup... Upgrades are in the works.. Thanks for the response. 

The reason I asked about he hints is because I see hints being replayed but the 
large compacted hints stable still sticks around, perhaps it is a bug with that 
version .

Rahul Neelakantan

 On Sep 5, 2014, at 1:03 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Fri, Sep 5, 2014 at 6:12 AM, Rahul Neelakantan ra...@rahul.be wrote:
 The questions are for Cassandra 1.0.8 but may apply to later versions too.
 
 Upgrade your very old version of Cassandra ASAP. I barely remember the 
 implementation of HH in your old version.
   
 1) When the coordinator notices that a node it is collecting hints for is 
 down for more than the max hint window, it stops collecting hints for that 
 node
 - Does it automatically purge the hints it collected so far, or does 
 removing them require manual intervention?
 
 It tries to deliver the hints it has stored so far, when it can. When it 
 delivers them, it deletes them.
   
 - What is the recommended way to remove hints if they are not 
 automatically removed? Stop node and delete hints CF db files? or use 
 truncate CF?
 
 Either way is fine. There's also a JMX method (maybe not in your old version) 
 to purge hints.
  
 2) When the coordinator replays hints to a node that has come back within 
 the max hint window, does it purge those hints once they are replayed or 
 does cleanup of those hints require manual intervention?
 
 The former. 
  
 3) If a node is flapping because of network issues and gossip keeps thinking 
 the node is down, causing the coordinator to reset the max hint window for 
 that node and start collecting hints again for the node
  - Does it automatically purge the hints collected in the previous windows?
 
 No.
  
  - Or Does it replay all hints collected in all windows that the node was 
 down?
 
 Yes.
 
 =Rob
  

Re: Question about EC2 and SSDs

[Rahul Neelakantan]

With SSD one drive should be sufficient for both data and commitLogs.

Rahul Neelakantan

 On Sep 4, 2014, at 8:05 PM, Steve Robenalt sroben...@highwire.org wrote:
 
 Hi all,
 
 We are migrating a small cluster on AWS from instances based on spinning 
 disks (using instance store) to SSD-backed instances and we're trying to pick 
 the proper instance type. Some of the recommendations for spinning disks say 
 to use different drives for log vs data partitions to avoid issues with seek 
 delays and contention for the disk heads. Since SSDs don't have the same seek 
 delays, is it still recommended to use 2 SSD drives? Or is one sufficient?
 
 Thanks,
 Steve
 

Re: disk space and tombstones

[Rahul Neelakantan]

Is that GC_grace 300 days?

Rahul Neelakantan

 On Aug 18, 2014, at 5:51 AM, Dimetrio dimet...@flysoft.ru wrote:
 
 In our Twitter-like application users have their own timelines with news from
 subscriptions. To populate timelines we're using fanout on write. But we
 forced to trim it to keep free disk space under control.
 
 We use wide rows pattern and trim them with DELETE by primary key USING
 TIMESTAMP. But it seems our efforts have no effect and disk free space
 decreases rapidly, even after compaction.
 
 It is clear for us that it's not the best use case for Cassandra, but maybe
 there is a way to decrease disk utilisation for this pattern?
 
 Our cluster consists of 15 c3.4xlarge nodes with 300 GB storage. Timeline's
 files take up 170 GB on each node.
 gc_grace is 300,
 rf=3
 
 
 
 --
 View this message in context: 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
 Nabble.com.

Re: A question about using 'update keyspace with strategyoptions' command

[Rahul Neelakantan]

Try running describe cluster from Cassandra-CLI to see if all nodes have the 
same schema version.

Rahul Neelakantan

 On Aug 5, 2014, at 6:13 AM, Sylvain Lebresne sylv...@datastax.com wrote:
 
 On Tue, Aug 5, 2014 at 11:40 AM, Lu, Boying boying...@emc.com wrote: 
 What I want to know is “are the strategy changed ?’ after the ‘udpate 
 keyspace with strategy_options…’ command returns successfully
 
 
 Like all schema changes, not necessarily on all nodes. You will have to check 
 for schema agreement between nodes.
 
  
 Not the data change.
 
  
 
 e.g. say I run the command ‘update keyspace with strategy_opitons [dc1: 3, 
 dc2:3]’ , when this command returns,
 
 are the strategy options already changed? Or I need to wait some time for 
 the strategy to be changed?
 
  
 
  
 
 From: Sylvain Lebresne [mailto:sylv...@datastax.com] 
 Sent: 2014年8月5日 16:59
 To: user@cassandra.apache.org
 
 
 Subject: Re: A question about using 'update keyspace with strategyoptions' 
 command
  
 
 Changing the strategy options, and in particular the replication factor, 
 does not perform any data replication by itself. You need to run a repair to 
 ensure data is replicated following the new replication.
 
  
 
 On Tue, Aug 5, 2014 at 10:52 AM, Lu, Boying boying...@emc.com wrote:
 
 Thanks. yes. I can use the ‘show keyspace’ command to check and see the 
 strategy does changed.
 
  
 
 But what I want to know is if the ‘update keyspace with strategy_options …’ 
 command is
 
 a ‘sync’ operation or a ‘async’ operation.
 
  
 
  
 
  
 
 From: Rahul Menon [mailto:ra...@apigee.com] 
 Sent: 2014年8月5日 16:38
 To: user
 Subject: Re: A question about using 'update keyspace with strategyoptions' 
 command
 
  
 
 Try the show keyspaces command and look for Options under each keyspace. 
 
  
 
 Thanks
 
 Rahul
 
  
 
 On Tue, Aug 5, 2014 at 2:01 PM, Lu, Boying boying...@emc.com wrote:
 
 Hi, All,
 
  
 
 I want to run ‘update keyspace with strategy_options={dc1:3, dc2:3}’ from 
 cassandra-cli to update the strategy options of some keyspace
 
 in a multi-DC environment.
 
  
 
 When the command returns successfully, does it mean that the strategy 
 options have been updated successfully or I need to wait
 
 some time for the change to be propagated  to all DCs?
 
  
 
 Thanks
 
  
 
 Boying
 
 

Re: How to get rid of stale info in gossip

[Rahul Neelakantan]

It no longer shows up, but I will definitely investigate that option. What 
exactly does it do?

Rahul Neelakantan

 On Aug 4, 2014, at 7:49 PM, Patricia Gorla patri...@thelastpickle.com wrote:
 
 Rahul,
 
 If the removed node is still showing up in gossip state, you can remove it 
 with 'unsafeAssassinate' via a JMX tool such as jconsole or jmxterm.
 
 
 On Fri, Jul 25, 2014 at 6:10 AM, Rahul Neelakantan ra...@rahul.be wrote:
 Yes, and this is a really old version of casandra 1.0.8.
 
 
 Rahul Neelakantan
 678-451-4545
 
 On Jul 25, 2014, at 7:29 AM, Mark Reddy mark.re...@boxever.com wrote:
 
 After removing a node, it's information can persist in the Gossiper for up 
 to 3 days, after which time it should be removed. 
 
 Are you having issues with a removed node state persisting for longer?
 
 
 Mark
 
 
 On Fri, Jul 25, 2014 at 11:33 AM, Rahul Neelakantan ra...@rahul.be wrote:
 Is there a way to get rid of stale information that shows up for 
 removed/dead nodes in gossip, without a complete cluster bounce?
 
 Rahul Neelakantan
 
 
 
 -- 
 Patricia Gorla
 @patriciagorla
 
 Consultant
 Apache Cassandra Consulting
 http://www.thelastpickle.com

Authentication for Mx4j

[Rahul Neelakantan]

Does anyone know how to enable basic authentication of MX4J with Cassandra? 
Mx4j supports it but not sure how to pass the variables to enable it. I was 
able to set the listen address and port for the http server, but can't get 
authentication to work.

Rahul Neelakantan

Re: Measuring WAN replication latency

[Rahul Neelakantan]

Rob,
Any ideas you can provide on how to do this will be appreciated, we would like 
to build a latency monitoring tool/dashboard that shows how long it takes for 
data to get sent across various DCs.

Rahul Neelakantan

 On Jul 29, 2014, at 8:53 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Tue, Jul 29, 2014 at 3:15 PM, Rahul Neelakantan ra...@rahul.be wrote:
 Does anyone know of a way to measure/monitor WAN replication latency for 
 Cassandra?
 
 No. [1]
 
 =Rob
  
 [1] There are ways to do something like this task, but you probably don't 
 actually want to do them. Trying to do them suggests that you are relying on 
 WAN replication timing for your application, which is something you almost 
 certainly do not want to do. Why do you believe you have this requirement?

Re: Measuring WAN replication latency

[Rahul Neelakantan]

Agreed... This is what we are trying right now.

Rahul Neelakantan

 On Jul 30, 2014, at 1:43 PM, Jeremy Jongsma jer...@barchart.com wrote:
 
 Yes, the results should definitely not be relied on as a future performance 
 indicator for key app functionality. but knowing roughly what your current 
 replication latency is (and whether it's outside of the normal average) can 
 inform client failover policies, debug data consistency issues, warn of 
 datacenter link congestion, etc.
 
 
 On Wed, Jul 30, 2014 at 12:02 PM, Robert Coli rc...@eventbrite.com wrote:
 On Wed, Jul 30, 2014 at 6:59 AM, Rahul Neelakantan ra...@rahul.be wrote:
 
 Any ideas you can provide on how to do this will be appreciated, we would 
 like to build a latency monitoring tool/dashboard that shows how long it 
 takes for data to get sent across various DCs.
 
 The brute force method described downthread by Jeremy Jongsma gives you 
 something like the monitoring you're looking for, but I continue to believe 
 it's probably a bad idea to try to design a system in this way.
 
 =Rob
 

Question about Vnodes

[Rahul Neelakantan]

Given the issue with repairs and Vnodes (currently expected to be fixed with 
the 3.0 release) I am considering reducing the number of tokens per node. One 
of my clusters has 8 nodes in it with 256 tokens per node. The main keyspace on 
it has 40+ column families and nodetool repair takes extremely long to complete 
on that keyspace.

I have two questions with respect to this
1) Is there a way/formula to determine an appropriate number of tokens per 
node. I am considering going as low as 32 or even 16 tokens per node.
2) What is the recommended procedure for reducing the number of Vnodes? Do I 
just reduce it in the cassandra.yaml and start removing tokens via nodetool?

Rahul Neelakantan

Re: unable to load data using sstableloader

[Rahul Neelakantan]

Is SStable loader being run on the same host as the Cassandra node 127.0.0.1 (I 
see your ring is made up of all loop back IPs).

Rahul Neelakantan
678-451-4545

 On Jul 29, 2014, at 4:55 AM, Akshay Ballarpure akshay.ballarp...@tcs.com 
 wrote:
 
 Thanks Duncan for quick help 
 now i am trying sstableloader but its not loading data...any suggestion ? 
 
 
 [root@CSL-simulation conf]# ps -eaf | grep -i cass 
 root 20322 1  0 Jul28 ?00:04:27 
 /usr/java/jdk1.7.0_60/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar 
 -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities 
 -XX:ThreadPriorityPolicy=42 -Xms7G -Xmx7G -Xmn1800M 
 -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true 
 -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.rmi.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -Dlog4j.configuration=log4j-server.properties 
 -Dlog4j.defaultInitOverride=true -cp 
 ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.8.jar:./../lib/apache-cassandra-clientutil-2.0.8.jar:./../lib/apache-cassandra-thrift-2.0.8.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang3-3.1.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-15.0.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.1.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.2.0.jar:./../lib/metrics-core-2.2.0.jar:./../lib/netty-3.6.6.Final.jar:./../lib/reporter-config-2.1.0.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/
  
 snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/super-csv-2.1.0.jar:./../lib/thrift-server-internal-only-0.3.3.jar
  org.apache.cassandra.service.CassandraDaemon 
 root 28246 26448  0 14:24 pts/300:00:00 grep -i cass 
 [root@CSL-simulation conf]# ../bin/sstableloader sample_new/data_new -d 
 localhost 
 Established connection to initial hosts 
 Opening sstables and calculating sections to stream 
 Streaming relevant part of 
 sample_new/data_new/sample_new-data_new-jb-1-Data.db to [/127.0.0.1] 
 Streaming session ID: 05c15ca0-16fe-11e4-bdb8-731053160c4f 
 progress: [/127.0.0.1 1/1 (100%)] [total: 100% - 0MB/s (avg: 0MB/s)] WARN 
 14:25:00,535 [Stream #05c15ca0-16fe-11e4-bdb8-731053160c4f] Stream failed 
 Streaming to the following hosts failed: 
 [/127.0.0.1] 
 java.util.concurrent.ExecutionException: 
 org.apache.cassandra.streaming.StreamException: Stream failed 
 [root@CSL-simulation conf]# ../bin/sstableloader sample_new/data_new -d 
 127.0.0.1 
 Established connection to initial hosts 
 Opening sstables and calculating sections to stream 
 Streaming relevant part of 
 sample_new/data_new/sample_new-data_new-jb-1-Data.db to [/127.0.0.1] 
 Streaming session ID: 161819e0-16fe-11e4-8377-731053160c4f 
 progress: [/127.0.0.1 1/1 (100%)] [total: 100% - 0MB/s (avg: 0MB/s)] WARN 
 14:25:27,957 [Stream #161819e0-16fe-11e4-8377-731053160c4f] Stream failed 
 Streaming to the following hosts failed: 
 [/127.0.0.1] 
 java.util.concurrent.ExecutionException: 
 org.apache.cassandra.streaming.StreamException: Stream failed 
 
 Thanks  Regards
 Akshay Ghanshyam Ballarpure
 Tata Consultancy Services
 Cell:- 9985084075
 Mailto: akshay.ballarp...@tcs.com
 Website: http://www.tcs.com
 
 Experience certainty.IT Services
Business Solutions
Consulting
  
 
 
 From: Duncan Sands duncan.sa...@gmail.com
 To:   user@cassandra.apache.org
 Date: 07/29/2014 12:58 PM
 Subject:  Re: unable to load data using sstableloader
 
 
 
 
 Hi Akshay,
 
 On 29/07/14 09:14, Akshay Ballarpure wrote:
  Yes,
  I have created keyspaces, but still i am getting error.
 
  cqlsh:sample_new DESCRIBE KEYSPACES ;
 
  system  sample  mykeyspace  test *sample_new* system_traces
 
  [root@CSL-simulation conf]# ../bin/sstableloader
  /root/Akshay/Cassandra/apache-cassandra-2.0.8/conf/SAMPLE_NEW/DATA_NEW/  -d
  localhost --debug
  Could not retrieve endpoint ranges:
  InvalidRequestException(why:No such keyspace: *SAMPLE_NEW*)
 
 here SAMPLE_NEW is in upper case, while describe keyspaces showed it lower 
 case, 
 sample_new.  Try using quote marks:
 
   CREATE KEYSPACE SAMPLE_NEW ...
 
 Ciao, Duncan

Re: unable to load data using sstableloader

[Rahul Neelakantan]

Sstabloader and Cassandra can't use the same network interface, clearly 
mentioned in the Cassandra documentation on DataStax. If you are running both 
Cassandra and SStableloader locally, you have to take some extra steps. Look at 
the post.

pushpalankajaya.blogspot.com/2012/06/bulk-loading-data-to-cassandra-using.html?m=1



Rahul Neelakantan

 On Jul 29, 2014, at 4:55 AM, Akshay Ballarpure akshay.ballarp...@tcs.com 
 wrote:
 
 Thanks Duncan for quick help 
 now i am trying sstableloader but its not loading data...any suggestion ? 
 
 
 [root@CSL-simulation conf]# ps -eaf | grep -i cass 
 root 20322 1  0 Jul28 ?00:04:27 
 /usr/java/jdk1.7.0_60/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar 
 -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities 
 -XX:ThreadPriorityPolicy=42 -Xms7G -Xmx7G -Xmn1800M 
 -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true 
 -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.rmi.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -Dlog4j.configuration=log4j-server.properties 
 -Dlog4j.defaultInitOverride=true -cp 
 ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.8.jar:./../lib/apache-cassandra-clientutil-2.0.8.jar:./../lib/apache-cassandra-thrift-2.0.8.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang3-3.1.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-15.0.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.1.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.2.0.jar:./../lib/metrics-core-2.2.0.jar:./../lib/netty-3.6.6.Final.jar:./../lib/reporter-config-2.1.0.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/
  
 snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/super-csv-2.1.0.jar:./../lib/thrift-server-internal-only-0.3.3.jar
  org.apache.cassandra.service.CassandraDaemon 
 root 28246 26448  0 14:24 pts/300:00:00 grep -i cass 
 [root@CSL-simulation conf]# ../bin/sstableloader sample_new/data_new -d 
 localhost 
 Established connection to initial hosts 
 Opening sstables and calculating sections to stream 
 Streaming relevant part of 
 sample_new/data_new/sample_new-data_new-jb-1-Data.db to [/127.0.0.1] 
 Streaming session ID: 05c15ca0-16fe-11e4-bdb8-731053160c4f 
 progress: [/127.0.0.1 1/1 (100%)] [total: 100% - 0MB/s (avg: 0MB/s)] WARN 
 14:25:00,535 [Stream #05c15ca0-16fe-11e4-bdb8-731053160c4f] Stream failed 
 Streaming to the following hosts failed: 
 [/127.0.0.1] 
 java.util.concurrent.ExecutionException: 
 org.apache.cassandra.streaming.StreamException: Stream failed 
 [root@CSL-simulation conf]# ../bin/sstableloader sample_new/data_new -d 
 127.0.0.1 
 Established connection to initial hosts 
 Opening sstables and calculating sections to stream 
 Streaming relevant part of 
 sample_new/data_new/sample_new-data_new-jb-1-Data.db to [/127.0.0.1] 
 Streaming session ID: 161819e0-16fe-11e4-8377-731053160c4f 
 progress: [/127.0.0.1 1/1 (100%)] [total: 100% - 0MB/s (avg: 0MB/s)] WARN 
 14:25:27,957 [Stream #161819e0-16fe-11e4-8377-731053160c4f] Stream failed 
 Streaming to the following hosts failed: 
 [/127.0.0.1] 
 java.util.concurrent.ExecutionException: 
 org.apache.cassandra.streaming.StreamException: Stream failed 
 
 Thanks  Regards
 Akshay Ghanshyam Ballarpure
 Tata Consultancy Services
 Cell:- 9985084075
 Mailto: akshay.ballarp...@tcs.com
 Website: http://www.tcs.com
 
 Experience certainty.IT Services
Business Solutions
Consulting
  
 
 
 From: Duncan Sands duncan.sa...@gmail.com
 To:   user@cassandra.apache.org
 Date: 07/29/2014 12:58 PM
 Subject:  Re: unable to load data using sstableloader
 
 
 
 
 Hi Akshay,
 
 On 29/07/14 09:14, Akshay Ballarpure wrote:
  Yes,
  I have created keyspaces, but still i am getting error.
 
  cqlsh:sample_new DESCRIBE KEYSPACES ;
 
  system  sample  mykeyspace  test *sample_new* system_traces
 
  [root@CSL-simulation conf]# ../bin/sstableloader
  /root/Akshay/Cassandra/apache-cassandra-2.0.8/conf/SAMPLE_NEW/DATA_NEW/  -d
  localhost --debug
  Could not retrieve endpoint ranges:
  InvalidRequestException(why:No such keyspace: *SAMPLE_NEW*)
 
 here

Measuring WAN replication latency

[Rahul Neelakantan]

Does anyone know of a way to measure/monitor WAN replication latency for 
Cassandra? For example I make a write to DC 1 of a multi DC setup using 
local_quorum , how long before that write is available for a read either 
locally in DC1 or remotely in DC2 using local_quorum.

Rahul Neelakantan

Re: Measuring WAN replication latency

[Rahul Neelakantan]

Here is the use case ... The Cassandra ring is going to span 6 data centers. 
The client can write in any data center (the client is also in 6 DCs) and turn 
back around and read from any other data center for this I assume you will tell 
me to use each quorum for the writes, however that is going to increase the 
time it takes for writes to be acknowledged. 

What I am looking for is a way to monitor either using some statistic exposed 
by Cassandra or using some sort of synthetic transaction of how my Cassandra 
replication looks and alert if there is slowness noticed.

Rahul Neelakantan

 On Jul 29, 2014, at 8:53 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Tue, Jul 29, 2014 at 3:15 PM, Rahul Neelakantan ra...@rahul.be wrote:
 Does anyone know of a way to measure/monitor WAN replication latency for 
 Cassandra?
 
 No. [1]
 
 =Rob
  
 [1] There are ways to do something like this task, but you probably don't 
 actually want to do them. Trying to do them suggests that you are relying on 
 WAN replication timing for your application, which is something you almost 
 certainly do not want to do. Why do you believe you have this requirement?

Cassandra version upgrade path

[Rahul Neelakantan]

Is there a tool/guide/document that shows you upgrade paths for different 
versions of Cassandra?

For example if you are on version X and want to go to version Y, here are the 
intermediate versions you need to upgrade to and here are some special 
precautions/Steps you need to take for so and so version while upgrading 

Rahul Neelakantan
678-451-4545

How to get rid of stale info in gossip

[Rahul Neelakantan]

Is there a way to get rid of stale information that shows up for removed/dead 
nodes in gossip, without a complete cluster bounce?

Rahul Neelakantan

Re: How to get rid of stale info in gossip

[Rahul Neelakantan]

Yes, and this is a really old version of casandra 1.0.8.


Rahul Neelakantan
678-451-4545

 On Jul 25, 2014, at 7:29 AM, Mark Reddy mark.re...@boxever.com wrote:
 
 After removing a node, it's information can persist in the Gossiper for up to 
 3 days, after which time it should be removed. 
 
 Are you having issues with a removed node state persisting for longer?
 
 
 Mark
 
 
 On Fri, Jul 25, 2014 at 11:33 AM, Rahul Neelakantan ra...@rahul.be wrote:
 Is there a way to get rid of stale information that shows up for 
 removed/dead nodes in gossip, without a complete cluster bounce?
 
 Rahul Neelakantan
 

Changing IPs of all nodes in a ring

[Rahul Neelakantan]

All,
I need to change the IPs of all nodes in my ring in a flash cut, at the same 
time. Any recommendations on how to do this?

Rahul Neelakantan

Re: Changing IPs of all nodes in a ring

[Rahul Neelakantan]

I am ok with taking upto 2 hours of planned downtime. The problem is all the 
IPs will change at the same time and the previous IPs will no longer be 
available. So it's either all old IPs or all new IPs. 

Rahul Neelakantan
678-451-4545

 On Jul 25, 2014, at 7:23 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Fri, Jul 25, 2014 at 3:54 PM, Rahul Neelakantan ra...@rahul.be wrote:
 I need to change the IPs of all nodes in my ring in a flash cut, at the same 
 time. Any recommendations on how to do this?
 
 What are your uptime requirements as you do this?
 
 Because no, there's no way to change the ip address on all Cassandra nodes in 
 a cluster simultaneously and have it stay available.
 
 https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/
 
 Is how you do it with a rolling restart.
 
 =Rob
 

Re: Changing IPs of all nodes in a ring

[Rahul Neelakantan]

The new IPs are not available before switch time. So I will try the all down 
method you mentioned. 

Do I need to do any move tokens of the new IPs to the old token -1 and the 
removeToken of the old tokens?  I ask this because the old IPs will continue to 
show in gossip info with a status of Normal.



Rahul Neelakantan
678-451-4545

 On Jul 25, 2014, at 9:06 PM, Robert Coli rc...@eventbrite.com wrote:
 
 On Fri, Jul 25, 2014 at 4:42 PM, Rahul Neelakantan ra...@rahul.be wrote:
 I am ok with taking upto 2 hours of planned downtime. The problem is all the 
 IPs will change at the same time and the previous IPs will no longer be 
 available. So it's either all old IPs or all new IPs. 
 
 Are the new IPs available before switchover time? If so, you can switch to 
 them with a rolling restart. If not :
 
 1) down entire cluster
 2) change ips in cassandra.yaml, including in seed list
 3) use auto_bootstrap:false technique, as in blog post, to bring them all back
  
 =Rob