Re: failure node rejoin

OK, that’s a bit more unexpected (to me at least) but I think the solution of running a rebuild or repair still applies. On Tue, 18 Oct 2016 at 15:45 Yuji Ito wrote: > Thanks Ben, Jeff > > Sorry that my explanation confused you. > > Only node1 is the seed node. > Node2

Re: failure node rejoin

Thanks Ben, Jeff Sorry that my explanation confused you. Only node1 is the seed node. Node2 whose C* data is deleted is NOT a seed. I restarted the failure node(node2) after restarting the seed node(node1). The restarting node2 succeeded without the exception. (I couldn't restart node2 before

Re: failure node rejoin

The unstated "problem" here is that node1 is a seed, which implies auto_bootstrap=false (can't bootstrap a seed, so it was almost certainly setup to start without bootstrapping). That means once the data dir is wiped, it's going to start again without a bootstrap, and make a single node

Re: failure node rejoin

OK, sorry - I think understand what you are asking now. However, I’m still a little confused by your description. I think your scenario is: 1) Stop C* on all nodes in a cluster (Nodes A,B,C) 2) Delete all data from Node A 3) Restart Node A 4) Restart Node B,C Is this correct? If so, this isn’t

Re: What is the point of counter type when we can do the same thing with int or bigint?

The only way to do this in Cassandra now is with counters - whether you add 1 or n, it's counters or ugly read before write and lightweight transactions. Counters give Cassandra the closest thing they'll ever have to vector clocks - they give the user a way to do commutative deltas in a

Re: What is the point of counter type when we can do the same thing with int or bigint?

Also are you saying counters are atomic? On Mon, Oct 17, 2016 at 6:43 PM, Kant Kodali wrote: > How about “Set the value 1 above what it is now" ? The same principle > should apply right? > > > > > On Mon, Oct 17, 2016 at 6:21 PM, Jeff Jirsa >

Re: What is the point of counter type when we can do the same thing with int or bigint?

How about “Set the value 1 above what it is now" ? The same principle should apply right? On Mon, Oct 17, 2016 at 6:21 PM, Jeff Jirsa wrote: > You cant use int/bigint to say “Set the value 2 above what it is now”, > unless you use a read to get the current value,

Re: What is the point of counter type when we can do the same thing with int or bigint?

You cant use int/bigint to say “Set the value 2 above what it is now”, unless you use a read to get the current value, then write using lightweight transactions, which have a significant performance penalty. The primary reason for this is because no individual Cassandra node is guaranteed

What is the point of counter type when we can do the same thing with int or bigint?

I just read the following link https://docs.datastax.com/en/cql/3.3/cql/cql_using/useCounters.html and I wonder what is the point of counter type when we can do the same thing with int or bigint? what are benefits of using counter data type?

Re: Adding disk capacity to a running node

Note that ebs iops scale with disk capacity – gp2 volumes < 3334 GB have “burst” capacity that you can exhaust. Practically speaking, this may not matter to you, but if you chose a very small volume size (say, 500G), your baseline throughput may be much lower than you anticipate.

Re: Adding disk capacity to a running node

Thanks for the detailed steps Ben! This gives me another option in case of emergency. On Mon, Oct 17, 2016 at 1:55 PM, Ben Bromhead wrote: > yup you would need to copy the files across to the new volume from the dir > you wanted to give additional space to. Rough steps

Re: Adding disk capacity to a running node

yup you would need to copy the files across to the new volume from the dir you wanted to give additional space to. Rough steps would look like: 1. Create EBS volume (make it big... like 3TB) 2. Attach to instance 3. Mount/format EBS volume 4. Stop C* 5. Copy full/troublesome

Re: Adding disk capacity to a running node

@Ben Interesting idea, is this also an option for situations where the disk is completely full and Cassandra has stopped? (Not that I want to go there). If this was the route taken, and we did mount --bind /mnt/path/to/large/sstable /mnt/newebs We would still need to do some manual copying

Re: Adding disk capacity to a running node

I've had luck using the st1 EBS type, too, for situations where reads are rare (the commit log still needs to be on its own high IOPS volume; I like using ephemeral storage for that). On Mon, Oct 17, 2016 at 3:03 PM, Branton Davis wrote: > I doubt that's true anymore.

Re: Adding disk capacity to a running node

Yup as everyone has mentioned ephemeral are fine if you run in multiple AZs... which is pretty much mandatory for any production deployment in AWS (and other cloud providers) . i2.2xls are generally your best bet for high read throughput applications on AWS. Also on AWS ephemeral storage will

Re: Adding disk capacity to a running node

Ephemeral is fine, you just need to have enough replicas (in enough AZs and enough regions) to tolerate instances being terminated. From: Vladimir Yudovin Reply-To: "user@cassandra.apache.org" Date: Monday, October 17, 2016 at 11:48 AM

Re: Adding disk capacity to a running node

There are, of course, people using EBS successfully, I didn't say there weren't and it wasn't my point. I was merely saying the reasoning to avoid ephemeral disk because your instance is going to move between machines and lose data is nonsense, in that they work just fine and have been heavily

Re: Adding disk capacity to a running node

If a node is restarted is not moved, no. That's not how it works. On Mon, Oct 17, 2016 at 12:01 PM Vladimir Yudovin wrote: > But after such restart node should be joined to cluster again and restore > data, right? > > Best regards, Vladimir Yudovin, > > > *Winguzone

Re: Adding disk capacity to a running node

I doubt that's true anymore. EBS volumes, while previously discouraged, are the most flexible way to go, and are very reliable. You can attach, detach, and snapshot them too. If you don't need provisioned IOPS, the GP2 SSDs are more cost-effective and allow you to balance IOPS with cost. On

Re: Adding disk capacity to a running node

But after such restart node should be joined to cluster again and restore data, right? Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra on Azure and SoftLayer. Launch your cluster in minutes. On Mon, 17 Oct 2016 14:55:49 -0400Jonathan Haddad j...@jonhaddad.com

Re: Adding disk capacity to a running node

Vladimir, *Most* people are running Cassandra are doing so using ephemeral disks. Instances are not arbitrarily moved to different hosts. Yes, instances can be shut down, but that's why you distribute across AZs. On Mon, Oct 17, 2016 at 11:48 AM Vladimir Yudovin wrote: >

Re: Adding disk capacity to a running node

It's extremely unreliable to use ephemeral (local) disks. Even if you don't stop instance by yourself, it can be restarted on different server in case of some hardware failure or AWS initiated update. So all node data will be lost. Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud

Re: Adding disk capacity to a running node

These are i2.2xlarge instances so the disks currently configured as ephemeral dedicated disks. On Mon, Oct 17, 2016 at 11:34 AM, Laing, Michael wrote: > You could just expand the size of your ebs volume and extend the file > system. No data is lost - assuming you are

Re: Adding disk capacity to a running node

You could just expand the size of your ebs volume and extend the file system. No data is lost - assuming you are running Linux. On Monday, October 17, 2016, Seth Edwards wrote: > We're running 2.0.16. We're migrating to a new data model but we've had an > unexpected increase in

Re: Adding disk capacity to a running node

We're running 2.0.16. We're migrating to a new data model but we've had an unexpected increase in write traffic that has caused us some capacity issues when we encounter compactions. Our old data model is on STCS. We'd like to add another ebs volume (we're on aws) to our JBOD config and hopefully

Re: Cassandra installation best practices

Hi Mehdi, You can refer  https://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettings.html  . ThanksAnuj On Mon, 17 Oct, 2016 at 10:20 PM, Mehdi Bada wrote: Hi all, It is exist some best practices when installing Cassandra on production

Re: Cassandra installation best practices

Hi, you can take a look on http://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettingsLinux.html Regarding file system type I guess ext4 is good enough. (Though RedHat now use XFS as default FS). Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra on

Cassandra installation best practices

Hi all, It is exist some best practices when installing Cassandra on production environment? Some standard to follow? For instance, the file system type etc..

Re: failure node rejoin

The exception you run into is expected behavior. This is because as Ben pointed out, when you delete everything (including system schemas), C* cluster thinks you're bootstrapping a new node. However, node2's IP is still in gossip and this is why you see the exception. I'm not clear the reasoning

Re: Adding disk capacity to a running node

I assume you're talking about Cassandra JBOD (just a bunch of disk) setup because you do mention it as adding it to the list of data directories. If this is the case, you may run into issues, depending on your C* version. Check this out: http://www.datastax.com/dev/blog/improving-jbod. Or another

Re: Inconsistencies in materialized views

https://issues.apache.org/jira/browse/CASSANDRA-11198 Which has problems "maybe" fixed by: https://issues.apache.org/jira/browse/CASSANDRA-11475 Which has it's own set of problems. One of these patches was merged into 3.7 which tells you are running a version 3.6 with known bugs. Also as the

Re: Adding disk capacity to a running node

Yes, Cassandra should keep percent of disk usage equal for all disk. Compaction process and SSTable flushes will use new disk to distribute both new and existing data. Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra on Azure and SoftLayer. Launch your cluster in minutes.

Adding disk capacity to a running node

We have a few nodes that are running out of disk capacity at the moment and instead of adding more nodes to the cluster, we would like to add another disk to the server and add it to the list of data directories. My question, is, will Cassandra use the new disk for compactions on sstables that

Re: Run sstablescrub in parallel

Thanks for response! It's just seemed to me that sstables are processed so independently, that there should be a workaround with sstablescrub. StandaloneScrubber.java#L120

Re: Scenarios when blocking read repair takes place

Sorry Krishna, didn't get were you were trying to say. Regards SIddharth Verma (Visit https://github.com/siddv29/cfs for a high speed cassandra full table scan) On Sat, Oct 15, 2016 at 11:50 PM, Krishna Chandra Prajapati < prajapat...@gmail.com> wrote: > Hi which side is this? > Mankapur? >

Inconsistencies in materialized views

Hi, We have a base table with ~300 million entries. And in a recent sanity activity, I saw approx ~33k entires(in one DC) which were in the materialized view, but not in the base table. (reads with quorum, DCAware) (I haven't done it the other way round yet, i.e. entries in base table but not in