Re: RE: Re: Re: High mutation stage in multi dc deployment

This is sufficiently atypical that many people aren't going to have enough intuition to figure it out without seeing your metrics / logs / debugging data (e.g. heap dumps). My only guess, and it's a pretty big guess, is that your write timeout is low enough (or network quality bad enough, though

RE: RE: Re: Re: High mutation stage in multi dc deployment

Kindly help in this regard. What could be the possible reason for load and mutation spike in india data center On 2021/07/20 00:14:56 MyWorld wrote: > Hi Arvinder, > It's a separate cluster. Here max partition size is 32mb. > > On 2021/07/19 23:57:27 Arvinder Dhillon wrote: > > Is this the same

RE: Re: Re: High mutation stage in multi dc deployment

Hi Arvinder, It's a separate cluster. Here max partition size is 32mb. On 2021/07/19 23:57:27 Arvinder Dhillon wrote: > Is this the same cluster with 1G partition size? > > -Arvinder > > On Mon, Jul 19, 2021, 4:51 PM MyWorld wrote: > > > Hi daemeon, > > We have already tuned the TCP settings to

Re: Re: High mutation stage in multi dc deployment

Is this the same cluster with 1G partition size? -Arvinder On Mon, Jul 19, 2021, 4:51 PM MyWorld wrote: > Hi daemeon, > We have already tuned the TCP settings to improve the bandwidth. Earlier > we had lot of hint and mutation msg drop which were gone after tuning TCP. > Moreover we are

RE: Re: High mutation stage in multi dc deployment

Hi daemeon, We have already tuned the TCP settings to improve the bandwidth. Earlier we had lot of hint and mutation msg drop which were gone after tuning TCP. Moreover we are writing with CL local quorum at US side, so ack is taken from local DC. I m still concern what could be reason of increase

RE: Re: High mutation stage in multi dc deployment

Hi Patrick, Currently we are using 3.11.6 apache cassandra version. We are performing write with CL local quorum in US side DC. We have 4-5 tables with supplier_details, supplier_prod_details, supplier_rating. We also have an mview attached with rating table. For batching part, I need to check

Re: High mutation stage in multi dc deployment

Hi Ashish, Can you give us some information about some of the details? Specifically some indication on the version of Cassandra, data model, consistency levels used, and how you are bulk loading. Is this a batch by any chance? Patrick On Mon, Jul 19, 2021 at 10:38 AM MyWorld wrote: > Hi all,

Re: High mutation stage in multi dc deployment

You may want to think about the latency impacts of a cluster that has one node "far away". This is such a basic design flaw that you need to do some basic learning, and some basic understanding of networking and latency. On Mon, Jul 19, 2021 at 10:38 AM MyWorld wrote: > Hi all, > >

High mutation stage in multi dc deployment

Hi all, Currently we have a cluster with 2 DC of 3 nodes each. One DC is in GCP-US while other is in GCP-India. Just to add here, configuration of every node accross both DC is same. Cpu-6, Ram-32gb, Heap-8gb We do all our write on US data center. While performing a bulk write on GCP US, we

Re: number of racks in a deployment with VMs

The reason for the zoho (and my) emails go to the spam box is because the Apache mailing list software is messing around with the DKIM signature and the "From:" address. I have created INFRA-21415 for this. On 15/02/2021 22:36, Kane Wilson

Re: number of racks in a deployment with VMs

There are operational advantages to having #racks == RF, however it's by no means mandatory. Having more racks than RF doesn't cause any availability/health/balance problems, it is only disadvantageous in that it makes some cluster maintenance tasks more expensive/unwieldy like repairs and DC

number of racks in a deployment with VMs

Hi, In a article by thelastpickle [1], i noticed: The key here is to configure the cluster so that for a given datacenter the number of racks is the same as the replication factor. When using virtual machines as Cassandra nodes we have to set up the cluster in a way that number of racks

Re: Deployment

ich node I meant), but obviously keeping application in same region as that of cassandra cluster would make sense, can't comment much on cloud deployment. Thanks,Rajesh On Sat, Jan 12, 2019 at 8:54 AM amit sehas wrote: I am new to Cassandra, i am wondering how the Cassandra applications are deplo

Re: Deployment

use it would have > bigger implications when stuffs like re-balancing would occur. So, your > application should be unaware where your data lies (in which node I meant), > but obviously keeping application in same region as that of cassandra > cluster would make sense, can't comment much

Re: Deployment

lication should be unaware where your data lies (in which node I meant), but obviously keeping application in same region as that of cassandra cluster would make sense, can't comment much on cloud deployment. Thanks,Rajesh On Sat, Jan 12, 2019 at 8:54 AM amit sehas wrote: I am new to Cassandra

Re: Deployment

your application should be unaware where your data lies (in which node I meant), but obviously keeping application in same region as that of cassandra cluster would make sense, can't comment much on cloud deployment. Thanks, Rajesh On Sat, Jan 12, 2019 at 8:54 AM amit sehas wrote: > I am new

Re: Deployment

In the cloud world all the Apps and servers are deployed independently. Having said that application has to submit request to post or get the data it’s the Cassandra takes care of routing it internally to get the data. Yes it’s better to have on same cloud for better performance. Hope you got some

Deployment

I am new to Cassandra, i am wondering how the Cassandra applications are deployed in the cloud. Does Cassandra have a client server architecture and the application is deployed as a 3rd tier that sends over queries to the clients, which then submit them to the Cassandra servers?  Or does the

Re: tokens and RF for multiple phases of deployment

, May 31, 2012 at 5:48 AM, aaron morton aa...@thelastpickle.comwrote: Could you provide some guide on how to assign the tokens in this growing deployment phases? background http://www.datastax.com/docs/1.0/install/cluster_init#calculating-tokens-for-a-multi-data-center-cluster Start

Re: tokens and RF for multiple phases of deployment

Could you provide some guide on how to assign the tokens in this growing deployment phases? background http://www.datastax.com/docs/1.0/install/cluster_init#calculating-tokens-for-a-multi-data-center-cluster Start with tokens for a 4 node cluster. Add the next 4 between between each

Re: tokens and RF for multiple phases of deployment

On Thu, May 31, 2012 at 5:48 AM, aaron morton aa...@thelastpickle.comwrote: Could you provide some guide on how to assign the tokens in this growing deployment phases? background http://www.datastax.com/docs/1.0/install/cluster_init#calculating-tokens-for-a-multi-data-center-cluster Start

Re: tokens and RF for multiple phases of deployment

...@thelastpickle.com wrote: Could you provide some guide on how to assign the tokens in this growing deployment phases? background http://www.datastax.com/docs/1.0/install/cluster_init#calculating-tokens-for-a-multi-data-center-cluster Start with tokens for a 4 node cluster. Add the next 4 between

tokens and RF for multiple phases of deployment

or on maintenance. Could you provide some guide on how to assign the tokens in this growing deployment phases? I looked at some docs but not very clear on how to assign tokens on the fail-over case. Also if we use the same RF (3) in both DC, and use EACH_QUORUM for write and LOCAL_QUORUM for read, can

Remote Hadoop Job Deployment

FYI... we finally got around to releasing a version of Virgil that includes the ability to deploy jobs to remote Hadoop clusters running against Cassandra Column Families. http://brianoneill.blogspot.com/2012/01/virgil-remote-hadoop-job-deployment-via.html This has enabled an army of people

Re: advice for EC2 deployment

private IP's of EC2 nodes I am facing problems. I am assuming if Cassandra is built for multi-DC setup it should be easily deployed with node1's DC1's public IP listed as seed in all nodes in DC2 and to gain idea about network topology? I have hit a dud for deployment in such scenario

Re: advice for EC2 deployment

nodes I am facing problems. I am assuming if Cassandra is built for multi-DC setup it should be easily deployed with node1's DC1's public IP listed as seed in all nodes in DC2 and to gain idea about network topology? I have hit a dud for deployment in such scenario

Re: advice for EC2 deployment

by just using public-dns? I am also looking into open vpn and how to deploy it. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/advice-for-EC2-deployment-tp6294613p6508278.html Sent from the cassandra-u...@incubator.apache.org mailing list

Re: advice for EC2 deployment

there nodes talking to each other across regions by just using public-dns? I am also looking into open vpn and how to deploy it. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/advice-for-EC2-deployment-tp6294613p6508278.html Sent from

Re: advice for EC2 deployment

we use a combination of Vyatta OpenVPN on the nodes that are EC2 and nodes that aren't Ec2works a treat. On Thu, Jun 23, 2011 at 10:23 PM, Sameer Farooqui cassandral...@gmail.com wrote: EC2Snitch doesn't currently support multi-Regions in Amazon. Tickets to track:

Re: advice for EC2 deployment

If you are not going to be multi-region straight away, but wish to be in the near future I would consider: - 1 region - 2 AZ's, with the same number of nodes - Using the EC2Snitch as is, this will map to 1 cassandra DC and 2 cassandra Racks - Using the NetworkTopology strategy For background

Re: advice for EC2 deployment

's public IP listed as seed in all nodes in DC2 and to gain idea about network topology? I have hit a dud for deployment in such scenario. Or is it there any way possible to use Private IP's for such a scenario in EC2, as Public Ip are less secure and costly? -- Will Oberman Civic

Re: advice for EC2 deployment

It's great advice, but I'm still torn. I've never done multi-region work before, and I'd prefer to wait for 0.8 with built-in inter-node security, but I'm otherwise ready to roll (and need to roll) cassandra out sooner than that. Given how well my system held up with a total single AZ failure,

Re: advice for EC2 deployment

Hi William, The default behavior of Ec2Snitch is outlined below: http://svn.apache.org/repos/asf/cassandra/trunk/src/java/org/apache/cassandra/locator/Ec2Snitch.java // Split us-east-1a or asia-1a into us-east/1a and asia/1a. String azone = new String(b ,UTF-8); String[]

Re: advice for EC2 deployment

if you migrate the instance, does Route53 automatically re-map all the information to the new ec2 instance? another issue is that cassandra only maintains the IP of the other nodes, and not the hostname (assumed based on output of the nodetool ring) ... which means, if you migrate the instance

Re: advice for EC2 deployment

Thanks Sasha. Fortunately/unfortunately I did realize the default current behavior of the Ec2Snitch, but my application isn't multi-region capable (yet), so I need to get intra-region redundancy. And having a SingleRegionEc2Snitch that did DC=ec2zone and RACK=??? would be much better for me

Re: advice for EC2 deployment

I don't think of it as migrating an instance, it's more of a destroy/start with EC2. But, I still think it would be very useful to spin up a set of instances with known hostnames (cassandra1, 2, 3... N) and be able to quickly SSH to them by doing ssh ec2u...@cassandra1.random.ec2.mydomain.com .

Re: advice for EC2 deployment

Oh, and Route53 doesn't do anything automatically, but there is an API to manage the DNS. It's up to you to run a task on instance boot/terminate, or a cron job if you want to do this trick (for now, seems like a solid future feature of Route53). Though, I hear geographical aware Route53 is

Re: advice for EC2 deployment

so can you not simply leverage a strategy that replicates data between racks and at some point in the future when you move to multi-dc upgrade the replication strategy to maintain the current replication and add in some replication between DC's ... ? i'll go re-read your posts to see if you've

Re: advice for EC2 deployment

I think you're right about changing NetworkToplogyStrategy, but the timing isn't working in my favor at this point. I wonder how bad that will really be On Wed, Apr 27, 2011 at 9:35 AM, Sasha Dolgy sdo...@gmail.com wrote: so can you not simply leverage a strategy that replicates data

Re: advice for EC2 deployment

Thanks Aaron! Unless no one on this list uses EC2, there were a few minor troubles end of last week through the weekend which taught me a lot about obscure failure modes in various applications I use :-) My original post was trying to be more redundant than fast, which has been by overall goal

Re: advice for EC2 deployment

One difference between Cassandra and MySQL replication may be when the network IO happens. Was the MySQL replication synchronous on transaction commit ? I was only aware that it had async replication, which means the client is not exposed to the network latency. In cassandra the network

Re: advice for EC2 deployment

I see what you're saying. I was able to control write latency on mysql using insert vs insert delayed (what I feel is MySQLs poor man's eventual consistency option) + the fact that replication was a background asynchronous process. In terms of read latency, I was able to do up to a few hundred

Re: Multi-DC Deployment

Cassandra doesn't replicate sstable corruptions. It detects corrupt data and only replicates good data. This is incorrect. Depending on the nature of the corruption it may spread to other nodes. Checksumming (done right) would be a great addition to alleiate this. Yes, there is code that tries

Re: Multi-DC Deployment

Assuming that you generally put an API on top of this, delivering to two or more systems then boils down to a message queue issue or some similar mechanism which handles secure delivery of messages. Maybe not trivial, but there are many products that can help you with this, and it is a lot easier

Re: Multi-DC Deployment

Queues replicate bad data just as well as anything else. The biggest source of bad data is broken app code... You will still need to implement a reconciliation/repair checker, as queues have their own failure modes when they get backed up. We have also looked at using queues to bounce data between

Re: Multi-DC Deployment

Hum... Seems like it could be an idea in a case like this with a mode where result is always returned (if possible), but where a flay saying if the consistency level was met, or to what level it was met (number of nodes answering for instance).? Terje On Tue, Apr 19, 2011 at 1:13 AM, Jonathan

Re: Multi-DC Deployment

If you want to use local quorum for a distributed setup, it doesn't make sense to have less than RF=3 local and remote. Three copies at both ends will give you high availability. Only one copy of the data is sent over the wide area link (with recent versions). There is no need to use mirrored or

Re: Multi-DC Deployment

If you have RF=3 in both datacenters, it could be discussed if there is a point to use the built in replication in Cassandra at all vs. feeding the data to both datacenters and get 2 100% isolated cassandra instances that cannot replicate sstable corruptions between each others My point is

Multi-DC Deployment

We are planning to deploy Cassandra on two data centers. Let us say that we went with three replicas with 2 being in one data center and last replica in 2nd Data center. What will happen to Quorum Reads and Writes when DC1 goes down (2 of 3 replicas are unreachable)? Will they timeout?

Re: Multi-DC Deployment

They will timeout until failure detector realizes the DC1 nodes are down (~10 seconds). After that they will immediately return UnavailableException until DC1 comes back up. On Mon, Apr 18, 2011 at 10:43 AM, Baskar Duraikannu baskar.duraikannu...@gmail.com wrote: We are planning to deploy

Re: Cassandra 2 DC deployment

You are right about the automatic fallback to ONE. Its quite possible, if 2 nodes die for some reason I will have the same problem. So probably the right thing to do would be to read/write at ONE only when we lose a DC by changing some manual configuration. Since we shouldn't be losing DCs

RE: Cassandra 2 DC deployment

@cassandra.apache.org Subject: Re: Cassandra 2 DC deployment When the down data center comes back up, the Quorum reads will result in a read-repair, so you will get valid data.   Besides that, hinted handoff will take care of getting data replicated to a previously down node. *Eventually* though

Cassandra 2 DC deployment

Hi experts, We are planning to deploy Cassandra in 2 datacenters. Let assume there are 3 nodes, RF=3, 2 nodes in 1 DC and 1 node in 2nd DC. Under normal operations, we would read and write at QUORUM. What we want to do though is if we lose a datacenter which has 2 nodes, DC1 in this case, we

Re: Cassandra 2 DC deployment

When the down data center comes back up, the Quorum reads will result in a read-repair, so you will get valid data. Besides that, hinted handoff will take care of getting data replicated to a previously down node. You're example is a little unrealistic because you could theoretically have a

Re: Cassandra 2 DC deployment

I think this is reasonable assuming you have enough backhaul to perform reads across DC if read requests hit DC2 (with one copy of data) or one replica from DC1 is down. Moreover, since you clearly stated that you would prefer availability over consistency, you should be prepared for stale reads

Re: Cassandra 2 DC deployment

When the down data center comes back up, the Quorum reads will result in a read-repair, so you will get valid data.   Besides that, hinted handoff will take care of getting data replicated to a previously down node. *Eventually* though, but yes. I.e., there would be no expectation to

Re: Few questions regarding cassandra deployment on windows

On Tue, Sep 7, 2010 at 17:42, kannan chandrasekaran ckanna...@yahoo.com wrote: Hi All, We are currently considering Cassandra for our application. Platform: * a single-node cluster. * windows '08 * 64-bit jvm For the sake of brevity let, Cassandra service =  a single node cassandra

Re: Few questions regarding cassandra deployment on windows

Thanks for the replies My comments in Bold... Kannan From: Gary Dusbabek gdusba...@gmail.com To: user@cassandra.apache.org Sent: Thu, September 9, 2010 5:43:31 AM Subject: Re: Few questions regarding cassandra deployment on windows On Tue, Sep 7, 2010 at 17:42, kannan chandrasekaran ckanna

Re: Few questions regarding cassandra deployment on windows

for this limitation ? If yes, Can someone please help me understand the reason... Thanks Kannan From: Courtney sa...@live.co.uk To: user@cassandra.apache.org Sent: Tue, September 7, 2010 5:31:46 PM Subject: Re: Few questions regarding cassandra deployment

Few questions regarding cassandra deployment on windows

Hi All, We are currently considering Cassandra for our application. Platform: * a single-node cluster. * windows '08 * 64-bit jvm For the sake of brevity let, Cassandra service = a single node cassandra server running as an embedded service inside a JVM My use cases: 1) Start with a

Re: Few questions regarding cassandra deployment on windows

This does not sound like a good application for Cassandra at all. Why are you using it? On Tue, Sep 7, 2010 at 3:42 PM, kannan chandrasekaran ckanna...@yahoo.com wrote: Hi All, We are currently considering Cassandra for our application. Platform: * a single-node cluster. * windows '08 *

Re: Few questions regarding cassandra deployment on windows

: Tue, September 7, 2010 4:38:41 PM Subject: Re: Few questions regarding cassandra deployment on windows This does not sound like a good application for Cassandra at all. Why are you using it? On Tue, Sep 7, 2010 at 3:42 PM, kannan chandrasekaran ckanna...@yahoo.com wrote: Hi All, We are currently

Re: Few questions regarding cassandra deployment on windows

, he did it, so ... From: kannan chandrasekaran Sent: Wednesday, September 08, 2010 1:20 AM To: user@cassandra.apache.org Subject: Re: Few questions regarding cassandra deployment on windows Can you please elaborate on why you think Cassandra would not be suitable for this ? Main reasons why

Re: Deployment on AWS and replication strategies

On Sat, Apr 3, 2010 at 8:23 PM, Mike Gallamore mike.e.gallam...@googlemail.com wrote: I didn't mean a real time determination, more of if the nodes aren't identical. For example if you have a cluster made up of a bunch of EC2 light instances and decide to add a large instance, it would be

Re: Deployment on AWS

On 2010-04-04, at 10:18 PM, Masood Mortazavi wrote: (My question remains. I'm interested in seed configuration practice/recipe when deploying on AWS. In the scenario, assume Cassandra sits behind some other part of the service -- say, web container -- that are then exposed publicly.

Re: Deployment on AWS

Woot. Ver much looking forward to this stuff Joe. On Sat, Apr 3, 2010 at 10:14 AM, Joe Stump j...@joestump.net wrote: On Apr 2, 2010, at 4:49 PM, Masood Mortazavi wrote: Is there a ready recipe for deploying a Cassandra cluster in AWS? ... (Seeds need some fixed IP addresses.) We have a

Re: Deployment on AWS

On Apr 3, 2010, at 1:53 PM, Benjamin Black wrote: What specific features are you looking for to operate on EC2? It seemed people weren't looking for features, but tools to help with the management. The two things we've created that people might be interested in are: 1. An EC2-specific

Re: Deployment on AWS

I'm pretty familiar with EC2, hence the question. I don't believe any patches are required to do these things. Regardless, as I noted in that ticket, you definitely do NOT need AWS credentials to determine your availability zone. It is available through the metadata web server for each instance

Re: Deployment on AWS

On Apr 3, 2010, at 2:54 PM, Benjamin Black wrote: I'm pretty familiar with EC2, hence the question. I don't believe any patches are required to do these things. Regardless, as I noted in that ticket, you definitely do NOT need AWS credentials to determine your availability zone. It is

Re: Deployment on AWS and replication strategies

Hi everyone, At my work we are in the early stages of moving our data which lives on EC2 machines from a Flare/memcache system to Cassandra so your chat has been interesting to me. I realize that this might complicate things and make things less simple but would it be useful for the nodes

Re: Deployment on AWS

Right, you determine AZ by looking at the metadata. us-east-1a is a different AZ from us-east-1b. You can't infer anything beyond that, either with the AWS API or guesses about IP addressing. My EC2 snitch recipe builds a config file for the property snitch that treats AZs like racks (just

Separate disks with cloud deployment

The FAQ page makes mention of using separate disks for the commit log and data directory. How would one go about achieving this in a cloud deployment such as Rackspace cloud servers or EC2 EBS? Or is it just preferred to use dedicated hardware to get the optimal performance? Thanks In Advance

Re: Separate disks with cloud deployment

On 03/25/2010 11:10 AM, Mark Greene wrote: The FAQ page makes mention of using separate disks for the commit log and data directory. How would one go about achieving this in a cloud deployment such as Rackspace cloud servers or EC2 EBS? Or is it just preferred to use dedicated hardware to get

Re: Separate disks with cloud deployment

On 03/25/2010 11:18 AM, Ethan Rowe wrote: [snip] I'll defer to the Rackspace folks regarding Rackspace Cloud; it has been I/O on average since you're dealing with a real, local disk. But I don't know about getting a second disk in that environment, though. That should have said better I/O

Re: Separate disks with cloud deployment

as sharing the machine with a bunch of other users, so I wouldn't worry too much. On Thu, Mar 25, 2010 at 10:10 AM, Mark Greene green...@gmail.com wrote: The FAQ page makes mention of using separate disks for the commit log and data directory. How would one go about achieving this in a cloud deployment