Re: Query timeouts after Cassandra Migration
Ankit, are the instance types identical in the new cluster, with I/O configuration identical at the system level, and are the Java settings for C* identical between the two clusters? With radical timing differences happening periodically, the two things I’d have on my radar would be garbage collections and problems in flushing dirty pages. Even if neither of those are the issue, one way or another, timeouts make me hunt for the resource everybody is queued up on. From: Erick Ramirez Reply-To: "user@cassandra.apache.org" Date: Thursday, February 6, 2020 at 10:08 PM To: "user@cassandra.apache.org" Subject: Re: Query timeouts after Cassandra Migration Message from External Sender So do you advise copying tokens in such cases ? What procedure is advisable ? Specifically for your case with 3 nodes + RF=3, it won't make a difference so leave it as it is. Latency increased on target cluster. Have you tried to run a trace of the queries which are slow? It will help you identify where the slowness is coming from. Cheers!
Re: Query timeouts after Cassandra Migration
> > So do you advise copying tokens in such cases ? What procedure is > advisable ? > Specifically for your case with 3 nodes + RF=3, it won't make a difference so leave it as it is. > Latency increased on target cluster. > Have you tried to run a trace of the queries which are slow? It will help you identify where the slowness is coming from. Cheers!
Re: Query timeouts after Cassandra Migration
Thanks Eric. So do you advise copying tokens in such cases ? What procedure is advisable ? Latency increased on target cluster. I’d double check on storage disks but it should be same. — Ankit On Thu, Feb 6, 2020 at 9:07 PM Erick Ramirez wrote: > I didn’t copy tokens since it’s an identical cluster and we have RF as 3 >> on 3 node cluster. Is it still needed , why? >> > > In C*, same number of nodes alone isn't enough. Clusters aren't really > identical unless token assignments are the same. In your case though since > each node has a full copy of the data (RF = N nodes), they "appear" > identical. > > I recently migrated Cassandra keyspace data from one Azure cluster (3 >> Nodes) to another (3 nodes different region) using simple sstable copy. >> Post this , we are observing overall response time has increased and >> timeouts every 20 mins. >> > > You mean the response time on the source cluster increased? Or the > destination cluster? I can't see how the copy could affect latency unless > you're using premium storage disks and you've maxed out the throughput on > them. For example, P30 disks are capped at 200MB/s. > > Do I need to copy anything from system* > > > No, system tables are local to a node. Only ever copy the application > keyspaces. Cheers! > -- *Thanks & Regards,* *Ankit Gadhiya*
Re: Query timeouts after Cassandra Migration
> > I didn’t copy tokens since it’s an identical cluster and we have RF as 3 > on 3 node cluster. Is it still needed , why? > In C*, same number of nodes alone isn't enough. Clusters aren't really identical unless token assignments are the same. In your case though since each node has a full copy of the data (RF = N nodes), they "appear" identical. I recently migrated Cassandra keyspace data from one Azure cluster (3 > Nodes) to another (3 nodes different region) using simple sstable copy. > Post this , we are observing overall response time has increased and > timeouts every 20 mins. > You mean the response time on the source cluster increased? Or the destination cluster? I can't see how the copy could affect latency unless you're using premium storage disks and you've maxed out the throughput on them. For example, P30 disks are capped at 200MB/s. Do I need to copy anything from system* No, system tables are local to a node. Only ever copy the application keyspaces. Cheers!
Re: Query timeouts after Cassandra Migration
Hi Michael, Thanks for your response. I didn’t copy tokens since it’s an identical cluster and we have RF as 3 on 3 node cluster. Is it still needed , why? Don’t see anything in cassandra log as such. I don’t have debugs enabled. Thanks & Regards, Ankit On Thu, Feb 6, 2020 at 1:47 PM Michael Shuler wrote: > Did you copy the tokens from cluster1 to new cluster2? Same Cassandra > version, same instance type/size? What to the logs say on cluster2 that > look different from the cluster1 norm? There are a number of possible > `nodetool` utilities that may help see what is happening on new cluster2. > > Michael > > On 2/6/20 8:09 AM, Ankit Gadhiya wrote: > > Hi Folks, > > > > I recently migrated Cassandra keyspace data from one Azure cluster (3 > > Nodes) to another (3 nodes different region) using simple sstable copy. > > Post this , we are observing overall response time has increased and > > timeouts every 20 mins. > > > > Has anyone faced such in their experiences ? > > Do I need to copy anything from system* > > Anything wrt statistics/cache ? > > > > Your time and responses on this are much appreciated. > > > > > > Thanks & Regards, > > Ankit > > -- > > *Thanks & Regards,* > > *Ankit Gadhiya* > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- *Thanks & Regards,* *Ankit Gadhiya*
Re: Query timeouts after Cassandra Migration
Did you copy the tokens from cluster1 to new cluster2? Same Cassandra version, same instance type/size? What to the logs say on cluster2 that look different from the cluster1 norm? There are a number of possible `nodetool` utilities that may help see what is happening on new cluster2. Michael On 2/6/20 8:09 AM, Ankit Gadhiya wrote: Hi Folks, I recently migrated Cassandra keyspace data from one Azure cluster (3 Nodes) to another (3 nodes different region) using simple sstable copy. Post this , we are observing overall response time has increased and timeouts every 20 mins. Has anyone faced such in their experiences ? Do I need to copy anything from system* Anything wrt statistics/cache ? Your time and responses on this are much appreciated. Thanks & Regards, Ankit -- *Thanks & Regards,* *Ankit Gadhiya* - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Query timeouts after Cassandra Migration
Hi Folks, I recently migrated Cassandra keyspace data from one Azure cluster (3 Nodes) to another (3 nodes different region) using simple sstable copy. Post this , we are observing overall response time has increased and timeouts every 20 mins. Has anyone faced such in their experiences ? Do I need to copy anything from system* Anything wrt statistics/cache ? Your time and responses on this are much appreciated. Thanks & Regards, Ankit -- *Thanks & Regards,* *Ankit Gadhiya*
