Have you run Cassandra 3.11.x on Alma9 or Alma8

2024-02-01 Thread Surbhi Gupta
Hi, Has any of you run Cassandra 3.11.x on Alma9 or Alma8? Any issues or concerns? We are going to upgrade from cent 7 to Alma8 or 9 , and wanted to understand if there is/are any known issue? Thanks Surbhi

Re: Repair errors

2023-08-11 Thread Surbhi Gupta
ntNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:474) >> at >> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor.lambda$execute$0(ClientNotifForwarder.java:108) >> at java.base/java.lang.Thread.run(Thread.java:829)

Re: Upgrade from 3.11.5 to 4.1.x

2023-08-10 Thread Surbhi Gupta
ase notes. > > However for production, I would recommend to go for 4.0.x latest stable > version. > > Regards > Ashish > > On Sat, 8 Jul, 2023, 05:44 Surbhi Gupta, wrote: > >> Hi, >> >> We have to upgrade from 3.11.5 to 4.1.x . >> Can we do it in one

Re: Materialized View inconsistency issue

2023-08-10 Thread Surbhi Gupta
l proposals avoiding MV failed for our project. Basically managing a > table like a MV (by deleting and inserting rows from code) is worse and more > corrupted than what MV does... > The worse issue is adding and removing nodes. Maybe cassandra 4 improves this > point (not tested yet). &

Materialized View inconsistency issue

2023-08-08 Thread Surbhi Gupta
Hi, We get complaints about Materialized View inconsistency issues. We are on 3.11.5 and on 3.11.5 Materialized Views were not production ready. We are ok to upgrade. On which version of cassandra MVs doesnt have inconsistency issues? Thanks Surbhi

Re: Repair errors

2023-08-04 Thread Surbhi Gupta
Can you please try to do nodetool describecluster from every node of the cluster? One time I noticed issue when nodetool status shows all nodes UN but describecluster was not. Thanks Surbhi On Fri, Aug 4, 2023 at 8:59 AM Joe Obernberger wrote: > Hi All - been using reaper to do repairs, but

Upgrade from 3.11.5 to 4.1.x

2023-07-07 Thread Surbhi Gupta
Hi, We have to upgrade from 3.11.5 to 4.1.x . Can we do it in one go ? Or do we have to go to an intermediate version first? Thanks Surbhi

Is there a way to find out if a server is part of application connection string?

2023-06-06 Thread Surbhi Gupta
Hi, We have a cluster with many applications connecting to it. We need to decommission few of the servers from the cluster . Without asking the application team, is there any way to know the ips of the application connection string? Does cassandra logs (system or debug) this information

Re: Bootstrapping new node throwing error - Mutation too large

2023-03-01 Thread Surbhi Gupta
> released about 3.5 years ago, with a large number of bugfixes available in > 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing > the version and performing a rolling restart of the instances. > > > – Scott > > On Mar 1, 2023, at 2:43 PM, Surbhi G

Bootstrapping new node throwing error - Mutation too large

2023-03-01 Thread Surbhi Gupta
Hi Cassandra Community, We have to expand our cluster and I tried to add the first node to the cluster and when the new node was bootstrapping , I noticed the error like below in the system.log, but the bootstrap process was successful . We are on 3.11.5 . ERROR [MutationStage-7] 2023-03-01

Re: Anyone connecting the Cassandra on a server

2021-11-19 Thread Surbhi Gupta
You can use tcpdump On Fri, 19 Nov 2021 at 10:34, Soumya Jena wrote: > You can just do a netstat on port 9042 to see if anything connected . > > Something like > netstat -anp | grep 9042 . > > Or you can also check for read/write client requests metrics . You can > check if specific tables are

Re: Streaming failure Issue

2021-10-05 Thread Surbhi Gupta
Hi , Try to adjust phi_convict_threshold and see if that helps. When we did migration from on prim to AWS, this was one of the factor to consider. Thanks On Tue, Oct 5, 2021 at 4:00 AM MyWorld wrote: > Hi all, > > Need urgent help. > We have one Physical Data Center of 5 nodes with 1 TB data

Re: Hints are not getting partially processed

2021-09-26 Thread Surbhi Gupta
I tried nodetool sethintedhandoffthrottlekb 0;nodetool pausehandoff;sleep 15;nodetool resumehandoff; But still the same On Sun, 26 Sept 2021 at 08:25, Surbhi Gupta wrote: > All nodes are up and running . > Checked system.log and debug.log but nothing useful i got it. > > >

Re: Hints are not getting partially processed

2021-09-26 Thread Surbhi Gupta
All nodes are up and running . Checked system.log and debug.log but nothing useful i got it. On Sun, 26 Sept 2021 at 08:03, Surbhi Gupta wrote: > I tried truncatehints and some of the hints file did not delete but some > hints file got deleted. > > > On Sun, 26 Sept 2021 at 04:0

Re: Hints are not getting partially processed

2021-09-26 Thread Surbhi Gupta
I tried truncatehints and some of the hints file did not delete but some hints file got deleted. On Sun, 26 Sept 2021 at 04:00, Surbhi Gupta wrote: > Hi, > > We are on 3.11.5 and we have 2 DC cassandra cluster. > Suddenly we started seeing hints issue . > system.logs

Hints are not getting partially processed

2021-09-26 Thread Surbhi Gupta
Hi, We are on 3.11.5 and we have 2 DC cassandra cluster. Suddenly we started seeing hints issue . system.logs says that hints are getting partially replayed and we are seeing hints dropped in target nodes after hinted_handoff_period is over. We checked debug.log as well but nothing we found

Which open source or free tool do you use to monitor cassandra clusters?

2021-06-16 Thread Surbhi Gupta
Hi, Which open source or free tool do you use to monitor cassandra clusters which have similar features like Opscenter? Thanks Surbhi

Re: Memory requirements for Cassandra reaper

2021-05-06 Thread Surbhi Gupta
Thanks a lot. On Tue, 4 May 2021 at 19:51, Erick Ramirez wrote: > 2GB is allocated to the Reaper JVM on startup (see > https://github.com/thelastpickle/cassandra-reaper/blob/2.2.4/src/packaging/bin/cassandra-reaper#L90-L91 > ). > > If you just want to test it out on a machine with only 8GB, you

Memory requirements for Cassandra reaper

2021-05-04 Thread Surbhi Gupta
Hi, What are the memory requirements for Cassandra reaper? I was trying to setup cassandra reaper on a 8GB box where cassandra is taking 3GB heap size , but i got error "Cannot allocate memory" Hence wanted to understand the memory requirements for cassandra reaper . What should be the size of

Dont want to split sstables for repaired and non repaired while repairing with -pr option

2021-03-24 Thread Surbhi Gupta
Hi, I dont want to split sstables ,repaired and non repaired , while repairing with -pr option. nodetool repair -pr splits the sstable into repaired and non repaired and disk size increases. I dont want to increase the disk size. What are my options ? Thanks Surbhi

Re: Best strategy to run repair

2021-03-22 Thread Surbhi Gupta
quite >> problematic. >> >> Subrange is the way to go, which is what cassandra-reaper will do for you >> if you have it set up. >> >> raft.so - Cassandra consulting, support, and managed services >> >> >> On Tue, Mar 23, 2021 at 7:33 AM Surbh

Best strategy to run repair

2021-03-22 Thread Surbhi Gupta
Hi, We are on open source 3.11.5 . We need to repair a production cluster . We are using num_token as 256 . What will be a better option to run repair ? 1. nodetool -pr (Primary range repair on all nodes, one node at a time) OR 2. nodetool -st -et (Subrange repair , taking the ranges for each

Re: Rollback Cassandra after 1 node upgrade

2020-09-04 Thread Surbhi Gupta
Hi Manish, Please provide both versions. Thanks Surbhi On Fri, Sep 4, 2020 at 8:55 PM manish khandelwal < manishkhandelwa...@gmail.com> wrote: > Hi > > We have been forced into rolling back our Cassandra after 1 node upgrade. > The node was upgraded 10 days ago. We have the backup of the old

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

2020-06-23 Thread Surbhi Gupta
3.11.1 > to 3.11.6 without having had any single issue for over a year. I just > upgraded all nodes to 3.11.6 for the sake of consistency. > > Von meinem iPhone gesendet > > Am 24.06.2020 um 02:56 schrieb Surbhi Gupta : > >  > > Hi , > > We have recently upgraded

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

2020-06-23 Thread Surbhi Gupta
Hi , We have recently upgraded from 3.11.0 to 3.11.5 . There is a sstable format change from 3.11.4 . We also had to expand the cluster and we also discussed about expansion first and than upgrade. But finally we upgraded and than expanded. As per our experience what I could tell you is, it is

Bootstrap resume , streamed all data again and 2nd bootstrap id in netstats

2020-06-04 Thread Surbhi Gupta
Hi, We are on 3.11.5 . We are trying to add a node in a DC and after all the streaming is done, no streaming is active in nodetool netstats output , the node was just waiting for 1 hour doing nothing. So we thought it might be hung, so we tried nodetool bootstrap resume But bootstrap resume ,

Re: Truncate Materialized View

2020-05-15 Thread Surbhi Gupta
> What exact error got ? If you think it is same as the bug, then you may > try to avoid the bug triggered condition. It says pending hints. So you may > let all hints applied, then try drop the view. > > Thanks, > > James > > On Fri, May 15, 2020 at 1:35 PM Surbh

Re: Truncate Materialized View

2020-05-15 Thread Surbhi Gupta
Anyone has truncated materialized views ? On Thu, 14 May 2020 at 11:59, Surbhi Gupta wrote: > Hi, > > We are on 3.11.0 . > We have 11 Materialized view on a table. > After discussion with application team , we found out that they are using > only 4 out of 11 .

Truncate Materialized View

2020-05-14 Thread Surbhi Gupta
Hi, We are on 3.11.0 . We have 11 Materialized view on a table. After discussion with application team , we found out that they are using only 4 out of 11 . We tried to drop the materialized view and got hit by the bug https://issues.apache.org/jira/browse/CASSANDRA-13696 which made our whole

Add a new node of 3.11.5 in a 3.11.0 Cassandra Cluster

2020-05-09 Thread Surbhi Gupta
Hi, We are facing some issue in bootstrapping new node in 3.11.0 and bootstrapping is failing. We have two tasks here : 1. Expand the cluster (Due to disk concern and dropped mutation) 2. Upgrade the cluster from 3.11.0 to 3.11.5 because of various bugs we are hitting in 3.11.0 . So my question

Re: Bootstraping is failing

2020-05-09 Thread Surbhi Gupta
d, what I wanted to understand , how tcp settings can effect the bootstrapping process ? Thanks Surbhi On Thu, 7 May 2020 at 17:01, Surbhi Gupta wrote: > When we are starting the node, it is starting bootstrap automatically and > restreaming the whole data again. It is not resuming . > &

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
detool/bootstrap.html) > to pick up where it last left off. Sorry for the late reply. > > > On Thu, May 7, 2020 at 2:22 PM Surbhi Gupta > wrote: > >> So after failed bootstrapped , if we start cassandra again on the new >> node , will it resume bootstrap or will it start

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
gt; > On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta > wrote: > >> streaming_socket_timeout_in_ms is 24 hour. >> So tcp settings should be changed on the new bootstrap node or on all >> nodes ? >> >> >> On Thu, 7 May 2020 at 13:23, Adam Scott wrote: >>

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
ive_intvl=10* > then run sysctl -p to cause the kernel to reload the settings > > 5 minutes (300) seconds is probably too long. > > On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta > wrote: > >> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time >> >>

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
ion? > > What does the following show? > cat /proc/sys/net/ipv4/tcp_keepalive_time > cat /proc/sys/net/ipv4/tcp_keepalive_intvl > cat /proc/sys/net/ipv4/tcp_keepalive_probes > > On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta > wrote: > >> Hi, >> >> We are

Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
Hi, We are trying to expand a datacenter and trying to add nodes but when node is bootstrapping , it goes half way through and then fail with below error, We have increased stremthroughput from 200 to 400 when we were trying for the 2nd time but still it failed. We are on 3.11.0 , using G1GC with

Re: OOM only on one datacenter nodes

2020-04-05 Thread Surbhi Gupta
unts objects by total retained size. Take a screenshot. > Send that. > > > > On Apr 5, 2020, at 6:51 PM, Surbhi Gupta wrote: > >  > I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. > > I checked the CPU and RAM both are same on all the nodes in DC

Re: OOM only on one datacenter nodes

2020-04-05 Thread Surbhi Gupta
I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. I checked the CPU and RAM both are same on all the nodes in DC1 and DC2. What specific parameter I should check on OS ? We are using CentOS release 6.10. Currently disk_access_modeis not set hence it is auto in our env.

OOM only on one datacenter nodes

2020-04-04 Thread Surbhi Gupta
Hi, We have two datacenter with 5 nodes each and have replication factor of 3. We have traffic on DC1 and DC2 is just for disaster recovery and there is no direct traffic. We are using 24cpu with 128GB RAM machines . For DC1 where we have live traffic , we don't see any issue, however for DC2

Re: Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

2020-03-09 Thread Surbhi Gupta
On Mon, 9 Mar 2020 at 09:36, Surbhi Gupta wrote: > > https://javadoc.io/static/org.apache.cassandra/cassandra-all/3.11.4/constant-values.html#org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.MAX_TERM_SIZE > The MAX_TERM_SIZE value is 1024 > > Can we change it ? > > > T

Re: Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

2020-03-09 Thread Surbhi Gupta
would like to understand the impact of this rejection. On Mon, 9 Mar 2020 at 08:47, Surbhi Gupta wrote: > We have SASI index . > Any solution ? > > On Thu, 5 Mar 2020 at 15:20, Surbhi Gupta > wrote: > >> Hi, >> >> We are in process of upgrading from 3.

Re: Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

2020-03-09 Thread Surbhi Gupta
We have SASI index . Any solution ? On Thu, 5 Mar 2020 at 15:20, Surbhi Gupta wrote: > Hi, > > We are in process of upgrading from 3.11.0 to 3.115 . > While upgrading SSTables we are noticing messages like below in system.log. > What are the significance of these mes

Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

2020-03-05 Thread Surbhi Gupta
Hi, We are in process of upgrading from 3.11.0 to 3.115 . While upgrading SSTables we are noticing messages like below in system.log. What are the significance of these messages? INFO [CompactionExecutor:3] 2020-03-05 16:12:41,393 PerSSTableIndexWriter.java:211 - Rejecting value (size 1.022KiB,

Downgrading from 3.11.5 to 3.11.0

2020-03-04 Thread Surbhi Gupta
Hi, As the SSTable file formats have changed from 3.11.4 to "md " https://docs.datastax.com/en/landing_page/doc/landing_page/compatibility.html We are going to take snapshots but still wanted to understand . After we do upgrades stable when we upgrade from 3.11.0 to 3.11.5 , and later in future

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
Application team confirmed that they are * not* referencing the dropped MVs anywhere for reading or writing On Tue, 18 Feb 2020 at 22:25, Surbhi Gupta wrote: > So should upgrading to 3.11.1 will solve this issue? > > On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta > wrote: >

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
So should upgrading to 3.11.1 will solve this issue? On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta wrote: > Thanks Eric... > > On Tue, 18 Feb 2020 at 22:06, Erick Ramirez > wrote: > >> Just to add to my above point because here we are dropping MV not a >>> regul

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
Thanks Eric... On Tue, 18 Feb 2020 at 22:06, Erick Ramirez wrote: > Just to add to my above point because here we are dropping MV not a >> regular table. >> And MV does read before write , Is this the reason we are seeing the >> below message? Trying to understand >> >> WARN

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
Just to add to my above point because here we are dropping MV not a regular table. And MV does read before write , Is this the reason we are seeing the below message? Trying to understand WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - Failed to read a hint for

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
Hi Eric, As per https://issues.apache.org/jira/browse/CASSANDRA-13696 , this issue happens even with write traffic "I did more investigation today. Seems it's more serious than I thought. Even there's no down node, "drop table" + write traffic, will trigger the problem." Thanks Surbhi On Tue,

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
3.11.0 in dev/test and prod . Just a thought On Tue, 18 Feb 2020 at 19:49, Surbhi Gupta wrote: > We are Cassandra 3.11.0 unfortunately :( > > On Tue, 18 Feb 2020 at 19:41, Erick Ramirez > wrote: > >> Clearly the hint error invoked the fs error handler - probably >>>

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
We are Cassandra 3.11.0 unfortunately :( On Tue, 18 Feb 2020 at 19:41, Erick Ramirez wrote: > Clearly the hint error invoked the fs error handler - probably incorrectly >> - which shut down the db. That’s not ok and deserves a JIRA. >> > > It's supposed to have been fixed by CASSANDRA-13696 in

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
Thanks Eric, Let me go back to the app team On Tue, Feb 18, 2020 at 6:49 PM Erick Ramirez wrote: > We are on cassandra 3.11 , we are using G1GC and using 16GB of heap. >> > > Which exact version of C* is it again? > >> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115 >>

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
rn others. > > The version would have been 3.0.15 or 3.11.3 as that is what we were > deploying on our clusters at the time. I think it was more likely 3.0.15. > > So sorry for the "vagueness" :( > > On Tue, Feb 18, 2020, 8:54 PM Surbhi Gupta > wrote: > >>

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
> We now disallow the use of MVs globally. > > On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta > wrote: > >> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap. >> >> So we had to drop 7 MVs in production, as soon as we dropped the first >> M

Re: Consequences of dropping Materialized views

2020-02-18 Thread Surbhi Gupta
://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta wrote: > Thanks Eric ... > This is helpful... > > > On Wed, 12 Feb 2020 at 17:46, Erick Ramirez > wrote: > >> There shoul

Re: Consequences of dropping Materialized views

2020-02-12 Thread Surbhi Gupta
astax.com | datastax.com <http://www.datastax.com> > <https://www.linkedin.com/company/datastax> > <https://www.facebook.com/datastax> <https://twitter.com/datastax> > <http://feeds.feedburner.com/datastax> <https://github.com/datastax/> > > <htt

Consequences of dropping Materialized views

2020-02-12 Thread Surbhi Gupta
Hi, So application team created 11 materialized views on a base table in production and we need to drop 7 Materialized views as they are not in use. Wanted to understand the impact of dropping the materialized views. We are on Cassandra 3.11.1 , multi datacenter with replication factor of 3 in

Re: Overload because of hint pressure + MVs

2020-02-11 Thread Surbhi Gupta
We are using G1 ... On Tue, 11 Feb 2020 at 08:51, Reid Pinchback wrote: > A caveat to the 31GB recommendation for G1GC. If you have tight latency > SLAs instead of throughput SLAs then this doesn’t necessary pan out to be > beneficial. > > > > Yes the GCs are less frequent, but they can hurt

Re: Overload because of hint pressure + MVs

2020-02-10 Thread Surbhi Gupta
Just to add , we are using 24GB heap size. On Mon, 10 Feb 2020 at 09:08, Surbhi Gupta wrote: > Hi Jon, > > We are on multi datacenter(On Prim) setup. > We also noticed too many messages like below: > > DEBUG [GossipStage:1] 2020-02-10 09:38:52,953 FailureDetector.java:457 - &

Re: Overload because of hint pressure + MVs

2020-02-10 Thread Surbhi Gupta
even when we can see that all nodes are UP ? Recommended value of phi_convict_threshold is 12 in AWS multi datacenter environment. Thanks Surbhi On Sun, 9 Feb 2020 at 21:42, Surbhi Gupta wrote: > Thanks a lot Jon.. > Will try the recommendations and let you know the results > >

Re: Overload because of hint pressure + MVs

2020-02-09 Thread Surbhi Gupta
s >> a fixed duplicate in JIRA somewhere). >> >> For clarification, you could update that ticket with a comment including >> your environmental details, usage of MV, etc. I'll bump the priority up >> and include some possible branchX fixvers. >> >> Michael &g

Overload because of hint pressure + MVs

2020-02-07 Thread Surbhi Gupta
Hi, We are getting hit by the below bug. Other than lowering hinted_handoff_throttle_in_kb to 100 any other work around ? https://issues.apache.org/jira/browse/CASSANDRA-13810 Any idea if it got fixed in later version. We are on Open source Cassandra 3.11.1 . Thanks Surbhi

Re: Nodes becoming unresponsive

2020-02-06 Thread Surbhi Gupta
I have limited options to use JDK based tools because in our environment we are running JRE . I tried to debug more and could see using top that Command is MutationStage in top output , Any clue we get from this ? top - 16:30:47 up 94 days, 5:33, 1 user, load average: 134.83, 142.48, 144.75

Re: Nodes becoming unresponsive

2020-02-05 Thread Surbhi Gupta
Sure Eric... I tried strace as well ...

Nodes becoming unresponsive

2020-02-05 Thread Surbhi Gupta
Hi, We have noticed in a Cassandra Cluster , one of the node has 100% cpu utilization, using top we can see that cassandra process is showing futex_wait . We are on CentOS release 6.10 (Final) .As per below document the futex bug was on Centos 6.6 .

Re: How to read content of hints file and apply them manually?

2020-01-28 Thread Surbhi Gupta
and high cpu issue ? On Tue, Jan 28, 2020 at 1:12 PM Patrick McFadin wrote: > I would definitely check the IO stats then, If you see latency going over > 20ms, you need to solve that problem. > > Patrick > > On Tue, Jan 28, 2020 at 12:01 PM Surbhi Gupta > wrote: > >&

Re: How to read content of hints file and apply them manually?

2020-01-28 Thread Surbhi Gupta
d at > the same time? These are exactly the symptoms I see when running Cassandra > on a SAN or NAS. > > Patrick > > On Mon, Jan 27, 2020 at 8:17 PM Surbhi Gupta > wrote: > > We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but > nothing helped .

Re: How to read content of hints file and apply them manually?

2020-01-27 Thread Surbhi Gupta
We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but nothing helped . Our hints related parameters are as below, if you don't find any parameter below then it is not set in our environment and should be of the default value. max_hint_window_in_ms: 1080 # 3 hours

Re: How to read content of hints file and apply them manually?

2020-01-27 Thread Surbhi Gupta
a fix to the underlying issue. Run a > full repair. > On Monday, January 27, 2020, 10:17:01 p.m. UTC, Surbhi Gupta < > surbhi.gupt...@gmail.com> wrote: > > > Hi, > > We are on Open source 3.11 . > We have a issue in one of the cluster where lots of hints gets piled up > and th

How to read content of hints file and apply them manually?

2020-01-27 Thread Surbhi Gupta
Hi, We are on Open source 3.11 . We have a issue in one of the cluster where lots of hints gets piled up and they don't get applied within hinted handoff period ( 3 hour in our case) . And load and CPU of the server goes very high. We see lot of messages in system.log and debug.log . Our read

Re: Cassandra is not showing a node up hours after restart

2019-11-24 Thread Surbhi Gupta
Before Cassandra shutdown, nodetool drain should be executed first. As soon as you do nodetool drain, others node will see this node down and no new traffic will come to this node. I generally gives 10 seconds gap between nodetool drain and Cassandra stop. On Sun, Nov 24, 2019 at 9:52 AM Paul

Re: Cassandra is not showing a node up hours after restart

2019-11-24 Thread Surbhi Gupta
It sounds silly but sometimes restarting again the node which is showing down from other nodes fix the issue. This looks like a gossip issue. On Sun, Nov 24, 2019 at 7:19 AM Paul Mena wrote: > I am in the process of doing a rolling restart on a 4-node cluster running > Cassandra 2.1.9. I

Re: Cassandra-stress testing

2019-08-20 Thread Surbhi Gupta
Have you tried ycsa? It is a tool from yahoo for stress testing nosql databases. On Tue, Aug 20, 2019 at 3:34 AM wrote: > Hi Everyone, > > > > Anyone before who have bused Cassandra-stress. I want to test if it’s > possible to load 600 milllions records per hour in Cassandra or > > Find a

Re: can i...

2019-03-07 Thread Surbhi Gupta
Send the details On Thu, Mar 7, 2019 at 8:45 AM Nick Hatfield wrote: > Use this email to get some insight on how to fix database issues in our > cluster? >

Re: Cassandra | Cross Data Centre Replication Status

2018-10-31 Thread Surbhi Gupta
Repair will take way more time then rebuild. On Wed, Oct 31, 2018 at 6:45 AM Kiran mk wrote: > Run the repair with -pr option on each node which will repair only the > > parition range. > > > > nodetool repair -pr > > On Wed, Oct 31, 2018 at 7:04 PM Surbhi Gupta &g

Re: Cassandra | Cross Data Centre Replication Status

2018-10-31 Thread Surbhi Gupta
Nodetool repair will take way more time than nodetool rebuild. How much data u have in your original data center? Repair should be run to make the data consistent in case of node down more than hintedhandoff period and dropped mutations. But as a thumb rule ,generally we run repair using opscenter

Re: Read timeouts when performing rolling restart

2018-09-12 Thread Surbhi Gupta
Another thing to notice is : system_auth WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} system_auth has a replication factor of 1 and even if one node is down it may impact the system because of the replication factor. On Wed, 12 Sep 2018 at 09:46, Steinmaurer,

Re: nodetool rebuild

2018-09-12 Thread Surbhi Gupta
Increase 3 throughput Compaction throughput Stream throughput Interdcstream throughput (if rebuilding from another DC) Make all of the above to 0 and see if there is any improvement and later set the value if u can’t leave these values to 0. On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk wrote:

Re: Log application Queries

2018-05-25 Thread Surbhi Gupta
nodetool settraceprobability 1 > and it is logging in system_traces. > > How is it different from nodeool setlogginglevel? > > > Regards, > Nitan K. > Cassandra and Oracle Architect/SME > Datastax Certified Cassandra expert > Oracle 10g Certified > > On Fri, May 25, 2018 at 11:

Re: Log application Queries

2018-05-25 Thread Surbhi Gupta
If using dse then u can enable in dse.yaml. # CQL slow log settings cql_slow_log_options: enabled: true threshold_ms: 0 ttl_seconds: 259200 As far as my understanding says setlogginglevel is used for changing the logging level as below but not for slow query . - ALL - TRACE - DEBUG

Re: Question About Reaper

2018-05-24 Thread Surbhi Gupta
efaultManagersHolder$2.run(SSLContextImpl.java:823) On 24 May 2018 at 14:12, Dennis Lovely <d...@aegisco.com> wrote: > looks like you're connecting to a service listening on SSL but you don't > have the CA used in your truststore > > On Thu, May 24, 2018 at 1:58 PM, Surbhi Gu

Re: Question About Reaper

2018-05-24 Thread Surbhi Gupta
(X509TrustManagerImpl.java:281) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ... 20 common frames omitted Any thought? On 24 May 2018 at 10:35, Surbhi Gupta <surbhi.gupt...@gmail.

Re: Question About Reaper

2018-05-24 Thread Surbhi Gupta
Another question, We use 9142 cqlsh port in one of the datacenter and on other datacenter we use 9042 port. How should we configure this ? On 24 May 2018 at 10:22, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: > What is the impact of > PARALLEL - all replicas at the same time ? &g

Re: Question About Reaper

2018-05-24 Thread Surbhi Gupta
rs. > > Cheers, > > Le mar. 22 mai 2018 à 07:28, Surbhi Gupta <surbhi.gupt...@gmail.com> a > écrit : > >> We are on Dse 4.8.15 and it is cassandra 2.1. >> What are the best configuration to use for reaper for 144 nodes with 256 >> vnodes and it shows aroun

Re: Question About Reaper

2018-05-21 Thread Surbhi Gupta
an vnodes, and is available with Cassandra 2.2 and onwards (the > improvement is especially beneficial with Cassandra 3.0+ as such token > ranges will be repaired in a single session). > > We have a gitter that you can join if you want to ask questions. > > Cheers, > > Le

Re: Question About Reaper

2018-05-21 Thread Surbhi Gupta
..you go > down further but it will have cascading effects in cpu and memory > consumption. > So test well. > > > On Monday, May 21, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: > >> Thanks a lot for your inputs, >> Abdul, how did u tune reaper? >

Re: Question About Reaper

2018-05-20 Thread Surbhi Gupta
hen proceed further >> ..i have tested so far in 1 dc only , we can actually have seperate reaper >> instance handling seperate dc but havent tested it yet. >> >> >> On Sunday, May 20, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: >> >>

Question About Reaper

2018-05-20 Thread Surbhi Gupta
Hi, We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . When we tried to start repairs from opscenter then it showed 1.9Million ranges to repair . And even after doing compaction and strekamthroughput to 0 , opscenter is not able to help us much to finish repair in 9 days timeframe

Re: Cassandra crashes....

2017-08-22 Thread Surbhi Gupta
16GB heap is too small for G1GC . Try at least 32GB of heap size On Tue, Aug 22, 2017 at 7:58 AM Fay Hou [Storage Service] ­ < fay...@coupang.com> wrote: > What errors do you see? > 16gb of 256 GB . Heap is too small. I would give heap at least 160gb. > > > On Aug 22, 2017 7:42 AM, "Thakrar,

Re: Tool to manage cassandra

2017-06-16 Thread Surbhi Gupta
If u are using dse then u can use opscenter On Fri, Jun 16, 2017 at 6:01 AM Ram Bhatia wrote: > Hi > > > > > > > > > > May I know, if there a tool similar to Oracle Enterprise Manager for > managing Cassandra ? > > > > > > > > > > Thank you in advance for your help, > >

Re: How do you do automatic restacking of AWS instance for cassandra?

2017-05-27 Thread Surbhi Gupta
We get the new AMI release with the new OS updates and we are not allowed to use the old AMI . On Sat, May 27, 2017 at 7:11 PM Jeff Jirsa <jji...@apache.org> wrote: > > > > > On 2017-05-27 18:04 (-0700), Surbhi Gupta <surbhi.gupt...@gmail.com> > wrote: > >

Re: How do you do automatic restacking of AWS instance for cassandra?

2017-05-27 Thread Surbhi Gupta
<(415)%20501-0198>London >> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>* >> >> >> *“All men dream, but not equally. Those who dream by night in the dusty >> recesses of their minds wake up in the day to find it was vanity, but the >> dreamers of the day are da

How do you do automatic restacking of AWS instance for cassandra?

2017-05-25 Thread Surbhi Gupta
Hi, Wanted to understand, how do you do automatic restacking of cassandra nodes on AWS? Thanks Surbhi

Re: Unsuccessful back-up and restore with differing counts

2017-05-13 Thread Surbhi Gupta
Below link has the method u r looking for http://datascale.io/cloning-cassandra-clusters-fast-way/ On Sat, May 13, 2017 at 9:49 AM srinivasarao daruna wrote: > I am using vnodes. Is there a documenation that you can suggest to > understand how to assign same tokens in

antlr-runtime-3.2.jar is turning into 0 bytes and dse is going down

2017-04-05 Thread Surbhi Gupta
Hi, We have single node instance where we have cassandra , mysql and application running at the same node for developers. We are at dse 4.8.9 and dse is going down after sometime . What we have noticed is that few of the jar at /usr/share/dse/common are turning into 0 bytes. Jars are as follows:

Re: How to find total data size of a keyspace.

2017-02-28 Thread Surbhi Gupta
Nodetool status key space_name . On Tue, Feb 28, 2017 at 4:53 AM anuja jain wrote: > Hi, > Using nodetool cfstats gives me data size of each table/column family and > nodetool ring gives me load of all keyspace in cluster but I need total > data size of one keyspace in the

Re: Cassandra Node Restart Stuck in STARTING?

2016-11-16 Thread Surbhi Gupta
Attaching the system.log can give more details ... On 16 November 2016 at 11:05, Daniel Subak wrote: > Hey everyone, > > Ran into an issue running a node restart where "nodetool netstats" > reported the node as "STARTING" with no streams when run locally. "nodetool >

Re: Priority for cassandra nodes in cluster

2016-11-12 Thread Surbhi Gupta
If u ask conceptually, it is possible but not recommended. If u really want to do it use the initial token setting and provide the broad range to the nodes where u want more data. But u need to understand about the replication factor consideration, if u keep rf as 3 on a 3 node cluster that means

Re: Keyspace/CF creation Timeouts

2016-10-25 Thread Surbhi Gupta
S > and 3 CF. > > write_request_timeout_in_ms = 1 -> 10 seconds > > On Tue, Oct 25, 2016 at 3:00 PM, Surbhi Gupta <surbhi.gupt...@gmail.com> > wrote: > >> As you have many keyspaces and column family to be created that might be >> the reason that within the stipulated time

Re: Keyspace/CF creation Timeouts

2016-10-25 Thread Surbhi Gupta
ith. > > On Tue, Oct 25, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> 1. Yes, all nodes are up and running, >> 2. We are using the Local_QUORUM. >> >> On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta <surbhi.gupt.

  1   2   >