Re: New DC / token distribution not balanced

2023-03-16 Thread Bowen Song via user
No, allocate_tokens_for_local_replication_factor does not exist in Cassandra 3. It was introduced in Cassandra 4.0. Now, may I interest you with an upgrade? Not only Cassandra 4 comes with a lots of improvements and bug fixes, it's also a fairly painless process. I find it much easier to

Re: New DC / token distribution not balanced

2023-03-16 Thread Max Campos
Does this exist for Cassandra 3.x? I know it was in DSE for DSE’s 3.x equivalent, and seems to be in Cassandra 4.x cassandra.yaml. I don’t see it here, though: https://github.com/apache/cassandra/blob/cassandra-3.11/conf/cassandra.yaml

Re: New DC / token distribution not balanced

2023-03-16 Thread Bowen Song via user
You may find "allocate_tokens_for_local_replication_factor" more useful than "allocate_tokens_for_keyspace" when you are spinning up a new DC. On 16/03/2023 06:25, Max Campos wrote: Update:  I figured out the problem! The “allocate_tokens_for_keyspace” value needs to be set for a keyspace

Re: New DC / token distribution not balanced

2023-03-16 Thread Max Campos
Update: I figured out the problem! The “allocate_tokens_for_keyspace” value needs to be set for a keyspace that has RF=3 for the DC being added. I just had the RF=3 set for the existing DC. I created a dummy keyspace with RF=3 for the new DC, set “allocate_tokens_for_keyspace=” and then

New DC / token distribution not balanced

2023-03-15 Thread Max Campos
Hi All - I’m having a lot of trouble adding a new DC and getting a balanced ring (i.e. every node has the same percentage of the token ring). My config: GossipingPropertyFileSnitch allocate_tokens_for_keyspace: num_tokens = 16 6 nodes in the new DC / 3 nodes in the existing DC Cassandra

unsubscribe

2023-03-12 Thread Mathieu Delsaut
Mathieu Delsaut Studies Engineer @ DSIMB & ENERGY-Lab Interne : 2357 | Externe : +262 262 93 86 08 .

Regression in cqlsh's "COPY … TO STDOUT" in 4.1?

2023-03-11 Thread Gustav Broberg
Hi, Did the COPY command with stdout as output stop working in cqlsh as of Cassandra 4.1? Or is this some issue with the official Docker image? This works for me with *4.0.8*: $> docker run -d --name cassandra --rm cassandra:4.0.8 $> docker exec cassandra cqlsh -e 'COPY

Re: Cassandra in Kubernetes: IP switch decommission issue

2023-03-10 Thread Inès Potier
Thanks for your response! Following your advice, I filed a Jira ticket here: https://issues.apache.org/jira/browse/CASSANDRA-18319 On Thu, Mar 9, 2023 at 11:16 AM Jeff Jirsa wrote: > I described something roughly similar to this a few years ago on the list. > The specific chain you're

Re: Cassandra in Kubernetes: IP switch decommission issue

2023-03-09 Thread Tom Nora
unsubscribe *Tom Nora * *Startup Growth & Funding* The Book -- HACKING THE CORE *linkedin | twitter

Re: Cassandra on SLES 15?

2023-03-09 Thread Elliott Sims via user
A quick search shows SLES 15 provides Java 11 (java-11-openjdk), which is just fine for Cassandra 4.x. On Wed, Mar 8, 2023 at 2:56 PM Eric Ferrenbach < eric.ferrenb...@milliporesigma.com> wrote: > We are running Cassandra 4.0.7. > > We are preparing to migrate our nodes from Centos to SUSE

Re: Cassandra in Kubernetes: IP switch decommission issue

2023-03-09 Thread Jeff Jirsa
I described something roughly similar to this a few years ago on the list. The specific chain you're describing isn't one I've thought about before, but if you open a JIRA for tracking and attribution, I'll ask some folks to take a peek at it. On Thu, Mar 9, 2023 at 10:57 AM Inès Potier wrote:

Re: Cassandra in Kubernetes: IP switch decommission issue

2023-03-09 Thread Inès Potier
Hi Cassandra community, Reaching out again in case anyone has recently faced the below issue. Additional opinions on this would be super helpful for us. Thanks in advance, Ines On Thu, Feb 23, 2023 at 3:40 PM Inès Potier wrote: > Hi Cassandra community, > > We have recently encountered a

Re: Does Coordinator select fastest node for Digest request In Read Path

2023-03-09 Thread Miklosovic, Stefan
Hi Ranju, I see this in the code: https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageProxy.java#L2096 From: ranju goel Sent: Thursday, March 9, 2023 13:20 To: user@cassandra.apache.org Subject: Does Coordinator

Does Coordinator select fastest node for Digest request In Read Path

2023-03-09 Thread ranju goel
Hi Everyone, If I have a Local_Quorum CL and RF=3, For Read Path Coordinator selects the fastest replica using dynamicSnitch for Read full data , but does it use dynamicSnitch ( or fastest replica) for reading digest data? or it chooses any of the replica for digest? Regards Ranju

Re: [RELEASE] Apache Cassandra 4.0.8 released

2023-03-09 Thread Brandon Williams
It was reported in CASSANDRA-18307 that the Debian and Redhat packages for 4.0.8 did not make it to the jfrog repository - this has now been corrected, sorry for any inconvenience. Kind Regards, Brandon On Tue, Feb 14, 2023 at 3:39 PM Miklosovic, Stefan wrote: > > The Cassandra team is pleased

Cassandra on SLES 15?

2023-03-08 Thread Eric Ferrenbach
We are running Cassandra 4.0.7. We are preparing to migrate our nodes from Centos to SUSE Linux. This page only mentions SLES 12 (not 15) https://cassandra.apache.org/doc/latest/cassandra/getting_started/installing.html This states SLES 12 Active support ends next year:

Coordinator stuck to replica node causing read time out

2023-03-06 Thread Ranju Jain via user
Hi Everyone, Nodetool status gives Cassandra node status as [ UP ] if this machine has private interface down but public interface up. Because this public interface is Up, it is resulting in Cassandra node status as UP. Due to this UP Status when coordinator is sending a read request towards

Re: Bootstrapping new node throwing error - Mutation too large

2023-03-01 Thread Surbhi Gupta
Thanks Scott On Wed, Mar 1, 2023 at 4:00 PM C. Scott Andreas wrote: > The performance implications would primarily be due to the challenge of > handling mutations this large themselves rather than the commitlog segment > size. These would occupy large, contiguous areas of heap and increase >

Re: Bootstrapping new node throwing error - Mutation too large

2023-03-01 Thread C. Scott Andreas
The performance implications would primarily be due to the challenge of handling mutations this large themselves rather than the commitlog segment size. These would occupy large, contiguous areas of heap and increase memory pressure in the process.Increasing  commit_log_segment_size_in_mb is

Bootstrapping new node throwing error - Mutation too large

2023-03-01 Thread Surbhi Gupta
Hi Cassandra Community, We have to expand our cluster and I tried to add the first node to the cluster and when the new node was bootstrapping , I noticed the error like below in the system.log, but the bootstrap process was successful . We are on 3.11.5 . ERROR [MutationStage-7] 2023-03-01

Accord and elector selection

2023-03-01 Thread Aaron Ploetz
Under the premise of an Accord transaction, how does replica/elector selection happen? Does it choose replicas based on the partition keys? Or is there another way? Thanks! Aaron

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

2023-02-27 Thread Molly Monroy
Hello all - Thank you to everyone who joined our Publicity and Marketing Group kick off meetings last month. We're excited to continue the conversation in March. *Our next Apache Cassandra Publicity & Marketing Group meetings are coming up this Wednesday, March 1 @ 8a PT (EMEA, UK, LATAM, NA) and

Re: nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Mitch Gitman
Sergio and Abe, thanks so much for responding to me so quickly! I managed to figure out the problem and the solution. In the Terraform scripts we used to stand up the EC2 instances, we have a template file for the jmxremote.access file with the content: monitorRole readonly controlRole

Re: nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Sergio
Hey! I would try to spin up a new node and see if the problem occurs on it. If it happens, I would check the history of changes on the cookbook recipe, if you don't find any problem on the new node you might replace all the nodes having problems one by one with a new one and decommission the

Re: nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Abe Ratnofsky
Hey Mitch, The security upgrade schedule that your colleague is working on may well be relevant. Is your entire cluster on 3.11.6 or are the failing hosts possibly on a newer version? Abe > On Feb 26, 2023, at 10:38, Mitch Gitman wrote: > >  > We're running Cassandra 3.11.6 on AWS EC2

nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Mitch Gitman
We're running Cassandra 3.11.6 on AWS EC2 instances. These clusters have been running for a few years. We're suddenly noticing now that on one of our clusters the nodetool command is failing on certain nodes but not on others. The failure: nodetool: Failed to connect to '...:7199' -

Cassandra in Kubernetes: IP switch decommission issue

2023-02-23 Thread Inès Potier
Hi Cassandra community, We have recently encountered a recurring old IP reappearance issue while testing decommissions on some of our Kubernetes Cassandra staging clusters. We have not yet found other references to this issue online. We could really use some additional inputs/opinions, both on

Re: JIRA account creation request

2023-02-17 Thread Srinivas Chamarthi
Hi Mick , I didn't receive any email. Could you pls. resend it ? thx srinivas On Thu, Feb 16, 2023 at 1:23 PM Mick Semb Wever wrote: > > HI Mick, >> >> Could you pls. help with JIRA account for me as well ? >> > > > > > Done Srinivas. You should have received an email. > > Welcome to the

RE: Cleanup

2023-02-17 Thread Durity, Sean R via user
Cleanup, by itself, uses all the compactors available. So, it is important to see if you have the disk space for multiple large cleanup compactions running at the same time. We have a utility to do cleanup more intelligently – it temporarily doubles compaction throughput, operates on a single

Re: Cleanup

2023-02-16 Thread Dipan Shah
Hi Marc, Changes done using "nodetool setcompactionthroughput" will only be applicable till Cassandra service restart. The throughput value will revert back to the settings inside cassandra.yaml post service restart. On Fri, Feb 17, 2023 at 1:04 PM Marc Hoppins wrote: > …and if it is altered

RE: Cleanup

2023-02-16 Thread Marc Hoppins
…and if it is altered via nodetool, is it altered until manually changed or service restart, so must be manually put pack? From: Aaron Ploetz Sent: Thursday, February 16, 2023 4:50 PM To: user@cassandra.apache.org Subject: Re: Cleanup EXTERNAL So if I remember right, setting

Re: Deletions getting omitted

2023-02-16 Thread shankha b
I checked the writetimstamp as suggested and they match to the deletion timestamp instead of insertion timestamp. That means the data got resurrected. I unfortunately cannot put trace on. Am I correct to conclude that? From: C. Scott Andreas Sent: Monday,

Re: Cleanup

2023-02-16 Thread Aaron Ploetz
So if I remember right, setting compaction_throughput_per_mb to zero effectively disables throttling, which means cleanup and compaction will run as fast as the instance will allow. For normal use, I'd recommend capping that at 8 or 16. Aaron On Thu, Feb 16, 2023 at 9:43 AM Marc Hoppins

RE: Cleanup

2023-02-16 Thread Marc Hoppins
Compaction_throughtput_per_mb is 0 in cassandra.yaml. Is setting it in nodetool going to provide any increase? From: Durity, Sean R via user Sent: Thursday, February 16, 2023 4:20 PM To: user@cassandra.apache.org Subject: RE: Cleanup EXTERNAL Clean-up is constrained/throttled by

RE: Cleanup

2023-02-16 Thread Durity, Sean R via user
Clean-up is constrained/throttled by compactionthroughput. If your system can handle it, you can increase that throughput (nodetool setcompactionthroughput) for the clean-up in order to reduce the total time. It is a node-isolated operation, not cluster-involved. I often run clean up on all

Re: Cleanup

2023-02-16 Thread manish khandelwal
There is no advantage of running cleanup if no new nodes are introduced. So cleanup time should remain same when adding new nodes. Cleanup is a local to node so network bandwidth should have no effect on reducing cleanup time. Dont ignore cleanup as it can cause you disks occupied without any

Cleanup

2023-02-16 Thread Marc Hoppins
Hulloa all, I read a thing re. adding new nodes where the recommendation was to run cleanup on the nodes after adding a new node to remove redundant token ranges. I timed this way back when we only had ~20G of data per node and it took approx. 5 mins per node. After adding a node on Tuesday,

Re: JIRA account creation request

2023-02-15 Thread Mick Semb Wever
> HI Mick, > > Could you pls. help with JIRA account for me as well ? > Done Srinivas. You should have received an email. Welcome to the Cassandra community.

Re: JIRA account creation request

2023-02-15 Thread Srinivas Chamarthi
HI Mick, Could you pls. help with JIRA account for me as well ? thx sri On Wed, Feb 15, 2023 at 2:43 PM Mick Semb Wever wrote: > > I would like to get my JIRA account created as I would like to contribute. >> Here are my details >> >> email address : manishkhandelwa...@gmail.com >> > > > Your

Re: JIRA account creation request for new contributor

2023-02-15 Thread Erick Ramirez
Mick has provisioned your account and you should receive a separate email from the jira server with further instructions. Cheers!

Re: JIRA account creation request

2023-02-15 Thread Mick Semb Wever
> I would like to get my JIRA account created as I would like to contribute. > Here are my details > > email address : manishkhandelwa...@gmail.com > Your jira account has been created. You should have received an email. regards, Mick

JIRA account creation request

2023-02-15 Thread manish khandelwal
Hi I would like to get my JIRA account created as I would like to contribute. Here are my details email address : manishkhandelwa...@gmail.com If any other information is needed please let me know. Regards Manish

Re: JIRA account creation request for new contributor

2023-02-15 Thread Erick Ramirez
Welcome! Sorry for the delay. Let me see if I can get your request expedited.  On Tue, 14 Feb 2023 at 07:31, Omair Muhi wrote: > Greetings, > > I would like to request a new JIRA account as I am interested in > contributing to the project. Here is my information: > > >- email address:

[RELEASE] Apache Cassandra 4.0.8 released

2023-02-14 Thread Miklosovic, Stefan
The Cassandra team is pleased to announce the release of Apache Cassandra version 4.0.8. Apache Cassandra is a fully distributed database. It is the right choice when you need scalability and high availability without compromising performance. http://cassandra.apache.org/ Downloads of source

Re: Pulling unreceived schema versions

2023-02-14 Thread Dipan Shah
Hello Joe, "Pulling unreceived schema versions" in Apache Cassandra means that a node is requesting schema updates from other nodes in the cluster that it has not yet received. This is a normal part of the Cassandra distributed architecture, as each node needs to stay up-to-date with the

RE: JIRA account creation request for new contributor

2023-02-13 Thread Omair Muhi
Greetings, I would like to request a new JIRA account as I am interested in contributing to the project. Here is my information: email address: omairm...@icloud.com preferred username: OMAIRMUHI alternate username: OMUHI Please let me know if there is any other information needed from my side.

Pulling unreceived schema versions

2023-02-13 Thread Joe Obernberger
Hi all - I'm seeing this message: "Pulling unreceived schema versions..." in the debug log being repeated exactly every minute, but I can't find what this means? Thank you! -Joe -- This email has been checked for viruses by AVG antivirus software. www.avg.com

Re: Replacing node w/o bootstrapping (streaming)?

2023-02-09 Thread Max Campos
Awesome - thanks Jeff! - Max > On Feb 9, 2023, at 6:45 pm, Jeff Jirsa wrote: > > You don’t have to do anything else. Just use smart rsync flags (including > delete). > > It’ll work fine just the way you described. No special start args. No > replacement flag > > Be sure you rsync the

Re: Replacing node w/o bootstrapping (streaming)?

2023-02-09 Thread Jeff Jirsa
You don’t have to do anything else. Just use smart rsync flags (including delete). It’ll work fine just the way you described. No special start args. No replacement flag Be sure you rsync the commitlog directory too . Flush and drain to be extra safe > On Feb 9, 2023, at 6:42 PM, Max

Replacing node w/o bootstrapping (streaming)?

2023-02-09 Thread Max Campos
Hi - We have a node whose root partition is flaking out. The disk that contains the Cassandra data, however, is healthy. We’d like to replace the dying node with a procedure like this: 0) OLD node is running, NEW node has never started Cassandra 1) rsync Cassandra data from OLD node to NEW

Re: Deletions getting omitted

2023-02-06 Thread C. Scott Andreas
Can you check the write timestamp of the data you're attempting to delete?https://docs.datastax.com/en/cql-oss/3.3/cql/cql_using/useWritetime.htmlIf the timestamp of the write is in the future (e.g., due to a time sync issue or an errant client-supplied timestamp at the time of that write),

Re: Deletions getting omitted

2023-02-06 Thread shankha b
By any chance do you think we are getting hit by a read repair? From: shankha b Sent: Monday, February 6, 2023 7:10 AM To: user@cassandra.apache.org Subject: Re: Deletions getting omitted I read through the ticket a few number of times. We have replication factor 3 and LocalQuorum. Do we

Re: Deletions getting omitted

2023-02-06 Thread shankha b
I read through the ticket a few number of times. We have replication factor 3 and LocalQuorum. Do we still think CASSANDRA-15690 is a possibility with RF = 3? From: shankha b Sent: Saturday, February 4, 2023 9:29 PM To: user@cassandra.apache.org Subject: Re:

Re: Deletions getting omitted

2023-02-04 Thread shankha b
I will look into raising the gc_grace_seconds. We are using LocalQuorum for all reads and writes. We do not use ALL exactly for outage reasons. From: Jeff Jirsa Sent: Saturday, February 4, 2023 8:44 PM To: user@cassandra.apache.org Subject: Re: Deletions

Re: Deletions getting omitted

2023-02-04 Thread Jeff Jirsa
While you'd expect only_purge_repaired_tombstones:true to be sufficient, your gc_grace_secnds of 1 hour is making you unusually susceptible to resurrecting data. (To be clear, you should be safe to do this, but if there is a bug hiding in there somewhere, your low gc_grace_seconds will make it

Deletions getting omitted

2023-02-04 Thread shankha b
We are facing an issue on one of our production systems where after we delete the data the data doesn't seem to get deleted. We have a Get call just after the delete call. The data shows up. Versions cassandra : 3.11.6 gocqlx : v2 v2.1.0 1. Client Settings: LocalQuorum 2. Number of

Re: Important news about Cassandra Summit

2023-02-03 Thread Wei Deng
Thank you for the update, Patrick. Appreciate all the work your team, the community and the Linux Foundation Events team have done. Looking forward to the virtual event and Cassandra Day series. > If you have open positions, call them out in this email thread or #cassandra in the ASF slack.

Important news about Cassandra Summit

2023-02-03 Thread Patrick McFadin
*Hello Cassandra Community,We all see what’s happening in tech right now. Cuts are being made, and budgets are frozen. For Cassandra Summit, this has translated to low sponsorship and registrations. The program committee has been discussing options with the Linux Foundation events team, and the

Re: Startup fails - 4.1.0

2023-02-03 Thread Joe Obernberger
Thank you Sean.  I had to remove two of the files and then it started. Cheers! -Joe On 2/3/2023 3:52 PM, Durity, Sean R via user wrote: In most cases, I would delete the corrupt commit log file and restart. Then run repairs on that node. I have seen cases where multiple files are corrupted

RE: Startup fails - 4.1.0

2023-02-03 Thread Durity, Sean R via user
In most cases, I would delete the corrupt commit log file and restart. Then run repairs on that node. I have seen cases where multiple files are corrupted and it is easier to remove all commit log files to get the node restarted. Sean R. Durity From: Joe Obernberger Sent: Friday, February 3,

Startup fails - 4.1.0

2023-02-03 Thread Joe Obernberger
Hi all - cluster had a power outage and one of the nodes in a 14 nodes cluster isn't starting with: DEBUG [MemtableFlushWriter:1] 2023-02-03 13:52:45,468 ColumnFamilyStore.java:1329 - Flushed to

Re: Changing tokens between datacenters

2023-01-30 Thread Elliott Sims
For dealing with allocate_tokens_for_keyspace in datacenter migrations, I've just created a dummy keyspace in the new DC with the desired topology, then removed it once everything's done. On Mon, Jan 30, 2023 at 3:36 PM Doug Whitfield wrote: > Hi folks, > > In our 3.11 deployments we are using

Changing tokens between datacenters

2023-01-30 Thread Doug Whitfield
Hi folks, In our 3.11 deployments we are using the feature called virtual nodes (vnodes). So far, we have always used the old default value 256 for the num_tokens parameter specified in the cassandra.yaml (see also example file attached), as follows: num_tokens: 256 #

Apache Cassandra Publicity & Marketing Group - Cassandra Summit Asks

2023-01-27 Thread Molly Monroy
Hello All: I will be supporting the publicity & marketing working group along with my colleagues at Constantia, which partners with Cassandra for publicity and marketing. For those of you I haven't met yet, I look forward to working with you! For our first order of business for this group ... We

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

2023-01-26 Thread Craig Kitterman
+1! On Thu, Jan 26, 2023 at 3:49 PM Aaron Ploetz wrote: > We are going to have our first gathering next Wednesday at 8AM PT > > > I'd like to help out, too, if you wouldn't mind forwarding that meeting on > to me. > > Thanks, > > Aaron > > -- Cheers, -C

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

2023-01-26 Thread Patrick McFadin
The meeting link I included should let you add it to your calendar. If not, let me know. Patrick On Thu, Jan 26, 2023 at 3:49 PM Aaron Ploetz wrote: > We are going to have our first gathering next Wednesday at 8AM PT > > > I'd like to help out, too, if you wouldn't mind forwarding that meeting

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

2023-01-26 Thread Aaron Ploetz
> > We are going to have our first gathering next Wednesday at 8AM PT I'd like to help out, too, if you wouldn't mind forwarding that meeting on to me. Thanks, Aaron

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

2023-01-26 Thread Patrick McFadin
Thanks for the positive reception on email and slack. We are going to have our first gathering next Wednesday at 8AM PT Link to calendar event:

Re: Cassandra Summit update for 2023-01-24

2023-01-25 Thread C. Scott Andreas
Hugely excited to this – thanks to the Program Committee and to the Linux Foundation for organizing!It's been a long few years away from conferences and I can't wait to see all of you.Beyond learning about what everyone is doing with Apache Cassandra, I'm looking forward to the hallway chats

Re: Unsubscribe

2023-01-24 Thread Erick Ramirez
Sorry to see you go. If you'd like to unsubscribe from the user ML, please email user-unsubscr...@cassandra.apache.org. Cheers!

Cassandra Summit update for 2023-01-24

2023-01-24 Thread Patrick McFadin
*Hello Cassandra Community!Quick take: - Register before 1/28 to get discount pricing. https://events.linuxfoundation.org/cassandra-summit/register/ - Use code CS23DS20 to get 20% off - Make sure and sign up for training the day on

Re: Upgrading Cassandra 3.11.14 → 4.1

2023-01-24 Thread Mick Semb Wever
On Mon, 16 Jan 2023 at 14:38, Lapo Luchini wrote: > is upgrading Cassandra 3.11.14 → 4.1 supported, > 3.11.14 → 4.1 is supported. It is recommended to go to the last patch version (i.e. 3.11.14) before the major upgrade. Make sure to ensure all sstables are upgraded to the current format

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

2023-01-24 Thread Mick Semb Wever
The market...@cassandra.apache.org list is created. To subscribe send an email to marketing-subscr...@cassandra.apache.org from the email address you want to subscribe from. If you are a committer you can alternately use Whimsy: https://whimsy.apache.org/committers/subscribe regards, Mick On

Unsubscribe

2023-01-24 Thread marine.itejeda
Unsubscribe

Re: removenode stuck - cassandra 4.1.0

2023-01-23 Thread Joe Obernberger
Thank you - I was just impatient.  :) -Joe On 1/23/2023 12:56 PM, Jeff Jirsa wrote: Those hosts are likely sending streams. If you do `nodetool netstats` on the replicas of the node you're removing, you should see byte counters and file counters - they should all be incrementing. If one of

Re: removenode stuck - cassandra 4.1.0

2023-01-23 Thread Jeff Jirsa
Those hosts are likely sending streams. If you do `nodetool netstats` on the replicas of the node you're removing, you should see byte counters and file counters - they should all be incrementing. If one of them isnt incremening, that one is probably stuck. There's at least one bug in 4.1 that

removenode stuck - cassandra 4.1.0

2023-01-23 Thread Joe Obernberger
I had a drive fail (first drive in the list) on a Cassandra cluster.  I've stopped the node (as it no longer starts), and am trying to remove it from the cluster, but the removenode command is hung (been running for 3 hours so far): nodetool removenode status is always reporting the same token

Re: Failed disks - correct procedure

2023-01-23 Thread Joe Obernberger
Some more observations.  If the first drive fails on a node, then you can't just remove it from the list.  Example: We have: /data/1/cassandra /data/2/cassandra /data/3/cassandra /data/4/cassandra ... If /data/1 fails, and I remove it from the list, when you try to start cassandra on that node

Re: Cassandra nightly process

2023-01-23 Thread Loïc CHANEL via user
Thanks for your help guys. You were right, the problem actually came from a very heavy data treatment that happens every 2 hours starting at midnight. The processing performance was heavily affected causing one node to write hints because communication with the other node was complicated. Best

[DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

2023-01-19 Thread Patrick McFadin
*Hello Cassandra Community!We are at a pivotal moment for the Cassandra community, with the first Cassandra Summit in 7 years coming up on March 13th, and a major release coming later this year with Cassandra 5.0. It is important that we come together to set the publicity strategy and direction

Re: Failed disks - correct procedure

2023-01-17 Thread C. Scott Andreas
Bumping this note from Andy downthread to make sure everyone has seen it and is aware:“Before you do that, you will want to make sure a cycle of repairs has run on the replicas of the down node to ensure they are consistent with each other.”When replacing an instance, it’s necessary to run repair

Re: Failed disks - correct procedure

2023-01-17 Thread Joe Obernberger
I come from the hadoop world where we have a cluster with probably over 500 drives.  Drives fail all the time; or well several a year anyway.  We remove that single drive from HDFS, HDFS re-balances, and when we get around to it, we swap in a new drive, format it, and add it back to HDFS.  We

RE: Failed disks - correct procedure

2023-01-17 Thread Durity, Sean R via user
For physical hardware when disks fail, I do a removenode, wait for the drive to be replaced, reinstall Cassandra, and then bootstrap the node back in (and run clean-up across the DC). All of our disks are presented as one file system for data, which is not what the original question was

RE: Failed disks - correct procedure

2023-01-17 Thread Marc Hoppins
HI all, I was pondering this very situation. We have a node with a crapped-out disk (not the first time). Removenode vs repairnode: in regard time, there is going to be little difference twixt replacing a dead node and removing then re-installing a node. There is going to be a bunch of

Re: Failed disks - correct procedure

2023-01-16 Thread Joe Obernberger
I'm using 4.1.0-1. I've been doing a lot of truncates lately before the drive failed (research project).  Current drives have about 100GBytes of data each, although the actual amount of data in Cassandra is much less (because of truncates and snapshots).  The cluster is not homo-genius; some

Re: Failed disks - correct procedure

2023-01-16 Thread Jeff Jirsa
Prior to cassandra-6696 you’d have to treat one missing disk as a failed machine, wipe all the data and re-stream it, as a tombstone for a given value may be on one disk and data on another (effectively redirecting data) So the answer has to be version dependent, too - which version were you

Re: Failed disks - correct procedure

2023-01-16 Thread Tolbert, Andy
Hi Joe, Reading it back I realized I misunderstood that part of your email, so you must be using data_file_directories with 16 drives? That's a lot of drives! I imagine this may happen from time to time given that disks like to fail. That's a bit of an interesting scenario that I would have to

Re: Failed disks - correct procedure

2023-01-16 Thread Joe Obernberger
Thank you Andy. Is there a way to just remove the drive from the cluster and replace it later?  Ordering replacement drives isn't a fast process... What I've done so far is: Stop node Remove drive reference from /etc/cassandra/conf/cassandra.yaml Restart node Run repair Will that work?  Right

Re: Failed disks - correct procedure

2023-01-16 Thread Tolbert, Andy
Hi Joe, I'd recommend just doing a replacement, bringing up a new node with -Dcassandra.replace_address_first_boot=ip.you.are.replacing as described here: https://cassandra.apache.org/doc/4.1/cassandra/operating/topo_changes.html#replacing-a-dead-node Before you do that, you will want to make

Re: Cassandra nightly process

2023-01-16 Thread Gábor Auth
Hi, On Mon, Jan 16, 2023 at 3:07 PM Loïc CHANEL via user < user@cassandra.apache.org> wrote: > So my question here is : am I missing a Cassandra internal process that is > triggered on a daily basis at 0:00 and 2:00 ? > I bet, it's not a Cassandra issue. Have you any other metrics about your

Re: Cassandra nightly process

2023-01-16 Thread Patrick McFadin
My general advice for any time you see hints accumulating, consider that smoke for the more pressing fire happening somewhere else. You correctly identified the right path to consider, which is some sort of scheduled activity. Cassandra doesn't have any scheduled internal jobs. Compactions happen

Failed disks - correct procedure

2023-01-16 Thread Joe Obernberger
Hi all - what is the correct procedure when handling a failed disk? Have a node in a 15 node cluster.  This node has 16 drives and cassandra data is split across them.  One drive is failing.  Can I just remove it from the list and cassandra will then replicate? If not - what? Thank you! -Joe

Re: Cassandra nightly process

2023-01-16 Thread Yakir Gibraltar
Check if you see packet loss at this time On Mon, Jan 16, 2023 at 4:08 PM Loïc CHANEL via user < user@cassandra.apache.org> wrote: > Hi team, > > I am currently running a 2-nodes Cassandra database. Although that's not > the best setup, the cluster is doing pretty fine. > Still, I noticed that

Cassandra nightly process

2023-01-16 Thread Loïc CHANEL via user
Hi team, I am currently running a 2-nodes Cassandra database. Although that's not the best setup, the cluster is doing pretty fine. Still, I noticed that for (at least) 5 days now, one of my two nodes is writing hints during the night, and then it recovers the data-sync with the other node in the

Upgrading Cassandra 3.11.14 → 4.1

2023-01-16 Thread Lapo Luchini
Hi all, is upgrading Cassandra 3.11.14 → 4.1 supported, or is it better to follow the 3.11.14 → 4.0 → 4.1 path? (I think it is okay as i found no record of deprecated old SSTable formats, but I couldn't manage to find any official documentation regarding upgrade paths… forgive me if it

Re: Compactions are stuck in 4.0.5 version

2023-01-14 Thread vaibhav khedkar
Thank you so much Scott. Increasing the value from 64 to 128 fixed the issue for us. We will certainly look at our data model and understand why the partitions are growing to such a large value. Thanks vaibhav On Fri, Jan 13, 2023 at 9:45 PM C. Scott Andreas wrote: > Hi Vaibhav, thanks for

Re: Compactions are stuck in 4.0.5 version

2023-01-13 Thread C. Scott Andreas
Hi Vaibhav, thanks for reaching out.Based on my understanding of this exception, this may be due to the index for this partition exceeding 2GiB (which is *extremely* large for a partition index component).Reducing the size of the column index below 2GiB may resolve this issue. You may be able

Compactions are stuck in 4.0.5 version

2023-01-13 Thread vaibhav khedkar
Hello All, We are facing an issue where few of the nodes are not able to complete compactions. We tried restarting, scrubbing and even rebuilding an entire node but nothing seems to work so far. It's a 10 Region installation with close to 150 nodes. Datatax support

Re: Connection Latency with Cassandra 4.0.x

2023-01-11 Thread MyWorld
In addition to this, the established connection count to db is increasing on every new API hit. Ideally existing connection should be reused. Moreover even if we restart Cassandra the connection from the API end do not get killed and remains the same. On Thu, 12 Jan, 2023, 02:16 Elliott Sims,

Re: Connection Latency with Cassandra 4.0.x

2023-01-11 Thread Elliott Sims
Consistently 200ms, during the back-and-forth negotiation rather than the handshake? That sounds suspiciously like Nagle interacting with Delayed ACK. On Wed, Jan 11, 2023 at 8:41 AM MyWorld wrote: > Hi all, > We are facing a connection latency of 200ms between API server and db > server

<    2   3   4   5   6   7   8   9   10   11   >