Re: Adding nodes

2022-07-12 Thread Jeff Jirsa
Your rack awareness problem is described in https://issues.apache.org/jira/browse/CASSANDRA-3810 from 2012. The fundamental problem is that Cassandra wont move data except during bootstrap, decom, and explicit moves. The implication here is exactly what you've encountered - if you tell cassandra

Re: Adding nodes

2022-07-12 Thread Bowen Song via user
You have some (many?) misunderstanding of how Cassandra works, and therefore many of your questions are hard to answer without educating you first and make you asking different but related and relevant questions instead. That's why you aren't getting any answer from us. We are not paid to do

RE: Adding nodes

2022-07-12 Thread Marc Hoppins
I posted system log data, GC log data, debug log data, nodetool data. I believe I had described the situation more than adequately. Yesterday, I was asking what I assumed to be reasonable questions regarding the method for adding new nodes to a new rack. Forgive me if it sounds unreasonable

Re: Adding nodes

2022-07-12 Thread Jeff Jirsa
On Tue, Jul 12, 2022 at 7:27 AM Marc Hoppins wrote: > > I was asking the questions but no one cared to answer. > This is probably a combination of "it is really hard to answer a question with insufficient data" and your tone. Nobody here gets paid to help you solve your company's problems

Re: Adding nodes

2022-07-12 Thread Jeff Jirsa
Cassandra isn't Hadoop. Most of the mistakes you're making is treating a complex distributed system like a different complex distributed system without understanding the nuance. Racks vs DCs are because you wouldn't ever want both copies of data on one rack, in case the top of rack switch or PDU

RE: Adding nodes

2022-07-12 Thread Durity, Sean R via user
In my experience C* is not cheaper storage than HDFS. If that is the goal, it may be painful. Each Cassandra DC has at least one full copy of the data set. For production data that I care about (that my app teams care about), we use RF=3 in each Cassandra DC. And I only use 1 Cassandra rack

Re: Adding nodes

2022-07-12 Thread Bowen Song via user
I think you are misinterpreting many concepts here. For a starter, a physical rack in a physical DC is not (does not have to be) a logical rack in a logical DC in Cassandra; and the allocate_tokens_for_local_replication_factor has nothing to do with replication factor (other than using it as

RE: Adding nodes

2022-07-12 Thread Marc Hoppins
There is likely going to be 2 racks in each DC. Adding the new node decided to quit after 12 hours. Node was overloaded and GC pauses caused the bootstrap to fail. I begin to see the pattern here. If replication is only within the same datacentre, and one starts off with only one rack then