To unsubscribe, please send an email to
user-unsubscr...@cassandra.apache.org
On 20/07/2022 18:34, emmanuel warreng wrote:
Unsubscribe
On Thu, Jul 7, 2022, 16:49 Marc Hoppins wrote:
Hi all,
Cluster of 2 DC and 24 nodes
DC1 (RF3) = 12 nodes, 16 tokens each
DC2 (RF3) = 12
Unsubscribe
On Thu, Jul 7, 2022, 16:49 Marc Hoppins wrote:
> Hi all,
>
> Cluster of 2 DC and 24 nodes
>
> DC1 (RF3) = 12 nodes, 16 tokens each
> DC2 (RF3) = 12 nodes, 16 tokens each
>
> Adding 12 more nodes to DC1: I installed Cassandra (version is the same
> across all nodes) but, after the
not a practical response as any business is
> unlikely to be spending speculative money.
>
>
>
> *From:* Jeff Jirsa
> *Sent:* Tuesday, July 12, 2022 4:43 PM
> *To:* cassandra
> *Cc:* Bowen Song
> *Subject:* Re: Adding nodes
>
>
>
> EXTERNAL
&g
to suggest that folk head off a pay for a course when there
are ‘pre-sales’ questions is not a practical response as any business
is unlikely to be spending speculative money.
*From:*Jeff Jirsa
*Sent:* Tuesday, July 12, 2022 4:43 PM
*To:* cassandra
*Cc:* Bowen Song
*Subject:* Re: Adding nodes
’ questions is not a practical response as any business is unlikely
to be spending speculative money.
From: Jeff Jirsa
Sent: Tuesday, July 12, 2022 4:43 PM
To: cassandra
Cc: Bowen Song
Subject: Re: Adding nodes
EXTERNAL
On Tue, Jul 12, 2022 at 7:27 AM Marc Hoppins
mailto:marc.hopp...@eset.com
On Tue, Jul 12, 2022 at 7:27 AM Marc Hoppins wrote:
>
> I was asking the questions but no one cared to answer.
>
This is probably a combination of "it is really hard to answer a question
with insufficient data" and your tone. Nobody here gets paid to help you
solve your company's problems
n Song via user
> *Sent:* Tuesday, July 12, 2022 12:29 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Adding nodes
>
>
>
> EXTERNAL
>
> For RF=2 in your DC1, you will not be able to achieve both strong
> consistency and single point of failure tolerance within that
, the whole DC is useless.
Sean R. Durity
INTERNAL USE
From: Marc Hoppins
Sent: Tuesday, July 12, 2022 8:49 AM
To: user@cassandra.apache.org; Bowen Song
Subject: [EXTERNAL] RE: Adding nodes
The data guys want 2 copies of data in DC1 and that data to be replicated
offsite to DC1 for 1 copy (DR
or whatever, it is a
serious limitation as not every business/organisation is going to have
multiple racks available.
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 8:57 PM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
I've noticed the joining node has a different
/organisation is
going to have multiple racks available.
From: Bowen Song via user
Sent: Monday, July 11, 2022 8:57 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I've noticed the joining node has a different rack than the rest of the nodes,
is this intended? Will you add all new
Obernberger
*Sent:* Monday, July 11, 2022 1:29 PM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
I too came from HBase and discovered adding several nodes at a time
doesn't work. Are you absolutely sure that the clocks are in sync
across the nodes? This has bitten me several
should be run after adding all
nodes to reduce unnecessary strain on the cluster.
From: Marc Hoppins
Sent: Monday, July 11, 2022 2:15 PM
To: user@cassandra.apache.org
Subject: RE: Adding nodes
All clocks are fine.
Why would time synch would affect whether or not a node appears in the nodetool
statu
Obernberger
Sent: Monday, July 11, 2022 1:29 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I too came from HBase and discovered adding several nodes at a time doesn't
work. Are you absolutely sure that the clocks are in sync across the nodes?
This has bitten me several
day, July 11, 2022 11:56 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
Checking on multiple nodes won't help if the joining node suffers
from any of the issues I described, as it will likely be flipping up
and down frequently, and the existing nodes in the cluster may
al=0.03 secs]
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 11:56 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
Checking on multiple nodes won't help if the joining node suffers from
any of the issues I described, as it will likely be flipping up and
dow
NEW NODE manually. No seconds. No pauses. No
timeouts. Just waiting.
From: Bowen Song via user
Sent: Monday, July 11, 2022 12:13 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
How long doe it take to add a new node? I'm 100% sure neither 90s nor 120s is
the answer
. Therefore, if I know when my first node is
finished, I will have an idea of how long before I check for the when
subsequent nodes can be joined.
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 11:25 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
Sleeping/pausing
ser=0.22 sys=0.00, real=0.03 secs]
From: Bowen Song via user
Sent: Monday, July 11, 2022 11:56 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
Checking on multiple nodes won't help if the joining node suffers from any of
the issues I described, as it will likely be flipping
in the cluster, has been so for
several weeks, and is also functioning without error.
*From:*Bowen Song via user
*Sent:* Monday, July 11, 2022 11:40 AM
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes
EXTERNAL
A node in joining state can disappearing from the cluster from other
Sent: Monday, July 11, 2022 11:40 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
A node in joining state can disappearing from the cluster from other nodes'
perspective if the joining node stops sending/receiving gossip messages to
other nodes. This can happen when
can be joined.
From: Bowen Song via user
Sent: Monday, July 11, 2022 11:25 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
Sleeping/pausing for a fixed amount of time between operations at best is a
hack to workaround an unknown issue, but it's almost always better
up but from the
original seed node, it does not appear in the nodetool status. Can
anyone shed any light on this phenomena?
*From:*Marc Hoppins
*Sent:* Monday, July 11, 2022 10:02 AM
*To:* user@cassandra.apache.org
*Cc:* Bowen Song
*Subject:* RE: Adding nodes
Well then…
I left this on Friday
*To:* cassandra
*Cc:* Bowen Song
*Subject:* Re: Adding nodes
EXTERNAL
Having a node UJ but not sending/receiving other streams is an invalid
state (unless 4.0 moved the streaming data out of netstats? I'm not
100% sure, but I'm 99% sure it should be there).
It likely stopped the bootstrap
up but from the original seed node, it
does not appear in the nodetool status. Can anyone shed any light on this
phenomena?
From: Marc Hoppins
Sent: Monday, July 11, 2022 10:02 AM
To: user@cassandra.apache.org
Cc: Bowen Song
Subject: RE: Adding nodes
Well then…
I left this on Friday (still
which is unlikely to cause problems? After all, in the modern world of
big (how big is big?) data, 600G per node is far less than the real BIG
big-data.
Marc
From: Jeff Jirsa
Sent: Friday, July 8, 2022 5:46 PM
To: cassandra
Cc: Bowen Song
Subject: Re: Adding nodes
EXTERNAL
Having a node UJ
issue of not being able to add more nodes.
>
> -Original Message-
> From: Bowen Song via user
> Sent: Friday, July 8, 2022 11:47 AM
> To: user@cassandra.apache.org
> Subject: Re: Adding nodes
>
> EXTERNAL
>
>
> I would assume that's 85 GB (i.e. gigabytes) then. Which
To:user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I would assume that's 85 GB (i.e. gigabytes) then. Which is approximately 79
GiB (i.e. gibibytes). This still sounds awfully slow - less than 1MB/s over a
full day (24 hours).
You said CPU and network aren't the bottleneck. Have
None of this addresses the issue of not being able to add more nodes.
-Original Message-
From: Bowen Song via user
Sent: Friday, July 8, 2022 11:47 AM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
I would assume that's 85 GB (i.e. gigabytes) then. Which
within 2minutes. It is
hardly practical in production.
-Original Message-
From: Bowen Song via user
Sent: Thursday, July 7, 2022 8:43 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day seems
, 2022 8:43 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes
EXTERNAL
86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day seems
obviously too long. I would check the network bandwidth, disk IO and CPU usage
and find out what is the bottleneck.
On 07/07/2022 15:48
86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day
seems obviously too long. I would check the network bandwidth, disk IO
and CPU usage and find out what is the bottleneck.
On 07/07/2022 15:48, Marc Hoppins wrote:
Hi all,
Cluster of 2 DC and 24 nodes
DC1 (RF3) = 12
What version are you using?
When you run `nodetool netstats` on the joining node, what is the output?
How much data is there per node (presumably more than 86G)?
On Thu, Jul 7, 2022 at 7:49 AM Marc Hoppins wrote:
> Hi all,
>
> Cluster of 2 DC and 24 nodes
>
> DC1 (RF3) = 12 nodes, 16 tokens
Increasing RF will result in nodes that previously didn't have a replica of
the data now being responsible for it. This means that a repair is required
after increasing the RF.
Until the repair completes you will suffer from inconsistencies in data.
For example, in a 3 node cluster with RF 2,
I think the token ranges that are clean/completed and potentially streamed down
to additional node , won’t be cleaned again so potentially you’ll need to run
cleanup once again.
Can you can stop cleanup, add additional node and start cleanup over again so
to get nodes clean in single shot!
On Sun, Jun 28, 2015 at 10:46 AM, Anuj Wadehra anujw_2...@yahoo.co.in
wrote:
Thanks Jake!! But I think most people have 2.0.x in Production right now
as 2.1.6 is very recently declared Production Ready. I think the bug is too
important to be left open in 2.0.x as it leads to data loss. Should
Thanks Jake!! But I think most people have 2.0.x in Production right now as
2.1.6 is very recently declared Production Ready. I think the bug is too
important to be left open in 2.0.x as it leads to data loss. Should I open JIRA?
ThanksAnuj Wadehra
On Thursday, 25 June 2015 2:47 AM,
This is no longer an issue in 2.1.
https://issues.apache.org/jira/browse/CASSANDRA-2434
We now make sure the replica we bootstrap from is the one that will no
longer own that range
On Wed, Jun 24, 2015 at 4:58 PM, Alain RODRIGUEZ arodr...@gmail.com wrote:
It looks to me that can indeed happen
It looks to me that can indeed happen theoretically (I might be wrong).
However,
- Hinted Handoff tends to remove this issue, if this is big worry, you
might want to make sure HH are enabled and well tuned
- Read Repairs (synchronous or not) might have mitigate things also, if you
read fresh
Start one node at a time. Wait 2 minutes before starting each node.
How much data and nodes you have already? Depending on that, the streaming
of data can stress on the resources you have.
I would recommend to start one and monitor, if things are ok, add another
one. And so on.
Regards,
Carlos
unsubscribe
On Apr 20, 2015, at 8:08 AM, Carlos Rolo r...@pythian.com wrote:
Independent of the snitch, data needs to travel to the new nodes (plus all
the keyspace information that goes via gossip). So I won't bootstrap them all
at once, even if it is only for network traffic generated.
Hi Colin,
To remove your address from the list, send a message to:
user-unsubscr...@cassandra.apache.org
Cheers,
Matt
*From:* Colin Clark [mailto:co...@clark.ws]
*Sent:* 20 April 2015 14:10
*To:* user@cassandra.apache.org
*Subject:* Re: Adding nodes to existing cluster
Thanks for the response.
Sure we'll monitor as we're adding nodes.
We're now using 6 nodes on each DC. (We have 2 DCs)
Each node contains ~800GB
Do you know how rack configurations are relevant here?
Do you see any reason to bootstrap them one by one if we're not using
rack awareness?
On Mon,
Independent of the snitch, data needs to travel to the new nodes (plus all
the keyspace information that goes via gossip). So I won't bootstrap them
all at once, even if it is only for network traffic generated.
Don't forget to run cleanup on the old nodes once all nodes are in place to
reclaim
The documentation is referring to Consistent Range Movements.
There is a change in 2.1 that won't allow you to bootstrap multiple nodes
at the same time unless you explicitly turn off consistent range movements.
Check out the jira:
https://issues.apache.org/jira/browse/CASSANDRA-2434
All the
OK.
Thanks.
I'll monitor the resources status (network, memory, cpu, io) as I go
and try to bootsrap them at chunks which seems not to have a bad
impact.
Will do regarding the cleanup.
Thanks!
On Mon, Apr 20, 2015 at 4:08 PM, Carlos Rolo r...@pythian.com wrote:
Independent of the snitch, data
On Mon, Nov 18, 2013 at 10:28 AM, Carlos Alvarez cbalva...@gmail.comwrote:
Here
http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/operations/ops_add_node_to_cluster_t.html
says
that it is needed to wait 2 minutes between adding nodes.
I was trying to figure out why, and
Thank you very much. I'll try to find the code.
Carlos.
On Thu, Nov 21, 2013 at 4:35 AM, Aaron Morton aa...@thelastpickle.comwrote:
Of the top of my head there are some delays in the code do with schema
synchronisation and bootstrapping. I’m not sure if it adds up to two
minutes but it
is this understanding correct we had a 12 node cluster with 256 vnodes on each
node (upgraded from 1.1), we added two additional nodes that streamed so much
data (600+Gb when other nodes had 150-200GB) during the joining phase that they
filled their local disks and had to be killed ?
Can you
Did you update num_tokens on the existing hosts and restart them, before
you tried bootstrapping in the new node? If the new node tried to stream
all the data in the cluster then this would be consistent with you having
missed that step.
You should see Calculating new tokens in the logs of the
Opened a ticket:
https://issues.apache.org/jira/browse/CASSANDRA-5525
On Mon, Apr 29, 2013 at 2:24 AM, aaron morton aa...@thelastpickle.comwrote:
is this understanding correct we had a 12 node cluster with 256 vnodes on
each node (upgraded from 1.1), we added two additional nodes that
We're going to try running a shuffle before adding a new node again... maybe
that will help
I don't think hurt but I doubt it will help.
It seems when new nodes join, they are streamed *all* sstables in the
cluster.
How many nodes did you join, what was the num_tokens ?
Did you
On Sun, Apr 28, 2013 at 2:19 PM, aaron morton aa...@thelastpickle.comwrote:
We're going to try running a shuffle before adding a new node again...
maybe that will help
I don't think hurt but I doubt it will help.
We had to bail on shuffle since we need to add capacity ASAP and not in 20
I am using the same version and observed something similar.
I've added a new node, but the instructions from Datastax did not work for me.
Then I ran nodetool rebuild on the new node. After finished this command, it
contained two times the load of the other nodes. Even when I ran nodetool
Small relief we're not the only ones that had this issue.
We're going to try running a shuffle before adding a new node again...
maybe that will help
- John
On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral
fsob...@igcorp.com.br wrote:
I am using the same version and
I believe that nodetool rebuild is used to add a new datacenter, not just
a new host to an existing cluster. Is that what you ran to add the node?
-Bryan
On Fri, Apr 26, 2013 at 1:27 PM, John Watson j...@disqus.com wrote:
Small relief we're not the only ones that had this issue.
We're
rishabh.agra...@impetus.co.in:
I am using storage port 7000 (deafult) across all nodes.
-Original Message-
From: Maki Watanabe [mailto:watanabe.m...@gmail.com]
Sent: Wednesday, March 14, 2012 3:42 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes to cluster (Cassandra 1.0.8
After clearing the system files I was able to successfully join the node.
thanks.
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Thursday, March 15, 2012 2:13 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
There have been two similar reports
Do you use same storage_port across 3 nodes?
Can you access to the storage_port of the seed node from the last (failed) node?
2012/3/14 Rishabh Agrawal rishabh.agra...@impetus.co.in:
I was able to successfully join a node to already existing one-node cluster
(without giving any intital_token),
I am using storage port 7000 (deafult) across all nodes.
-Original Message-
From: Maki Watanabe [mailto:watanabe.m...@gmail.com]
Sent: Wednesday, March 14, 2012 3:42 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
Do you use same storage_port
To: user@cassandra.apache.org
Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
Do you use same storage_port across 3 nodes?
Can you access to the storage_port of the seed node from the last (failed)
node?
2012/3/14 Rishabh Agrawal rishabh.agra...@impetus.co.in:
I was able to successfully
14, 2012 3:42 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes to cluster (Cassandra 1.0.8)
Do you use same storage_port across 3 nodes?
Can you access to the storage_port of the seed node from the last (failed)
node?
2012/3/14 Rishabh Agrawal rishabh.agra...@impetus.co.in:
I
]
Sent: Monday, November 01, 2010 6:08 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes in new data center
On Mon, Nov 1, 2010 at 6:01 PM, Henry Luo h...@choicestream.com wrote:
We have a cluster running in one data center, and are adding some in a new
data center. There are some data
-
From: Henry Luo
Sent: Tuesday, November 02, 2010 12:10 PM
To: user@cassandra.apache.org
Subject: RE: Adding nodes in new data center
As suggested, we did do removetoken, set autobootstrap true, initial token and
start. Now the new nodes are in the Boostrapping state for a while, but nothing
On Mon, Nov 1, 2010 at 6:01 PM, Henry Luo h...@choicestream.com wrote:
We have a cluster running in one data center, and are adding some in a new
data center. There are some data already in the current cluster.
We did something wrong at first by not having AutoBootstrap on, when we saw
no
You need to specify your initial tokens. LoadBalance really doesn't do a
good job of balancing the load. Take a look at Load Balancing in
http://wiki.apache.org/cassandra/Operations There is a little python script
in there to help you pick tokens for a given cluster size.
If you don't want to
65 matches
Mail list logo
