Re: node decommission

No, decommission does not decrease the load, as it only streams the data to other nodes, but doesn't remove them locally. However, decommission also shouldn't increase the load either. I can't offer an explanation for the load increase in your case. On 26/09/2022 15:03, Marc Hoppins wrote:

node decommission

Hulloa all, I started a decommission. Node load was 1.08TiB. After 6 or so hours the load is at 1.12TiB. Shouldn't it be DECREASING?

Re: Best approach for node decommission

If it helps you can use 'nodetool netstats' to monitor . If you do not see any streams stuck there , then you should be good . Also if you are concerned , you can run "nodetool repair" after the decommission just to make sure . I think what you are doing is okay . If the node is alive then

Best approach for node decommission

Hi All, Can anybody let me know best approach for decommissiong a node in the cluster. My cluster is using vnodes, is there any way to verify all the data of the decommissioning node has been moved to remaining nodes, before completely shutting down the server. I followed below procedure :

Re: Deleted data comes back on node decommission

4th and 5th >> node >> 7) delete that record from table (consistency 'all') >> 8) 'nodetool flush' on all 5 nodes, wait gc_grace_seconds, 'nodetool >> compact' on nodes which responsible for that key, check that key and >> tombstone gone using sstabledump >> 9) d

Re: Deleted data comes back on node decommission

nodetool getendpoints' find key that moved to both 4th and 5th > node > 7) delete that record from table (consistency 'all') > 8) 'nodetool flush' on all 5 nodes, wait gc_grace_seconds, 'nodetool > compact' on nodes which responsible for that key, check that key and > tombston

Re: Deleted data comes back on node decommission

stone gone using sstabledump 9) decommission 5th node, decommission 4th node 10) select data from table where key=key (consistency quorum) And the row is here. It sounds like bug in cassandra but since it is documented here https://docs.datastax.com/en/cassandra/3.0/cassandra/operati ons/opsAddNode

Re: Deleted data comes back on node decommission

Hi Max, I don't know if it's related to your issue but on a side note, if you decide to use Reaper (and use full repairs, not incremental ones), but mix that with "nodetool repair", you'll end up with 2 pools of SSTables that cannot get compacted together. Reaper uses subrange repair which

Re: Deleted data comes back on node decommission

The generation (integer id in file names) doesn’t matter for ordering like this It matters in schema tables for addition of new columns/types, but it’s irrelevant for normal tables - you could do a user defined compaction on 31384 right now and it’d be rewritten as-is (minus purgable data)

Re: Deleted data comes back on node decommission

Hi, Kurt. Thank you for response. Repairs are marked as 'done' without errors in reaper history. Example of 'wrong order': * file mc-31384-big-Data.db contains tombstone:     {     "type" : "row",     "position" : 7782,     "clustering" : [ "9adab970-b46d-11e7-a5cd-a1ba8cfc1426"

Re: Deleted data comes back on node decommission

' on all 5 nodes, wait gc_grace_seconds, 'nodetool compact' on nodes which responsible for that key, check that key and tombstone gone using sstabledump 9) decommission 5th node, decommission 4th node 10) select data from table where key=key (consistency quorum) And the row is here. It sounds like

Re: Deleted data comes back on node decommission

Are you positive your repairs are completing successfully? Can you send through an example of the data in the wrong order? What you're saying certainly shouldn't happen, but there's a lot of room for mistakes. On 14 Dec. 2017 20:13, "Python_Max" wrote: > Thank you for

Re: Deleted data comes back on node decommission

Thank you for reply. No, I did not execute 'nodetool cleanup'. Documentation https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRemoveNode.html does not mention that cleanup is required. Do yo think that extra data which node is not responsible for can lead to zombie data?

Re: Deleted data comes back on node decommission

> there (I used sstabledump to detect this state). > > Some of that rows have tombstone and data in wrong order: the data located at > more recent sstable by index but relative tombstone is located in previous > sstable (shouldn't it be opposite?). CQL select does not return that

Deleted data comes back on node decommission

(correct behaviour) until node decommission. sstables with non-compacted rows have very old repairedAt value (about 10 days before the first deleted row in sstable which should have been compacted long time ago). 2. Streaming. When moving the node out of cluster 'nodetool decommission' is used

Re: node decommission throttled

On Thu, Dec 8, 2016 at 3:27 AM, Aleksandr Ivanov wrote: > On sending side no high CPU/IO/etc utilization. > But on receiving node I see that one "STREAM-IN" thread takes 100% CPU and > it just doesn't scale by design since "Each stream is a single thread" >

Re: node decommission throttled

Nope, no MVs On Thu, Dec 8, 2016 at 11:31 AM, Benjamin Roth wrote: > Just an educated guess: you have materialized Views? They are known to > Stream very slow > > Am 08.12.2016 10:28 schrieb "Aleksandr Ivanov" : > >> Yes, I use compression. >> Tried

Re: node decommission throttled

Just an educated guess: you have materialized Views? They are known to Stream very slow Am 08.12.2016 10:28 schrieb "Aleksandr Ivanov" : > Yes, I use compression. > Tried without and it gave ~15% increase in speed, but is still too low > (~35Mbps) > > On sending side no high

Re: node decommission throttled

On sending side no high CPU/IO/etc utilization. But on receiving node I see that one "STREAM-IN" thread takes 100% CPU and it just doesn't scale by design since "Each stream is a single thread" ( http://www.mail-archive.com/user@cassandra.apache.org/msg42095.html) Maybe your System cannot

Re: node decommission throttled

Yes, I use compression. Tried without and it gave ~15% increase in speed, but is still too low (~35Mbps) On sending side no high CPU/IO/etc utilization. But on receiving node I see that one "STREAM-IN" thread takes 100% CPU and it just doesn't scale by design since "Each stream is a single

Re: node decommission throttled

Maybe your System cannot Stream faster. Is your cpu or hd/ssd fully utilized? Am 07.12.2016 16:07 schrieb "Eric Evans" : > On Tue, Dec 6, 2016 at 9:54 AM, Aleksandr Ivanov wrote: > > I'm trying to decommission one C* node from 6 nodes cluster and see

Re: node decommission throttled

On Tue, Dec 6, 2016 at 9:54 AM, Aleksandr Ivanov wrote: > I'm trying to decommission one C* node from 6 nodes cluster and see that > outbound network traffic on this node doesn't go over ~30Mb/s. > Looks like it is throttled somewhere in C* Do you use compression? Try taking a

node decommission throttled

I'm trying to decommission one C* node from 6 nodes cluster and see that outbound network traffic on this node doesn't go over ~30Mb/s. Looks like it is throttled somewhere in C* Should stream_throughput_outbound_megabits_per_sec limit apply on "decommissioning streams" as well? >From my

Stale node decommission?

Hi guys, I’m decommissioning a node and the nodetool netstats bytes count doesn’t move, have you seen it before? Mode: LEAVING Unbootstrap 6eeec0c0-6d9a-11e5-addd-8bc1b1cfbf0a /XXX.XXX.XXX.XXX Sending 0 files, 2773603294 bytes total. Already sent 0 files, 0 bytes total

Re: Stale node decommission?

On Thu, Oct 8, 2015 at 5:20 AM, Carlos Alonso wrote: > Hi guys, I’m decommissioning a node and the nodetool netstats bytes count > doesn’t move, have you seen it before? > Your decom stream is hung, if you don't have a version with the new non-infinite timeout for streams,

Re: Stale node decommission?

I ended up killing the node (nodetool removenode) because the streaming was not working (probably because of the different sstables versions) These guys were appearing in logs: ERROR [STREAM-IN-/XXX.XXX.XXX.XXX] 2015-10-08 12:52:43,202 StreamSession.java:505 - [Stream

Maximum node decommission // bootstrap at once.

We're in the middle of migrating datacenters. We're migrating from 13 nodes to 30 nodes in the new datacenter. The plan was to bootstrap the 30 nodes first, wait until they have joined. then we're going to decommission the old ones. How many nodes can we bootstrap at once? How many can we

Re: Maximum node decommission // bootstrap at once.

Honestly, we've had more luck bootstrapping in our old DC (defining topology properties as the new DC) and using rsync to migrate the data files to new machines in the new datacenter. We had 10gig within the datacenter but significantly less than this cross-DC, which lead to a lot of broken

Re: Maximum node decommission // bootstrap at once.

On Tue, Oct 6, 2015 at 12:32 PM, Kevin Burton wrote: > How many nodes can we bootstrap at once? How many can we decommission? > short answer : 1 node can join or part at simultaneously longer answer : https://issues.apache.org/jira/browse/CASSANDRA-2434 /

Re: Maximum node decommission // bootstrap at once.

I'm not sure which is faster/easier. Just joining one box at a time and then decommissioning or using replace_address. this stuff is always something you do rarely and then more complex than it needs to be. This complicates long term migration too. Having to have gigabit is somewhat of a

Re: Maximum node decommission // bootstrap at once.

OH. interesting. Yeah. That's another strategy. We've already done a bunch of TCP tuning... we get about 1Gbit with large TCP windows. So I think we have that part done. It's sad that CS can't resume... Plan be we will just rsync the data.. Does it pretty much work just by putting the data in

Re: Maximum node decommission // bootstrap at once.

On Tue, Oct 6, 2015 at 2:14 PM, Kevin Burton wrote: > Plan be we will just rsync the data.. Does it pretty much work just by > putting the data in a directory or do you have to do anything special? > http://www.pythian.com/blog/bulk-loading-options-for-cassandra/ Be

Re: Maximum node decommission // bootstrap at once.

Robert, I might be misinterpreting you but I *think* your link is talking about bootstrapping a new node by bulk loading replica data from your existing cluster? I was referring to using Cassandra's bootstrap to get the node to join and run (as a member of DC2 but with physical residence in DC1),

Re: Maximum node decommission // bootstrap at once.

On Tue, Oct 6, 2015 at 3:41 PM, Bryan Cheng wrote: > Robert, I might be misinterpreting you but I *think* your link is talking > about bootstrapping a new node by bulk loading replica data from your > existing cluster? > Sure... more generically it's about treating the