Re: Migrating from DSE5.1.2 to Opensource cassandra

2018-12-06 Thread Jonathan Koppenhofer
Just to add a few additional notes on the in-place replacement. * We had to remove system.local and system.peers * Since we remove those system tables, you also have to put replace_address_first_boot in cassandra-env with the same IP address. * We also temporarily add the node as a seed to avoid

Tool to decide which node to decommission (using vnodes)

2018-12-06 Thread John Sumsion
Here is a tool I worked on to figure out which node to decommission that will leave you with the most even token balance afterwards. https://github.com/jdsumsion/vnode-decommission-calculator Feel free to use or enhance as you desire. John...

Determine size of duplicate data

2018-12-06 Thread Ian Spence
Hello, Is there a way to determine the size of “duplicate” date? I.E. data that a node no longer owns after expanding the ring? Data that is removed with a nodetool cleanup? Thank you! Ian Spence intermediate devops engineer Global Relay

Re: Sporadic high IO bandwidth and Linux OOM killer

2018-12-06 Thread Riccardo Ferrari
Hi, To be honest I've never seen the OOM in action on those instances. My Xmx was 8GB just like yours and that let me think you have some process that is competing for memory, is it? Do you have any cron, any backup, anything that can trick the OOMKiller ? My unresponsiveness was seconds long.

streaming errors with sstableloader

2018-12-06 Thread Ivan Iliev
Hello community, I'm receiving some strange streaming errors while trying to restore certain sstables snapshots with sstableloader to a new cluster. While the cluster is up and running and nodes are communicating with each other, I can see streams failing to the nodes with no obvious reason and

Re: Sporadic high IO bandwidth and Linux OOM killer

2018-12-06 Thread Oleksandr Shulgin
On Thu, Dec 6, 2018 at 11:14 AM Riccardo Ferrari wrote: > > I had few instances in the past that were showing that unresponsivveness > behaviour. Back then I saw with iotop/htop/dstat ... the system was stuck > on a single thread processing (full throttle) for seconds. According to > iotop that

Re: Sporadic high IO bandwidth and Linux OOM killer

2018-12-06 Thread Riccardo Ferrari
Alex, I had few instances in the past that were showing that unresponsivveness behaviour. Back then I saw with iotop/htop/dstat ... the system was stuck on a single thread processing (full throttle) for seconds. According to iotop that was the kswapd0 process. That system was an ubuntu 16.04

Fwd: Re: How to gracefully decommission a highly loaded node?

2018-12-06 Thread onmstester onmstester
After few hours, i just removed the node. done another node decommissioned, which finished successfully (the writer app was down, so no pressure on the cluster)  Started another node decommission (third), Since didn't have time to wait for decommissioning to finish, i started the writer