Nodetool Repair --full

[Jonathan Baynes]

Hi Community,

Can someone confirm, as the documentation out on the web is so contradictory 
and vague.

Nodetool repair -full if I call this, do I need to run this on ALL my nodes or 
is just the once sufficient?

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0)7884111546
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: How do you monitoring Cassandra Cluster?

[Jonathan Baynes]

I’m using Quest Foglight, very good if you have budget for it??

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0)7884111546
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
—
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading



From: Rahul Singh [mailto:rahul.xavier.si...@gmail.com]
Sent: 21 June 2018 15:15
To: user@cassandra.apache.org; user@cassandra.apache.org
Subject: Re: How do you monitoring Cassandra Cluster?

I’ve collected a bunch at 
http://leaves.anant.us/#!/?tag=cassandra,monitoring<https://urldefense.proofpoint.com/v2/url?u=http-3A__leaves.anant.us_-23-21_-3Ftag-3Dcassandra-2Cmonitoring&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ZIASPHopBGyBDLyYLE8hmO-g4WzF4M3QpMj4R-VcCLk&s=ppcLGaAXBihlMYI3iNod_cOkaYUEWIXCBbi5ZL5SMs0&e=>

I reommend Grafana / Prometheus if you don’t have DSE (which has OpsCenter)


--
Rahul Singh
rahul.si...@anant.us<mailto:rahul.si...@anant.us>

Anant Corporation
On Jun 19, 2018, 1:06 PM -0400, Romain Gérard 
mailto:romain.ger...@erebe.eu>>, wrote:

Hi Felipe,

You can use this project 
https://github.com/criteo/cassandra_exporter<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_criteo_cassandra-5Fexporter&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ZIASPHopBGyBDLyYLE8hmO-g4WzF4M3QpMj4R-VcCLk&s=rW9TdHXho8o-6oWLMZUxHNFDS4gDMeDqjxMMixlomwc&e=>
 if you are using Prometheus (Disclamer, I am one of the author of it).
There is included a Grafana dashboard that aggregate metrics per cluster for 
you, and in the "edit view" of each chart there are hidden queries that let you 
drill down by nodes.
In any case, you can look at the json in order to grasp how to aggregate 
queries.

Let me know if you need more information regarding the setup.

If you end-up using the project, give a star to the project it is always 
pleasant,
Regards,
Romain Gérard

On Jun 18 2018, at 3:25 pm, Felipe Esteves 
mailto:felipe.este...@b2wdigital.com>> wrote:

Hi, everyone,

I'm running some tests to monitor Cassandra 3.x with jmx_exporter + prometheus 
+ grafana.
I've managed to config it all and use the dashboard 
https://grafana.com/dashboards/5408<https://urldefense.proofpoint.com/v2/url?u=https-3A__grafana.com_dashboards_5408&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ZIASPHopBGyBDLyYLE8hmO-g4WzF4M3QpMj4R-VcCLk&s=P6v555HpxIkv4BSdpbTESNAuBGzo4RmKYZTCfwFlIFY&e=>

However, I still can't aggregate metrics from all my cluster, just nodes 
individually.
Any tips on how to do that?

Also, Opscenter gives some datacenter aggregation, I think it comes from 
nodetool as I didn't seen any metrics about that.
Anyone having success on that?

cheers!

Em qua, 28 de jun de 2017 às 19:43, Petrus Gomes 
mailto:petru...@gmail.com>> escreveu:
I'm using JMX+Prometheus and Grafana.
JMX = 
https://github.com/prometheus/jmx_exporter<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_prometheus_jmx-5Fexporter&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ZIASPHopBGyBDLyYLE8hmO-g4WzF4M3QpMj4R-VcCLk&s=7n7CnZc_rckkQ2h-lHuGpWuVIbdiinXYAD22IXQ8Ddc&e=>
Prometheus + Grafana = 
https://prometheus.io/docs/visualization/grafana/<https://urldefense.proofpoint.com/v2/url?u=https-3A__prometheus.io_docs_visualization_grafana_&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ZIASPHopBGyBDLyYLE8hmO-g4WzF4M3QpMj4R-VcCLk&s=4e0XS2_1x-teGDTHdosxEP2g92IKFyH338kNZ5JY6Oc&e=>

There are some dashboard examples like that: 
https://grafana.com/dashboards/371<https://urldefense.proofpoint.com/v2/url?u=https-3A__grafana.com_dashboards_371&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ZIASPHopBGyBDLyYLE8hmO-g4WzF4M3QpMj4R-VcCLk&s=EPiqDUO-0ZBgQuhHM6TX_uArTgViVWlJlp_NBX05Am0&e=>
Looks good.

Thanks,
Petrus Silva

On Wed, Jun 28, 2017 at 5:55 AM, Peng Xiao 
<2535...@qq.com<mailto:2535...@qq.com>> wrote:
Dear All,

we are currently using Cassandra 2.1.13,and it has grown to 5TB size with 32 
nodes in one DC.
For monitoring,opsCenter does not  send alarm and not free in h

RE: Cassandra recommended server uptime?

[Jonathan Baynes]

I haven’t restarted a node (service) unless there are changes made to config, 
and for restarting the server its incredibly rare. In fact since Production has 
been up I think ive restrted the servers twice (both for DR occurences) in a 
Year.

There is very little need to reboot the servers and unless you need to keep 
changing config I wouldn’t worry about restarting nodes.

From: Vsevolod Filaretov [mailto:vsfilare...@gmail.com]
Sent: 17 July 2018 13:23
To: user@cassandra.apache.org
Subject: Re: Cassandra recommended server uptime?

@rahul.xavier.si...@gmail.com, 
@simon.fontana.oscars...@ericsson.com,

Thank you for answers.

I've got a really heavy data model, but no GC problems.

Moreover, gclog shows very efficient GC process and hardly any GC pauses.

So, the question remains open: how general is the practice of semi-periodical 
C* node restarts?

Best regards,
Vsevolod.

вт, 17 июл. 2018 г., 14:39 Rahul Singh 
mailto:rahul.xavier.si...@gmail.com>>:
It’s likely that if you have server stability issues its because of data model 
or compaction strategy configurations which lead to out of memory issues or 
massive GC pauses. Rebooting wouldn’t solve those issues.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation
On Jul 17, 2018, 7:28 AM -0400, Simon Fontana Oscarsson 
mailto:simon.fontana.oscars...@ericsson.com>>,
 wrote:

Not anything that I'm aware of. Cassandra can run at months/years without 
rebooting.
It is better to monitor your nodes and if you find anything abnormal a restart 
can help.

--
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com

On tis, 2018-07-17 at 12:09 +0300, Vsevolod Filaretov wrote:

Good time of day everyone;

Does Cassandra have a "recommended uptime"? I.e., does regular Cassandra node 
reboots help anything? Are periodical node reboots recommended for general 
system stability?

Best regards, Vsevolod.



This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: JOB | Permanent Java Development Manager (Montreal, Canada)

[Jonathan Baynes]

Agree we don't need this.

-Original Message-
From: Tobin Landricombe [mailto:to...@sensatus.com]
Sent: 01 August 2018 16:21
To: user@cassandra.apache.org
Subject: Re: JOB | Permanent Java Development Manager (Montreal, Canada)

Spam. Let this one in and the channel will fill up with them.

> On 1 Aug 2018, at 16:15, Eric Evans  wrote:
>
> On Tue, Jul 31, 2018 at 11:42 PM James Tobin  wrote:
>> Hello, I'm working with an employer that is looking to hire (for
>> their Montreal office) a permanent development manager that has
>> extensive hands-on Java coding experience.  Consequently I had hoped
>> that some members of this mailing list may like to discuss with me
>> further; off-list.  I can be reached using "JamesBTobin (at) Gmail (dot) 
>> Com".
>> Kind regards, James
>
> I don't think this is appropriate; An employer looking for Cassandra
> experience seems OK, but this just looks like recruiter spam to me.
>
> I'll let others on the list chime-in if they disagree, but I'm
> inclined to unsubscribe/ban anyone making posts like this.
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org



This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Error during truncate: Cannot achieve consistency level ALL , how to fix it

[Jonathan Baynes]

What RF is your system_auth keyspace?

If its one, match it to the user keyspace, and restart the node.

From: sha p [mailto:shatestt...@gmail.com]
Sent: 19 September 2018 11:49
To: user@cassandra.apache.org
Subject: Error during truncate: Cannot achieve consistency level ALL , how to 
fix it

Hi All,
 I am new to Cassandra. Following below link

https://grokonez.com/spring-framework/spring-data/start-spring-data-cassandra-springboot#III_Sourcecode


I have three node cluster , keyspace set with RF = 2 , but when I run this 
application from above source code bellow error is thrown
 Caused by: com.datastax.driver.core.exceptions.TruncateException: Error during 
truncate: Cannot achieve consistency level ALL """ 


What wrong i am doing here ..How to fix it ? Plz help me.

Regards,
Shyam



This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Rolling back Cassandra upgrades (tarball)

[Jonathan Baynes]

Hi community,

Wonder if you can help me.

I have a cluster on v3.0.11 I am planning to upgrade this to 3.10.
Is rolling back the binaries a viable solution?

From reading blogs and such, I’ve read that downgrading is a no-no. Whilst it 
is possible literally and in theory possible, many of the Cassandra experts say 
you can only go forward and not back.

In our lab environment I have attempted a rollback.  The version was on 3.10, I 
have rolled it back to 3.0.11 and then upgraded back to 3.10 , no sstable 
restoration, just a straight copy back of the binaries of the old version, and 
and copy over of the new binaries. A rolling restart and then a repair. 
Everything went smoothly, no issues. The repair process has been working 
without issue since, I’ve not noticed any issues with the cluster.

Firstly Is there anything I should be aware of, would this have caused an issue?

In a production upgrade scenario should  I avoid rolling back, unless I 
restore? I’d like to understand if it’s a viable solution or not.

The upgrade we are due to perform is from the tarball files, the bash script 
that we have engineered work quickly and efficiently, I don’t have any worries 
of the upgrade, I just need to understand if I should factor rollback as part 
of this plan.

In the event that a failure of the upgrade occurs should I plan to restore the 
data and reinstall the version?

Thanks in advance
J




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: openjdk for cassandra production cluster

[Jonathan Baynes]

I’m using OpenJDK on version 3.10 (rehl 7.3) in Production and have the warning 
in the logs. I have a stable system, I think you can ignore it.

From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: 10 October 2018 15:46
To: user@cassandra.apache.org
Subject: Re: openjdk for cassandra production cluster

The warning should be removed (if it hasn’t already), it’s unnecessary at this 
point

On Wed, Oct 10, 2018 at 7:41 AM Prachi Rath 
mailto:prachirat...@gmail.com>> wrote:
HI users,
I have created a cassandra cluster with openjdk 1.8.0_181 version.(cassandra 
2.1.17)
started each node, cluster looks healthy,but in  the log files saw the WARN
message below:

WARN [main] 2014-01-28 06:02:17,861 CassandraDaemon.java (line 155) OpenJDK
is not recommended. Please upgrade to the newest Oracle Java release

Is this message  WARN informational only or can it be real issue?
Did any one noticed something like this (Or using openjdk for production 
environment)

Thanks ,
Prachi
--
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade



This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: Cassandra issue “Unable to gossip with any seeds”

[Jonathan Baynes]

Hi Bhargav


I had this recently when making changes to a cluster, and It was down to the 
Listen_address & broadcast_address. Check your cassandra.yaml and make sure 
that your listen_address/broadcast_address and "seeds" values match. Make sure 
the Data folder is empty as well.

I would go through your yaml, be diligent its there that is the problem.

Thanks
J


From: Mokkapati, Bhargav (Nokia - IN/Chennai) 
[mailto:bhargav.mokkap...@nokia.com]
Sent: 04 July 2017 14:00
To: user@cassandra.apache.org
Subject: Cassandra issue “Unable to gossip with any seeds”

Hi Team,

I am getting the below issue when I started a node. “Unable to gossip with any 
seeds”

[cid:image001.jpg@01D2F4D3.1E74E5E0]

I am using Apache Cassandra 3.0.6 version.

I have 5 nodes in the cluster,10.50.20.151 , 10.50.20.152 , 10.50.20.153 , 
10.50.20.154 , 10.50.20.155

Seed nodes -  10.50.20.153 , 10.50.20.154 , 10.50.20.155

All the nodes have same Cassandra.yaml configuration file

I am getting this issue only in node2 (10.50.20.152). It didn’t even joined the 
cluster.

[cid:image002.png@01D2F4D3.1E74E5E0]


Also node2 is pinging from seed nodes and vice versa.

[cid:image003.png@01D2F4D3.1E74E5E0]


Please let me know if any of you have faced the same issue and found the 
solution.

Thank you very much

Regards,
Bhargav









This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Move Production data to Development Cluster

[Jonathan Baynes]

Hi,

Can anyone help me. I'm trying (and failing) to move my 3 node C* data from my 
Production Environment to my Development 3 node cluster.

Here is the fine print...

Oracle Linux 7.3
C* 3.0.11

3 Nodes ((virtual Nodes 256))
1 Keyspace (replication factor 3) Quorum Consistency
1 table

Snapshot taken on each node.

Attempt 1

I've tried the following 
(http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSnapshotRestoreNewCluster.html)



  1.  From the old cluster, retrieve the list of tokens associated with each 
node's IP:

$ nodetool ring | grep ip_address_of_node | awk '{print $NF ","}' | xargs



I've done this for all 3 nodes, placed them together in one string

  1.  In the 
cassandra.yaml<http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml>
 file for each node in the new cluster, add the list of tokens you obtained in 
the previous step to the 
initial_token<http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configCassandra_yaml.html#configCassandra_yaml__initial_token>
 parameter using the same num_tokens setting as in the old cluster.
Added all the tokens from step one

  1.  Make any other necessary changes in the new cluster's 
cassandra.yaml<http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml>
 and property files so that the new nodes match the old cluster settings. Make 
sure the seed nodes are set for the new cluster.
  2.  Clear the system table data from each new node:

$ sudo rm -rf /var/lib/cassandra/data/system/*

This allows the new nodes to use the initial tokens defined in the 
cassandra.yaml<http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml>
 when they restart.

  1.  Start each node using the specified list of token ranges in new cluster's 
cassandra.yaml<http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml>:

initial_token: -9211270970129494930, -9138351317258731895, 
-8980763462514965928, ...

  1.  Create schema in the new cluster. All the schemas from the old cluster 
must be reproduced in the new cluster.
  2.  Stop the 
node<http://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartStopTOC.html>.
 Using nodetool refresh is unsafe because files within the data directory of a 
running node can be silently overwritten by identically named just-flushed 
SSTables from memtable flushes or compaction. Copying files into the data 
directory and restarting the node will not work for the same reason.
  3.  Restore the SSTable files 
snapshotted<http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsBackupSnapshotRestore.html>
 from the old cluster onto the new cluster using the same directories, while 
noting that the UUID component of target directory names has changed. Without 
restoration, the new cluster will not have data to read upon restart.
  4.  Restart the node.

When  I restart I get errors in the Yaml file pointing to the token ranges.

If I take out the tokens for 2 of the nodes and just use one node(s) tokens I 
can restore the data, but I get only a third of the row count id expect.

I then noticed I was only restoring  the snapshot from that one node, so , that 
made sense

So then I took all of the snapshots, from all of the nodes, placed them into a 
folder, re added all the tokens and re ran the process, but I get the token 
range error in the yaml again.


Attempt 2

So then I tried  SSTableLoader from the same folder with all the (3 nodes 
snapshots) and then I get corruption on the SSTable..


Advice

I've tried this so many ways its getting confusing.. what's right?

Can anyone give me some pointers as to the best route to migrate data from 
cluster to cluster? The Documentation is so vague and not detailed enough?

Do I copy the snapshots from all the nodes?
Do I just work on one node at a time?

Any suggestions please??

Thanks
J



Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This 

Starting Cassandrs after restore of Data - get error

[Jonathan Baynes]

Hi again,

Trying to restart my nodes after restoring snapshot data, initial tokens have 
been added in as per the instructions online.

In system.log I get this error (same error is I run nodetool cleanup)

Exception encountered during startup: The number of initial tokens (by 
initial_token) specified is different from num_tokens value


On both Cluster A and Cluster B the Num_tokens = 256

Ive taken the initial tokens from running this script

nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk 
'{print $NF ","}' | xargs > /tmp/tokens

when pasting in the tokens originally I got the an error, but this was due to 
the spacing between the tokens. That error has been resolved I'm just left with 
this one?

Any ideas

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: Starting Cassandrs after restore of Data - get error

[Jonathan Baynes]

Yes both clusters match I've checked 3 Times and diff'd it as well. Would file 
format have any affect I'm amending on windows machine and returning the file 
back to Linux

Thanks
J

Sent from my iPhone

On 7 Jul 2017, at 17:43, Nitan Kainth 
mailto:ni...@bamlabs.com>> wrote:

Jonathan,

Make sure initial tokens have values from back up cluster i.e. 256 tokens. It 
is possible to have typo.

On Jul 7, 2017, at 9:14 AM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:

Hi again,

Trying to restart my nodes after restoring snapshot data, initial tokens have 
been added in as per the instructions online.

In system.log I get this error (same error is I run nodetool cleanup)

Exception encountered during startup: The number of initial tokens (by 
initial_token) specified is different from num_tokens value


On both Cluster A and Cluster B the Num_tokens = 256

Ive taken the initial tokens from running this script

nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk 
'{print $NF ","}' | xargs > /tmp/tokens

when pasting in the tokens originally I got the an error, but this was due to 
the spacing between the tokens. That error has been resolved I’m just left with 
this one?

Any ideas

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com<mailto:jonathan.bay...@tradeweb.com>

<http://www.tradeweb.com/>   follow us:  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=6Y8NdF3H6u8tRTi5hVhL4s48FRlJILendSdwl23uaVc&s=rrL3dDGS2nYy57Xhn_q3b6YMS_ndgUljZQ8p0Q8XYdg&e=>
   
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=6Y8NdF3H6u8tRTi5hVhL4s48FRlJILendSdwl23uaVc&s=r5pLltW1HrPoL3HU0c5_KYUe0nkBfo1iv8b_LKZdfSE&e=>
—
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com<mailto:contac...@tradeweb.com> or by writing to us at 
the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited 
(company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our 
privacy policy, visit our website @ www.tradeweb.com<http://www.tradeweb.com/>.




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Removing Columns from production table

[Jonathan Baynes]

Hi

Is there anything I need to do after dropping a column and adding in a column 
to flush cassandra of the changes? We are experiencing issues with our front 
end application, and the developers are asking if the issue was caused by the 
change in schema, as I've done the drop and add of a column, have I missed 
something, do I need to make any further changes to the system with schema 
changes?

Using Cassandra 3.0.11

Thanks
J





This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: Removing Columns from production table

[Jonathan Baynes]

Thanks Jeff I have a horrible feeling it may be. I'll get the errors from the 
dev guys Monday and email the group, hopefully I can tie this down.

Thanks

Sent from my iPhone

> On 18 Aug 2017, at 17:24, Jeff Jirsa  wrote:
>
> Cassandra-13004


This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

C* 3 node issue -Urgent

[Jonathan Baynes]

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I've 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this

[cid:image004.png@01D31BF4.B7FB3460]

In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error
[cid:image005.png@01D31BF4.B7FB3460]

When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

I will also  mention I am on:

C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7

From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I've 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this

[cid:image004.png@01D31BF6.93C48130]

In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error
[cid:image005.png@01D31BF6.93C48130]

When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com<mailto:jonathan.bay...@tradeweb.com>

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

Yes I have the correct credentials I’m using cassandra/cassandra (superuser)
To test that theory I tried a different user and got this

Connection error: ('Unable to connect to any servers', {'10.172.115.63': 
AuthenticationFailed('Failed to authenticate to 10.172.115.63: Error from 
server: code=0100 [Bad credentials] message="Username and/or password are 
incorrect"',)})

Yes you are correct I do  have PasswordAuthenticator turned on .

From: Akhil Mehra [mailto:akhilme...@gmail.com]
Sent: 23 August 2017 10:05
To: user@cassandra.apache.org
Subject: Re: C* 3 node issue -Urgent

The cqlsh image say bad credentials. Just confirming that you have the correct 
username/password when logging on.

By turing on authentication I am assuming you mean using the 
PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.

Cheers,
Akhil
On 23/08/2017, at 8:59 PM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:

I will also  mention I am on:

C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7

From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org<mailto:user@cassandra.apache.org>'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I’ve 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this



In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error


When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com<mailto:jonathan.bay...@tradeweb.com>

<http://www.tradeweb.com/>   follow us:  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=h5M2yR_HjCB_ghUa2rzYYKcwH5VjWUPf_ttbCNUrdqA&e=>
   
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=yzqb445p2cV3fJfSIGPOcGNEkP7zIfxMWNo3jSw7TmM&e=>
—
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading

__

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

When trying to connect to CQLSH I get this

“Cannot achieve consistency level QUORUM”

2 of my 3 nodes are down. So this error is correct. But how do I sign into 
CQLSH to change this? Or better, how do I get the other 2 nodes back up?

From: Akhil Mehra [mailto:akhilme...@gmail.com]
Sent: 23 August 2017 10:05
To: user@cassandra.apache.org
Subject: Re: C* 3 node issue -Urgent

The cqlsh image say bad credentials. Just confirming that you have the correct 
username/password when logging on.

By turing on authentication I am assuming you mean using the 
PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.

Cheers,
Akhil
On 23/08/2017, at 8:59 PM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:

I will also  mention I am on:

C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7

From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org<mailto:user@cassandra.apache.org>'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I’ve 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this



In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error


When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com<mailto:jonathan.bay...@tradeweb.com>

<http://www.tradeweb.com/>   follow us:  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=h5M2yR_HjCB_ghUa2rzYYKcwH5VjWUPf_ttbCNUrdqA&e=>
   
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=yzqb445p2cV3fJfSIGPOcGNEkP7zIfxMWNo3jSw7TmM&e=>
—
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading


This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauth

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

@Kurt, You have hit the nail on the head.

Whilst it was an issue with the cassandra-env.sh file, and I fixed this by 
overwriting it with the default file, essentially rolling it back, which got me 
back in, the larger issue is with the system_auth db it is a REPL factor of 1 
and thus as stated in this doc 
http://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/sec/secConfSysAuthKeyspRepl.html

Attention: To prevent a potential problem logging into a secure cluster, set 
the replication factor of the system_auth and dse_security keyspaces to a value 
that is greater than 1. In a multi-node cluster, using the default of 1 
prevents logging into any node when the node that stores the user data is down.

Thank you ALL for coming back to me and assisting, Kurt, good shout and thanks 
for getting me where I need to be, which is back up and running.

Keep up the great work Community!!

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: 23 August 2017 11:14
To: User
Subject: Re: C* 3 node issue -Urgent

The cassandra user requires QUORUM consistency to be achieved for 
authentication. Normal users only require ONE. I suspect your system_auth 
keyspace has an RF of 1, and the node that owns the cassandra users data is 
down.

Steps to recover:
1. Turn off authentication on all the nodes
2. Restart the nodes and make sure they are UN
3. Alter system_auth to have a higher RF than 1 (3 is probably appropriate)
4. Turn auth back on and restart
5. Create a new user and use that from now on.

​



This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Cassandra Setup Question

[Jonathan Baynes]

Hi Community,

Quick question regarding Replication Factor.

In my Production Environment I currently have 6 nodes this will grow to 10 
shortly), over 2 datacentres, so currently 3 in each, we want to have 
Active/Passive setup so the Client will only speak to DC 1 via load balancing 
policy, but we want the data to replicate across both DCs , so that in the 
event of a DC failure the second DC acts as our DR DC and has  all the data as 
well.

The Consistency will be Quorum, but replication Factor does that need to be 3 
or 6.

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: system_auth replication factor in Cassandra 2.1

[Jonathan Baynes]

I recently came across an issue where by my user Keyspace was replicated by 3 
(I have 3 nodes) but my system_auth was default to 1, we also use 
authentication, I then lost 2 of my nodes and because authentication wasn’t 
replicated I couldn’t log in.

Once I resolved the issue, and got the nodes back up, I could then log back in, 
I too asked the community what was going on , and I was pointed to this

http://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/sec/secConfSysAuthKeyspRepl.html

it clearly states the following

Attention: To prevent a potential problem logging into a secure cluster, set 
the replication factor of the system_auth and dse_security keyspaces to a value 
that is greater than 1. In a multi-node cluster, using the default of 1 
prevents logging into any node when the node that stores the user data is down.



From: Chuck Reynolds [mailto:creyno...@ancestry.com]
Sent: 30 August 2017 16:51
To: user@cassandra.apache.org
Subject: system_auth replication factor in Cassandra 2.1

So I’ve read that if your using authentication in Cassandra 2.1 that your 
replication factor should match the number of nodes in your datacenter.

Is that true?

I have two datacenter cluster, 135 nodes in datacenter 1 & 227 nodes in an AWS 
datacenter.

Why do I want to replicate the system_auth table that many times?

What are the benefits and disadvantages of matching the number of nodes as 
opposed to the standard replication factor of 3?


The reason I’m asking the question is because it seems like I’m getting a lot 
of authentication errors now and they seem to happen more under load.

Also, querying the system_auth table from cqlsh to get the users seems to now 
timeout.


Any help would be greatly appreciated.

Thanks



This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: Cassandra - Nodes can't restart due to java.lang.OutOfMemoryError: Direct buffer memory

[Jonathan Baynes]

Can you tell us what version you are on?

-Original Message-
From: qf zhou [mailto:zhouqf2...@gmail.com]
Sent: 31 August 2017 10:52
To: user@cassandra.apache.org
Subject: Cassandra - Nodes can't restart due to java.lang.OutOfMemoryError: 
Direct buffer memory

I am running a cluster with 3 nodes. Then it occurs the following errors. After 
I restart one node,  the error happens again.
I don't know why.  Who Can help me ? Thank you !!



ERROR [ReadStage-31] 2017-08-31 15:08:20,878 JVMStabilityInspector.java:141 - 
JVM state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_111]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
~[na:1.8.0_111]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
~[na:1.8.0_111]
at 
org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:109) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:46) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:406)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:333)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:121) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:93) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:156) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:40) 
~[apache-cassandra-3.9.jar:3.9]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
 ~[caffeine-2.2.6.jar:na]
at 
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1892) 
~[na:1.8.0_111]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
 ~[caffeine-2.2.6.jar:na]
at 
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:220)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:138) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1709)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:102)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.initializeIterator(UnfilteredRowIteratorWithLowerBound.java:105)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.partitionLevelDeletion(LazilyInitializedUnfilteredRowIterator.java:81)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.partitionLevelDeletion(UnfilteredRowIteratorWithLowerBound.java:165)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:585)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.Sing

RE: Cassandra - Nodes can't restart due to java.lang.OutOfMemoryError: Direct buffer memory

[Jonathan Baynes]

I wonder if its related to this bug  (below) that’s currently unresolved, 
albeit it being reproduced way back in 2.1.11

https://issues.apache.org/jira/browse/CASSANDRA-10689


From: qf zhou [mailto:zhouqf2...@gmail.com]
Sent: 31 August 2017 10:58
To: user@cassandra.apache.org
Subject: Re: Cassandra - Nodes can't restart due to java.lang.OutOfMemoryError: 
Direct buffer memory

I am usingCassandra 3.9 with cqlsh 5.0.1.
在 2017年8月31日，下午5:54，Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> 写道：

again




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Understanding Messages in the Debug.log

[Jonathan Baynes]

Good Morning Community,

Can anyone help me understand what these messages (and I have a lot of them in 
the log) mean, they must be a symptom of something but I am unsure how to 
starting working this out.

I'm using C* 3.0.11, CQL 5.0.1.

I have 6 node ring over 2 DC's (3 in each DC) each of the nodes has this 
message in the log.

The message in the debug log is
DEBUG [GossipStage:1] 2017-09-21 09:19:52,627 FailureDetector.java:456 - 
Ignoring interval time of 2000275419


There are so many of these messages, I'd like to find the root cause of this, 
if anyone could help that would be great. Or if they can be ignored, how do I 
supress these?

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: Understanding Messages in the Debug.log

[Jonathan Baynes]

Thanks Jacob,

I’ve already seen this, but it doesn’t answer my question, as this is a 
production Cluster, which at present, has no data and is not being written too, 
its being built to go live shortly, so with no data, a Keyspace with a table 
designed but no data in it, only system databases, what on earth could be 
causing load on the cluster to cause the error message??



From: Jacob Shadix [mailto:jacobsha...@gmail.com]
Sent: 21 September 2017 12:34
To: user@cassandra.apache.org
Subject: Re: Understanding Messages in the Debug.log

check out this thread - 
https://www.mail-archive.com/user@cassandra.apache.org/msg52313.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mail-2Darchive.com_user-40cassandra.apache.org_msg52313.html&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=T5vuzfn8ElJgnkkZ4sQTkQUMwN-_GEOacqRbeVayQUQ&e=>

-- Jacob Shadix

On Thu, Sep 21, 2017 at 4:56 AM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:
Good Morning Community,

Can anyone help me understand what these messages (and I have a lot of them in 
the log) mean, they must be a symptom of something but I am unsure how to 
starting working this out.

I’m using C* 3.0.11, CQL 5.0.1.

I have 6 node ring over 2 DC’s (3 in each DC) each of the nodes has this 
message in the log.

The message in the debug log is
DEBUG [GossipStage:1] 2017-09-21 09:19:52,627 FailureDetector.java:456 - 
Ignoring interval time of 2000275419


There are so many of these messages, I’d like to find the root cause of this, 
if anyone could help that would be great. Or if they can be ignored, how do I 
supress these?

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street 
Avenue<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=8MElkv-02VTEMtwP2-AvmBTjsk3O-VMBNd3JtOBzp1o&e=>
  •  London EC2Y 
9DT<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=8MElkv-02VTEMtwP2-AvmBTjsk3O-VMBNd3JtOBzp1o&e=>
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com<mailto:jonathan.bay...@tradeweb.com>

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=qEj7m_M2g7N23-PtTGZmpbyO13Zqu-0qypSZg4i3qlA&e=>
[cid:image003.jpg@01CD26AD.4165F110] 
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=B5FSJ8OU-4_K4gemgZ7FC02TH3tkIGnZ6oSynGFEyNs&e=>
—
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com<mailto:contac...@tradeweb.com> or by writing to us at 
the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited 
(company number 3912826), 1 Fore Street Avenue London EC2Y 
9DT<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjid

RE: Understanding Messages in the Debug.log

[Jonathan Baynes]

Thanks Jeff,

They are on all instances of the cluster, with the same timeframe (give or 
take).

Ill look into your suggestions., thank you for your guidance.

From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: 21 September 2017 12:48
To: user@cassandra.apache.org
Subject: Re: Understanding Messages in the Debug.log

Could also be a network hiccup, an NTP problem, a disk pause, or anything that 
causes a long gc pause including misconfigured heap. Are they all on the same 
machine? Or all instances in the cluster?

If there are so many you need to root cause it, you probably want to look for 
correlation with other entries in the debug log, or start inspecting the 
heap/thread pools with some combination of tpstats and dumping the heap and 
looking at the running threads may give you an indication of what’s going on.

--
Jeff Jirsa


On Sep 21, 2017, at 4:37 AM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:
Thanks Jacob,

I’ve already seen this, but it doesn’t answer my question, as this is a 
production Cluster, which at present, has no data and is not being written too, 
its being built to go live shortly, so with no data, a Keyspace with a table 
designed but no data in it, only system databases, what on earth could be 
causing load on the cluster to cause the error message??



From: Jacob Shadix [mailto:jacobsha...@gmail.com]
Sent: 21 September 2017 12:34
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Understanding Messages in the Debug.log

check out this thread - 
https://www.mail-archive.com/user@cassandra.apache.org/msg52313.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mail-2Darchive.com_user-40cassandra.apache.org_msg52313.html&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=T5vuzfn8ElJgnkkZ4sQTkQUMwN-_GEOacqRbeVayQUQ&e=>

-- Jacob Shadix

On Thu, Sep 21, 2017 at 4:56 AM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:
Good Morning Community,

Can anyone help me understand what these messages (and I have a lot of them in 
the log) mean, they must be a symptom of something but I am unsure how to 
starting working this out.

I’m using C* 3.0.11, CQL 5.0.1.

I have 6 node ring over 2 DC’s (3 in each DC) each of the nodes has this 
message in the log.

The message in the debug log is
DEBUG [GossipStage:1] 2017-09-21 09:19:52,627 FailureDetector.java:456 - 
Ignoring interval time of 2000275419


There are so many of these messages, I’d like to find the root cause of this, 
if anyone could help that would be great. Or if they can be ignored, how do I 
supress these?

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street 
Avenue<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=8MElkv-02VTEMtwP2-AvmBTjsk3O-VMBNd3JtOBzp1o&e=>
  •  London EC2Y 
9DT<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=8MElkv-02VTEMtwP2-AvmBTjsk3O-VMBNd3JtOBzp1o&e=>
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com<mailto:jonathan.bay...@tradeweb.com>

<http://www.tradeweb.com/>   follow us:  
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=qEj7m_M2g7N23-PtTGZmpbyO13Zqu-0qypSZg4i3qlA&e=>
   
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=0LpreILsiIHqijRbJ-hSyC7R1Hq3hMhquXS3xcIDLjo&s=B5FSJ8OU-4_K4gemgZ7FC02TH3tkIGnZ6oSynGFEyNs&e=>
—
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and

Issue with New Production Cluster

[Jonathan Baynes]

Hi Community,

I have a 6 node ring, covering 2 DC's, the ring isn't being used yet and we are 
just in the connectivity and testing phase. So the boxes are NOT under any load.

I've gone to connect to CQLSH this afternoon and I've had this returned:

cqlsh xx.xxx.xxx.xx -u cassandra -p cassandra
Connection error: ('Unable to connect to any servers', { xx.xxx.xxx.xx': 
OperationTimedOut('errors=Timed out creating connection (5 seconds), 
last_host=None',)})

Status of the service is "Running"
Nodetool status is UP for all nodes.
Tested telnet on all ports and they are all up and connecting.
Checked the PID is up for cassandra (it is)

Any idea how I can debug the cause of this further?


Thanks
J



Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

RE: Issue with New Production Cluster

[Jonathan Baynes]

 Handshake @15:28 is about the same time as I just tried it and it working. 
So it has an inability to handshake with the nodes??

Any suggestions would be great, I have to go to the Network team with this, but 
I’m not sure where to start…

From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: 29 September 2017 14:54
To: cassandra
Subject: Re: Issue with New Production Cluster

What version?
If you retry immediately, does it reconnect?
Anything in the logs?

What you describe is atypical - timeouts on queries can (and will) happen 
occasionally under load, but timeout on connect is atypical. Any sign of 
networking issues/slowness/dns problems/etc?


On Fri, Sep 29, 2017 at 6:43 AM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:
Hi Community,

I have a 6 node ring, covering 2 DC’s, the ring isn’t being used yet and we are 
just in the connectivity and testing phase. So the boxes are NOT under any load.

I’ve gone to connect to CQLSH this afternoon and I’ve had this returned:

cqlsh xx.xxx.xxx.xx -u cassandra -p cassandra
Connection error: ('Unable to connect to any servers', { xx.xxx.xxx.xx’: 
OperationTimedOut('errors=Timed out creating connection (5 seconds), 
last_host=None',)})

Status of the service is “Running”
Nodetool status is UP for all nodes.
Tested telnet on all ports and they are all up and connecting.
Checked the PID is up for cassandra (it is)

Any idea how I can debug the cause of this further?


Thanks
J



Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street 
Avenue<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=>
  •  London EC2Y 
9DT<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=>
P +44 (0)20 77760988  •  F +44 (0)20 7776 
3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com<mailto:jonathan.bay...@tradeweb.com>

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=saD3ZbF7LhTn3IHCHWSzhGp8Ns6SIRO0s0KXCnEbqLU&e=>
[cid:image003.jpg@01CD26AD.4165F110] 
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=Duw_ud1x7HIX3h-0UEtKorCrlKn8C5tUoH6tkrvXDy0&e=>
—
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com<mailto:contac...@tradeweb.com> or by writing to us at 
the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited 
(company number 3912826), 1 Fore Street Avenue London EC2Y 
9DT<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=VSn9NuhaGsP_VCpKoBZFV2hz_46iJ3VhVk5rYBKHeN8&e=>.
 To see our privacy policy, visit our website @ 
www.tradeweb.com<http://www.tradeweb.com>.

RE: node restart causes application latency

[Jonathan Baynes]

I’m following this intensely as the discussion is important to me as well in 
understanding the restarting of a node correctly, currently (3.0.11) I run 
Disable thrift, disable gossip then drain, from reading this If I understand 
correctly is there any point in doing anything other than drain?

From: Mike Torra [mailto:mto...@salesforce.com]
Sent: 13 February 2018 15:10
To: user@cassandra.apache.org
Subject: Re: node restart causes application latency

Then could it be that calling `nodetool drain` after calling `nodetool 
disablegossip` is what causes the problem?

On Mon, Feb 12, 2018 at 6:12 PM, kurt greaves 
mailto:k...@instaclustr.com>> wrote:

​Actually, it's not really clear to me why disablebinary and thrift are 
necessary prior to drain, because they happen in the same order during drain 
anyway. It also really doesn't make sense that disabling gossip after drain 
would make a difference here, because it should be already stopped. This is all 
assuming drain isn't erroring out.




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Tracing cql code being run through the drive

[Jonathan Baynes]

Hi Community,

Can anyone help me understand what class's id need to set logging on , if I 
want to capture the cql commands being run through the driver, similar to how 
profiler (MSSQL) would work? I need to see what's being run, and if the query 
is actually getting to cassandra?

Has anyone had any experience in doing this?

Thanks in advance.

J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0)7884111546
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Node fails to start - error in Log (CassandraDaemon.java:710 - Exception encountered during startup)

[Jonathan Baynes]

Hi Community,

Can anyone give me some pointers where to look, or start to look for where this 
issue is coming from?


On startup I get this in the logs and my service goes into a failed state. 
Active(exited)

ERROR [main] 2018-02-23 09:56:36,267 CassandraDaemon.java:710 - Exception 
encountered during startup
java.lang.ExceptionInInitializerError: null
at org.apache.cassandra.db.Keyspace.(Keyspace.java:301) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:129) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:106) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:887) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:354) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:110) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:179) 
[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) 
[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) 
[apache-cassandra-3.0.11.jar:3.0.11]
Caused by: java.lang.NullPointerException: null
at 
org.apache.cassandra.config.DatabaseDescriptor.getConcurrentViewWriters(DatabaseDescriptor.java:1199)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.db.view.ViewManager.(ViewManager.java:57) 
~[apache-cassandra-3.0.11.jar:3.0.11]
... 9 common frames omitted

Any pointers would be handy as NULL exception is very hard to investigate.

Cassandra 3.0.11, on linux 7.3 (vm)

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0)7884111546
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/>   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 
<https://www.linkedin.com/company/tradeweb?trk=top_nav_home>
[cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.