You could try reverting your JMX authentication changes for the time being if getting you nodes up is a priority.
At least you will be able to isolate the problem i.e. is it the Password authenticator or the jmx changes causing the problem. Cheers, Akhil PS Sorry for the silly questions just trying to eliminate the obvious. > On 23/08/2017, at 9:14 PM, Jonathan Baynes <[email protected]> > wrote: > > When trying to connect to CQLSH I get this > > “Cannot achieve consistency level QUORUM” > > 2 of my 3 nodes are down. So this error is correct. But how do I sign into > CQLSH to change this? Or better, how do I get the other 2 nodes back up? > > From: Akhil Mehra [mailto:[email protected] <mailto:[email protected]>] > Sent: 23 August 2017 10:05 > To: [email protected] <mailto:[email protected]> > Subject: Re: C* 3 node issue -Urgent > > The cqlsh image say bad credentials. Just confirming that you have the > correct username/password when logging on. > > By turing on authentication I am assuming you mean using the > PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml. > > Cheers, > Akhil > > On 23/08/2017, at 8:59 PM, Jonathan Baynes <[email protected] > <mailto:[email protected]>> wrote: > > I will also mention I am on: > > C* 3.0.11 > Linux Oracle red hat 7.1 > Java 1.8.0.31 > Python 2.7 > > From: Jonathan Baynes > Sent: 23 August 2017 09:47 > To: '[email protected] <mailto:[email protected]>' > Cc: Stewart Allman > Subject: C* 3 node issue -Urgent > > Hi Everyone. > > I need the communities help here. > > I have attempted this morning to turn on JMX authentication for Nodetool. > I’ve gone into the Cassandra-env.sh file and updated the following: > > LOCAL_JMX=No > JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true" > JVM_OPTS="$JVM_OPTS > -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password" > > Then I restarted each node, I have 3 of them in a single DC, Single Rack. > 115.63 (seed) > 115.64 > 115.65 (seed) > > Nodetool status shows this > > <image004.png> > > In the Yaml I have Authentication turned ON. > Cassandra has been working flawlessly up until the moment I restarted the > nodes. > > as it stands right now.. > > 115.63 is up, but I cannot connect to CQLSH > I get the following error > <image005.png> > > When I check the logs I on 115.64 and 115.65 I can see this > > WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 > - CassandraRoleManager skipped default role setup: some nodes were not ready > INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 > - Setup task failed with error, rescheduling > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level LOCAL_ONE > at > org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) > ~[apache-cassandra-3.0.11.jar:3.0.11] > at > org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) > ~[apache-cassandra-3.0.11.jar:3.0.11] > at > org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) > ~[apache-cassandra-3.0.11.jar:3.0.11] > at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) > ~[apache-cassandra-3.0.11.jar:3.0.11] > at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) > ~[apache-cassandra-3.0.11.jar:3.0.11] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > ... 38 common frames omitted > Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot > achieve consistency level LOCAL_ONE > > > I have no idea what to do, so any help would be amazing right now.. > > Thanks > J > > Jonathan Baynes > DBA > Tradeweb Europe Limited > Moor Place • 1 Fore Street Avenue • London EC2Y 9DT > P +44 (0)20 77760988 • F +44 (0)20 7776 3201 • M +44 (0)xxxx xxxxxx > [email protected] <mailto:[email protected]> > > <image006.jpg> <http://www.tradeweb.com/> follow us: <image007.jpg> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=h5M2yR_HjCB_ghUa2rzYYKcwH5VjWUPf_ttbCNUrdqA&e=> > <image008.jpg> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=yzqb445p2cV3fJfSIGPOcGNEkP7zIfxMWNo3jSw7TmM&e=> > ————————————————————————————————————————— > A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for > electronic fixed income, derivatives and ETF trading > > ________________________________________________________________________ > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) please > notify the sender immediately and destroy it. Any unauthorized copying, > disclosure or distribution of the material in this e-mail is strictly > forbidden. Tradeweb reserves the right to monitor all e-mail communications > through its networks. If you do not wish to receive marketing emails about > our products / services, please let us know by contacting us, either by email > at [email protected] <mailto:[email protected]> or by writing to us > at the registered office of Tradeweb in the UK, which is: Tradeweb Europe > Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To > see our privacy policy, visit our website @ www.tradeweb.com > <http://www.tradeweb.com/>. > > ________________________________________________________________________ > > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) please > notify the sender immediately and destroy it. Any unauthorized copying, > disclosure or distribution of the material in this e-mail is strictly > forbidden. Tradeweb reserves the right to monitor all e-mail communications > through its networks. If you do not wish to receive marketing emails about > our products / services, please let us know by contacting us, either by email > at [email protected] <mailto:[email protected]> or by writing to us > at the registered office of Tradeweb in the UK, which is: Tradeweb Europe > Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To > see our privacy policy, visit our website @ www.tradeweb.com > <http://www.tradeweb.com/>. >
