Re: configure pooling options to avoid BusyPoolException

[Akhil Mehra]

Since queries are asynchronously executed, you will need some mechanism in your 
code to queue your request. Try setting your setMaxQueueSize to meet your need. 
By default its 256

 
http://docs.datastax.com/en/latest-java-driver-api/com/datastax/driver/core/PoolingOptions.html#setMaxQueueSize-int-

If setMaxQueueSize does not do the trick for you then you might need to look at 
throttling your queries.

Regards,
Akhil


> On 24/08/2017, at 6:40 AM, Avi Levi  wrote:
> 
> Hi ,
> I need to execute large amount (millions) of select queries. but I am getting 
> BusyPoolExcption how can I avoid that ? I tried to configure the pooling 
> options but couldn't see that it had any impact 
> Any advice ?
> Failed to execute query SELECT * FROM my_table WHERE id = 'some_uuid' AND x 
> >= 1503501104 AND y < 1503501224;
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) 
> tried for query failed (tried: localhost/0:0:0:0:0:0:0:1:9042 
> (com.datastax.driver.core.exceptions.BusyPoolException: 
> [localhost/0:0:0:0:0:0:0:1] Pool is busy (no available connection and the 
> queue has reached its max size 256)))

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

These guesses will have to do. I thought something was wrong with such old 
SSTables.
Thanks for your help investigating!

On Wednesday, August 23, 2017, 3:09:34 AM PDT, kurt greaves 
 wrote:

Ignore me, I was getting the major compaction for LCS mixed up with STCS. 
Estimated droppable tombstones tends to be fairly accurate. If your SSTables in 
level 2 have that many tombstones I'd say that's definitely the reason L3 isn't 
being compacted.
As for how you got here in the first place, hard to say. Maybe a burst of 
writes over a period a long time ago? Or possibly repairs streaming a lot of 
data? Hard to tell what happened in the past without lots of metrics.​

RE: Getting all unique keys

[Durity, Sean R]

DataStax Enterprise bundles spark and spark connector on the DSE nodes and 
handles much of the plumbing work (and monitoring, etc.). Worth a look.


Sean Durity

From: Avi Levi [mailto:a...@indeni.com]
Sent: Tuesday, August 22, 2017 2:46 AM
To: user@cassandra.apache.org
Subject: Re: Getting all unique keys

Thanks Christophe, we will definitely consider that in the future.

On Mon, Aug 21, 2017 at 3:01 PM, Christophe Schmitz 
> wrote:
Hi Avi,

The spark-project documentation is quite good, as well as the 
spark-cassandra-connector github project, which contains some basic examples 
you can easily get inspired from. A few random advice you might find usefull:
- You will want one spark worker on each node, and a spark master on either one 
of the node, or on a separate node.
- Pay close attention at your port configuration (firewall) as the spark error 
log does not always give you the right hint.
- Pay close attention at your heap size. Make sure to configure your heap size 
such as Cassandra heap size + spark heap size < your node memory (take into 
account Cassandra off heap usage if enabled, OS etc...)
- If your Cassandra data center is used in production, make sure you throttle 
read / write from Spark, pay attention to your latencies, and consider using a 
separate analytic cassandra data center if you get serious with Spark.
- More or less everyone I know find that writing spark jobs in scala is 
natural, while writing them in java is painful :D

Getting spark running will be a bit of an investment at the beginning, but 
overall you will find out it allows you to run queries you can't naturally run 
in Cassandra, like the one you described.

Cheers,

Christophe

On 21 August 2017 at 16:16, Avi Levi > 
wrote:
Thanks Christophe,
we didn't want to add too many moving parts but is sound like a good solution. 
do you have any reference / link that I can look at ?

Cheers
Avi

On Mon, Aug 21, 2017 at 3:43 AM, Christophe Schmitz 
> wrote:
Hi Avi,

Have you thought of using Spark for that work? If you collocate the spark 
workers on each Cassandra nodes, the spark-cassandra connector will split 
automatically the token range for you in such a way that each spark worker only 
hit the Cassandra local node. This will also be done in parallel. Should be 
much faster that way.

Cheers,
Christophe


On 21 August 2017 at 01:34, Avi Levi > 
wrote:
Thank you very much , one question . you wrote that I do not need distinct here 
since it's a part from the primary key. but only the combination is unique 
(PRIMARY KEY (id, timestamp) ) . also if I take the last token and feed it back 
as you showed wouldn't I get overlapping boundaries ?

On Sun, Aug 20, 2017 at 6:18 PM, Eric Stevens 
> wrote:
You should be able to fairly efficiently iterate all the partition keys like:

select id, token(id) from table where token(id) >= -9204925292781066255 limit 
1000;
 id | system.token(id)
+--
...
 0xb90ea1db5c29f2f6d435426dccf77cca6320fac9 | -7821793584824523686

Take the last token you receive and feed it back in, skipping duplicates from 
the previous page (on the unlikely chance that you have two ID's with a token 
collision on the page boundary):

select id, token(id) from table where token(id) >= -7821793584824523686 limit 
1000;
 id | system.token(id)
+-
...
 0xc6289d729c9087fb5a1fe624b0b883ab82a9bffe | -434806781044590339

Continue until you have no more results.  You don't really need distinct here: 
it's part of your primary key, it must already be distinct.

If you want to parallelize it, split the ring into n ranges and include it as 
an upper bound for each segment.

select id, token(id) from table where token(id) >= -9204925292781066255 AND 
token(id) < $rangeUpperBound limit 1000;


On Sun, Aug 20, 2017 at 12:33 AM Avi Levi 
> wrote:
I need to get all unique keys (not the complete primary key, just the partition 
key) in order to aggregate all the relevant records of that key and apply some 
calculations on it.


CREATE TABLE my_table (



id text,



timestamp bigint,



value double,



PRIMARY KEY (id, timestamp) )

I know that to query like this

SELECT DISTINCT id FROM my_table

is not very efficient but how about the approach presented 

configure pooling options to avoid BusyPoolException

[Avi Levi]

Hi ,
I need to execute large amount (millions) of select queries. but I am
getting BusyPoolExcption how can I avoid that ? I tried to configure the
pooling options but couldn't see that it had any impact
Any advice ?

Failed to execute query SELECT * FROM my_table WHERE id = 'some_uuid'
AND x >= 1503501104 AND y < 1503501224;
com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: localhost/0:0:0:0:0:0:0:1:9042
(com.datastax.driver.core.exceptions.BusyPoolException:
[localhost/0:0:0:0:0:0:0:1] Pool is busy (no available connection and
the queue has reached its max size 256)))

Re: Restarting an existing node hangs

[Jeff Jirsa]

jstack to dump threads, get a heap dump, or turn up your logging (to debug
or trace) until you can figure out what exactly it's doing.



On Wed, Aug 23, 2017 at 11:16 AM, Mark Furlong 
wrote:

> Cassandra doesn’t exit and continues to run with very little CPU usage
> shown in top.
>
>
>
> *Thanks*
>
> *Mark*
>
> *801-705-7115 <(801)%20705-7115> office*
>
>
>
> *From:* Jeff Jirsa [mailto:jji...@gmail.com]
> *Sent:* Wednesday, August 23, 2017 12:07 PM
> *To:* cassandra 
> *Subject:* Re: Restarting an existing node hangs
>
>
>
> Typically if that sstable is damaged you'd see some sort of message. If
> you recently changed bloom filter or index intervals for that table, it may
> be silently rebuilding the other components of that sstable. Does cassandra
> exit or does it just keep churning away?
>
>
>
>
>
>
>
> On Wed, Aug 23, 2017 at 10:20 AM, Mark Furlong 
> wrote:
>
> I had an existing node go down. I don’t know the cause of this. I am
> starting Cassandra and I can see in the log that it starts and then hangs
> on the opening of an sstable. Is there anything I can do to fix the sstable?
>
>
>
> I’m on OSC 2.1.12.
>
>
>
> *Thanks in advance,*
>
> *Mark Furlong*
>
> Sr. Database Administrator
>
> *mfurl...@ancestry.com *
> M: 801-859-7427 <(801)%20859-7427>
>
> O: 801-705-7115 <(801)%20705-7115>
>
> 1300 W Traverse Pkwy
>
> Lehi, UT 84043
>
>
>
>
>
> ​[image: http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]
>
>
>
>
>
>
>

RE: Restarting an existing node hangs

[Mark Furlong]

Cassandra doesn’t exit and continues to run with very little CPU usage shown in 
top.

Thanks
Mark
801-705-7115 office

From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Wednesday, August 23, 2017 12:07 PM
To: cassandra 
Subject: Re: Restarting an existing node hangs

Typically if that sstable is damaged you'd see some sort of message. If you 
recently changed bloom filter or index intervals for that table, it may be 
silently rebuilding the other components of that sstable. Does cassandra exit 
or does it just keep churning away?



On Wed, Aug 23, 2017 at 10:20 AM, Mark Furlong 
> wrote:
I had an existing node go down. I don’t know the cause of this. I am starting 
Cassandra and I can see in the log that it starts and then hangs on the opening 
of an sstable. Is there anything I can do to fix the sstable?

I’m on OSC 2.1.12.

Thanks in advance,
Mark Furlong

Sr. Database Administrator

mfurl...@ancestry.com
M: 801-859-7427
O: 801-705-7115
1300 W Traverse Pkwy
Lehi, UT 84043





​[http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]

Re: Restarting an existing node hangs

[Jeff Jirsa]

See also: https://issues.apache.org/jira/browse/CASSANDRA-11163

On Wed, Aug 23, 2017 at 11:07 AM, Jeff Jirsa  wrote:

> Typically if that sstable is damaged you'd see some sort of message. If
> you recently changed bloom filter or index intervals for that table, it may
> be silently rebuilding the other components of that sstable. Does cassandra
> exit or does it just keep churning away?
>
>
>
> On Wed, Aug 23, 2017 at 10:20 AM, Mark Furlong 
> wrote:
>
>> I had an existing node go down. I don’t know the cause of this. I am
>> starting Cassandra and I can see in the log that it starts and then hangs
>> on the opening of an sstable. Is there anything I can do to fix the sstable?
>>
>>
>>
>> I’m on OSC 2.1.12.
>>
>>
>>
>> *Thanks in advance,*
>>
>> *Mark Furlong*
>>
>> Sr. Database Administrator
>>
>> *mfurl...@ancestry.com *
>> M: 801-859-7427 <(801)%20859-7427>
>>
>> O: 801-705-7115 <(801)%20705-7115>
>>
>> 1300 W Traverse Pkwy
>>
>> Lehi, UT 84043
>>
>>
>>
>>
>>
>> ​[image: http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]
>>
>>
>>
>>
>>
>
>

Re: Restarting an existing node hangs

[Jeff Jirsa]

Typically if that sstable is damaged you'd see some sort of message. If you
recently changed bloom filter or index intervals for that table, it may be
silently rebuilding the other components of that sstable. Does cassandra
exit or does it just keep churning away?



On Wed, Aug 23, 2017 at 10:20 AM, Mark Furlong 
wrote:

> I had an existing node go down. I don’t know the cause of this. I am
> starting Cassandra and I can see in the log that it starts and then hangs
> on the opening of an sstable. Is there anything I can do to fix the sstable?
>
>
>
> I’m on OSC 2.1.12.
>
>
>
> *Thanks in advance,*
>
> *Mark Furlong*
>
> Sr. Database Administrator
>
> *mfurl...@ancestry.com *
> M: 801-859-7427 <(801)%20859-7427>
>
> O: 801-705-7115 <(801)%20705-7115>
>
> 1300 W Traverse Pkwy
>
> Lehi, UT 84043
>
>
>
>
>
> ​[image: http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]
>
>
>
>
>

Restarting an existing node hangs

[Mark Furlong]

I had an existing node go down. I don’t know the cause of this. I am starting 
Cassandra and I can see in the log that it starts and then hangs on the opening 
of an sstable. Is there anything I can do to fix the sstable?

I’m on OSC 2.1.12.

Thanks in advance,
Mark Furlong

Sr. Database Administrator

mfurl...@ancestry.com
M: 801-859-7427
O: 801-705-7115
1300 W Traverse Pkwy
Lehi, UT 84043





​[http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]

Re: Bootstrapping a node fails because of compactions not keeping up

[kurt greaves]

>
> But if it also streams, it means I'd still be under-pressure if I am not
> mistaken. I am under the assumption that the compactions are the by-product
> of streaming too many SStables at the same time, and not because of my
> current write load.
>
Ah yeah I wasn't thinking about the capacity problem, more of the
performance impact from the node being backed up with compactions. If you
haven't already, you should try disable stcs in l0 on the joining node. You
will likely still need to do a lot of compactions, but generally they
should be smaller. The  option is -Dcassandra.disable_stcs_in_l0=true

>  I just noticed you were mentioning L1 tables too. Why would that affect
> the disk footprint?

If you've been doing a lot of STCS in L0, you generally end up with some
large SSTables. These will eventually have to be compacted with L1. Could
also be suffering the problem of streamed SSTables causing large
cross-level compactions in the higher levels as well.
​

Re: Cassandra Setup Question

[Carlos Rolo]

Use networktopologystrategy as replication strategy and make sure you have
dc1: 3 and dc2: 3.

This way you have 3 replicas in each DC.



On 23 Aug 2017 12:53, "Jonathan Baynes" 
wrote:

> Hi Community,
>
>
>
> Quick question regarding Replication Factor.
>
>
>
> In my Production Environment I currently have 6 nodes this will grow to 10
> shortly), over 2 datacentres, so currently 3 in each, we want to have
> Active/Passive setup so the Client will only speak to DC 1 via load
> balancing policy, but we want the data to replicate across both DCs , so
> that in the event of a DC failure the second DC acts as our DR DC and has
>  all the data as well.
>
>
>
> The Consistency will be Quorum, but replication Factor does that need to
> be 3 or 6.
>
>
>
> Thanks
>
> J
>
>
>
> *Jonathan Baynes*
>
> DBA
> Tradeweb Europe Limited
>
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988 <+44%2020%207776%200988>  •  F +44 (0)20 7776 3201
> <+44%2020%207776%203201>  •  M +44 (0) xx
>
> jonathan.bay...@tradeweb.com
>
>
>
> [image: cid:image001.jpg@01CD26AD.4165F110] 
> follow us:  *[image: cid:image002.jpg@01CD26AD.4165F110]*
>    [image:
> cid:image003.jpg@01CD26AD.4165F110] 
>
> —
>
> A leading marketplace  for
> electronic fixed income, derivatives and ETF trading
>
>
>
> 
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy it. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden. Tradeweb reserves the right to monitor all e-mail
> communications through its networks. If you do not wish to receive
> marketing emails about our products / services, please let us know by
> contacting us, either by email at contac...@tradeweb.com or by writing to
> us at the registered office of Tradeweb in the UK, which is: Tradeweb
> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
> 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.
>

-- 


--

Cassandra Setup Question

[Jonathan Baynes]

Hi Community,

Quick question regarding Replication Factor.

In my Production Environment I currently have 6 nodes this will grow to 10 
shortly), over 2 datacentres, so currently 3 in each, we want to have 
Active/Passive setup so the Client will only speak to DC 1 via load balancing 
policy, but we want the data to replicate across both DCs , so that in the 
event of a DC failure the second DC acts as our DR DC and has  all the data as 
well.

The Consistency will be Quorum, but replication Factor does that need to be 3 
or 6.

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 

[cid:image003.jpg@01CD26AD.4165F110] 
-
A leading marketplace for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: Bootstrapping a node fails because of compactions not keeping up

[Stefano Ortolani]

Hi Kurt,

On Wed, Aug 23, 2017 at 11:32 AM, kurt greaves  wrote:

>
> ​1) You mean restarting the node in the middle of the bootstrap with
>> join_ring=false? Would this option require me to issue a nodetool boostrap
>> resume, correct? I didn't know you could instruct the join via JMX. Would
>> it be the same of the nodetool boostrap command?
>
> write_survey is slightly different to join_ring. TBH I haven't used
> write_survey myself for this purpose, but I believe it should work. The
> code certainly indicates that it will bootstrap and stream, but just not
> join the ring until you trigger a joinRing with JMX. (You'll have to look
> up the specific JMX call to make, there is one somewhere...)
>

But if it also streams, it means I'd still be under-pressure if I am not
mistaken. I am under the assumption that the compactions are the by-product
of streaming too many SStables at the same time, and not because of my
current write load.


>
> 2) Yes, they are streamed into L0 as far as I can see (although they are
>> mainly L2/L3 on the existing nodes). I don't think I can do much about it
>> as long as I am using LCS, am I right?
>
>  Code seems to imply that it should always keep SSTable levels. Sounds
> like something is wrong if it is not :/.
>

I don't have the logs anymore (machine had other issues, replacing disks)
so I can't be sure of this. Definitely I remember a high volume of
"Bootstrapping - doing STCS in L0" messages, which alone do not imply
though that all sstables were streamed at L0, so I can't be sure of this. I
just noticed you were mentioning L1 tables too. Why would that affect the
disk footprint?

Thanks!
Stefano

Re: C* 3 node issue -Urgent

[kurt greaves]

Common trap. It's an unfortunate default that is not so easy to change.​

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

@Kurt, You have hit the nail on the head.

Whilst it was an issue with the cassandra-env.sh file, and I fixed this by 
overwriting it with the default file, essentially rolling it back, which got me 
back in, the larger issue is with the system_auth db it is a REPL factor of 1 
and thus as stated in this doc 
http://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/sec/secConfSysAuthKeyspRepl.html

Attention: To prevent a potential problem logging into a secure cluster, set 
the replication factor of the system_auth and dse_security keyspaces to a value 
that is greater than 1. In a multi-node cluster, using the default of 1 
prevents logging into any node when the node that stores the user data is down.

Thank you ALL for coming back to me and assisting, Kurt, good shout and thanks 
for getting me where I need to be, which is back up and running.

Keep up the great work Community!!

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: 23 August 2017 11:14
To: User
Subject: Re: C* 3 node issue -Urgent

The cassandra user requires QUORUM consistency to be achieved for 
authentication. Normal users only require ONE. I suspect your system_auth 
keyspace has an RF of 1, and the node that owns the cassandra users data is 
down.

Steps to recover:
1. Turn off authentication on all the nodes
2. Restart the nodes and make sure they are UN
3. Alter system_auth to have a higher RF than 1 (3 is probably appropriate)
4. Turn auth back on and restart
5. Create a new user and use that from now on.

​



This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: Bootstrapping a node fails because of compactions not keeping up

[kurt greaves]

> ​1) You mean restarting the node in the middle of the bootstrap with
> join_ring=false? Would this option require me to issue a nodetool boostrap
> resume, correct? I didn't know you could instruct the join via JMX. Would
> it be the same of the nodetool boostrap command?

write_survey is slightly different to join_ring. TBH I haven't used
write_survey myself for this purpose, but I believe it should work. The
code certainly indicates that it will bootstrap and stream, but just not
join the ring until you trigger a joinRing with JMX. (You'll have to look
up the specific JMX call to make, there is one somewhere...)

2) Yes, they are streamed into L0 as far as I can see (although they are
> mainly L2/L3 on the existing nodes). I don't think I can do much about it
> as long as I am using LCS, am I right?

 Code seems to imply that it should always keep SSTable levels. Sounds like
something is wrong if it is not :/.

Re: C* 3 node issue -Urgent

[kurt greaves]

The cassandra user requires QUORUM consistency to be achieved for
authentication. Normal users only require ONE. I suspect your system_auth
keyspace has an RF of 1, and the node that owns the cassandra users data is
down.

Steps to recover:
1. Turn off authentication on all the nodes
2. Restart the nodes and make sure they are UN
3. Alter system_auth to have a higher RF than 1 (3 is probably appropriate)
4. Turn auth back on and restart
5. Create a new user and use that from now on.

​

Re: Bootstrapping a node fails because of compactions not keeping up

[Stefano Ortolani]

Hi Kurt,

1) You mean restarting the node in the middle of the bootstrap with
join_ring=false? Would this option require me to issue a nodetool boostrap
resume, correct? I didn't know you could instruct the join via JMX. Would
it be the same of the nodetool boostrap command?
2) Yes, they are streamed into L0 as far as I can see (although they are
mainly L2/L3 on the existing nodes). I don't think I can do much about it
as long as I am using LCS, am I right?

Thanks for the help, much appreciated!

Cheers,
Stefano

On Wed, Aug 23, 2017 at 10:52 AM, kurt greaves  wrote:

> Well, that sucks. Be interested if you could find out if any of the
> streamed SSTables are retaining their levels.
> To answer your questions:
> 1) No.  However, you could set your nodes to join in write_survey mode,
> which will stop them from joining the ring and you can initiate the join
> over JMX when you are ready (once compactions catch up).
> 2) A 1TB compaction might be completely plausible if you are using
> compression and lots of SSTables are being streamed into L0 and triggering
> STCS compactions, or being compacted with L1.
>
> ​
>

Re: Cassandra isn't compacting old files

[kurt greaves]

Ignore me, I was getting the major compaction for LCS mixed up with STCS.
Estimated droppable tombstones tends to be fairly accurate. If your
SSTables in level 2 have that many tombstones I'd say that's definitely the
reason L3 isn't being compacted.

As for how you got here in the first place, hard to say. Maybe a burst of
writes over a period a long time ago? Or possibly repairs streaming a lot
of data? Hard to tell what happened in the past without lots of metrics.
​

Re: Bootstrapping a node fails because of compactions not keeping up

[kurt greaves]

Well, that sucks. Be interested if you could find out if any of the
streamed SSTables are retaining their levels.
To answer your questions:
1) No.  However, you could set your nodes to join in write_survey mode,
which will stop them from joining the ring and you can initiate the join
over JMX when you are ready (once compactions catch up).
2) A 1TB compaction might be completely plausible if you are using
compression and lots of SSTables are being streamed into L0 and triggering
STCS compactions, or being compacted with L1.

​

Re: C* 3 node issue -Urgent

[Akhil Mehra]

I am assuming the following guide or similar was followed to add JMX 
authentication:

http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureJmxAuthentication.html
 




> On 23/08/2017, at 9:14 PM, Jonathan Baynes  
> wrote:
> 
> When trying to connect to CQLSH I get this
>  
> “Cannot achieve consistency level QUORUM”
>  
> 2 of my 3 nodes are down. So this error is correct. But how do I sign into 
> CQLSH to change this? Or better, how do I get the other 2 nodes back up?
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com ] 
> Sent: 23 August 2017 10:05
> To: user@cassandra.apache.org 
> Subject: Re: C* 3 node issue -Urgent
>  
> The cqlsh image say bad credentials. Just confirming that you have the 
> correct username/password when logging on.
> 
> By turing on authentication I am assuming you mean using the 
> PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
> 
> Cheers,
> Akhil
> 
> On 23/08/2017, at 8:59 PM, Jonathan Baynes  > wrote:
>  
> I will also  mention I am on:
>  
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>  
> From: Jonathan Baynes 
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org '
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>  
> Hi Everyone.
>  
> I  need the communities help here.
>  
> I have attempted this morning to turn on JMX authentication for Nodetool. 
> I’ve gone into the Cassandra-env.sh file and updated the following:
>  
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS 
> -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>  
> Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>  
> Nodetool status shows this
>  
> 
>  
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the 
> nodes.
> 
> as it stands right now.. 
>  
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error 
> 
>  
> When I check the logs I on 115.64 and 115.65 I can see this
>  
> WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 
> - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
> consistency level LOCAL_ONE
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
> ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot 
> achieve consistency level LOCAL_ONE
>  
>  
> I have no idea what to do, so any help would be amazing right now..
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
> jonathan.bay...@tradeweb.com 
>  
>     follow us:   
> 
> 
> 

Re: C* 3 node issue -Urgent

[Akhil Mehra]

You could try reverting your JMX authentication changes for the time being if 
getting you nodes up is a priority.

At least you will be able to isolate the problem i.e. is it the Password 
authenticator or the jmx changes causing the problem.

Cheers,
Akhil

PS Sorry for the silly questions just trying to eliminate the obvious.

> On 23/08/2017, at 9:14 PM, Jonathan Baynes  
> wrote:
> 
> When trying to connect to CQLSH I get this
>  
> “Cannot achieve consistency level QUORUM”
>  
> 2 of my 3 nodes are down. So this error is correct. But how do I sign into 
> CQLSH to change this? Or better, how do I get the other 2 nodes back up?
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com ] 
> Sent: 23 August 2017 10:05
> To: user@cassandra.apache.org 
> Subject: Re: C* 3 node issue -Urgent
>  
> The cqlsh image say bad credentials. Just confirming that you have the 
> correct username/password when logging on.
> 
> By turing on authentication I am assuming you mean using the 
> PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
> 
> Cheers,
> Akhil
> 
> On 23/08/2017, at 8:59 PM, Jonathan Baynes  > wrote:
>  
> I will also  mention I am on:
>  
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>  
> From: Jonathan Baynes 
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org '
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>  
> Hi Everyone.
>  
> I  need the communities help here.
>  
> I have attempted this morning to turn on JMX authentication for Nodetool. 
> I’ve gone into the Cassandra-env.sh file and updated the following:
>  
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS 
> -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>  
> Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>  
> Nodetool status shows this
>  
> 
>  
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the 
> nodes.
> 
> as it stands right now.. 
>  
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error 
> 
>  
> When I check the logs I on 115.64 and 115.65 I can see this
>  
> WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 
> - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
> consistency level LOCAL_ONE
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
> ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot 
> achieve consistency level LOCAL_ONE
>  
>  
> I have no idea what to do, so any help would be amazing right now..
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
> jonathan.bay...@tradeweb.com 
>  
>     follow us:   
> 
> 
> 

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

When trying to connect to CQLSH I get this

“Cannot achieve consistency level QUORUM”

2 of my 3 nodes are down. So this error is correct. But how do I sign into 
CQLSH to change this? Or better, how do I get the other 2 nodes back up?

From: Akhil Mehra [mailto:akhilme...@gmail.com]
Sent: 23 August 2017 10:05
To: user@cassandra.apache.org
Subject: Re: C* 3 node issue -Urgent

The cqlsh image say bad credentials. Just confirming that you have the correct 
username/password when logging on.

By turing on authentication I am assuming you mean using the 
PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.

Cheers,
Akhil
On 23/08/2017, at 8:59 PM, Jonathan Baynes 
> wrote:

I will also  mention I am on:

C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7

From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I’ve 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this



In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error


When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com

   follow us:  

   

—
A leading marketplace for electronic 
fixed income, derivatives and ETF trading


This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves 

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

Yes I have the correct credentials I’m using cassandra/cassandra (superuser)
To test that theory I tried a different user and got this

Connection error: ('Unable to connect to any servers', {'10.172.115.63': 
AuthenticationFailed('Failed to authenticate to 10.172.115.63: Error from 
server: code=0100 [Bad credentials] message="Username and/or password are 
incorrect"',)})

Yes you are correct I do  have PasswordAuthenticator turned on .

From: Akhil Mehra [mailto:akhilme...@gmail.com]
Sent: 23 August 2017 10:05
To: user@cassandra.apache.org
Subject: Re: C* 3 node issue -Urgent

The cqlsh image say bad credentials. Just confirming that you have the correct 
username/password when logging on.

By turing on authentication I am assuming you mean using the 
PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.

Cheers,
Akhil
On 23/08/2017, at 8:59 PM, Jonathan Baynes 
> wrote:

I will also  mention I am on:

C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7

From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I’ve 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this



In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error


When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com

   follow us:  

   

—
A leading marketplace for electronic 
fixed income, derivatives and ETF trading


This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received 

Re: C* 3 node issue -Urgent

[Akhil Mehra]

The cqlsh image say bad credentials. Just confirming that you have the correct 
username/password when logging on.

By turing on authentication I am assuming you mean using the 
PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.

Cheers,
Akhil

> On 23/08/2017, at 8:59 PM, Jonathan Baynes  
> wrote:
> 
> I will also  mention I am on:
>  
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>  
> From: Jonathan Baynes 
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org '
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>  
> Hi Everyone.
>  
> I  need the communities help here.
>  
> I have attempted this morning to turn on JMX authentication for Nodetool. 
> I’ve gone into the Cassandra-env.sh file and updated the following:
>  
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS 
> -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>  
> Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>  
> Nodetool status shows this
>  
> 
>  
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the 
> nodes.
> 
> as it stands right now.. 
>  
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error 
> 
>  
> When I check the logs I on 115.64 and 115.65 I can see this
>  
> WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 
> - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
> consistency level LOCAL_ONE
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
> ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot 
> achieve consistency level LOCAL_ONE
>  
>  
> I have no idea what to do, so any help would be amazing right now..
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
> jonathan.bay...@tradeweb.com 
>  
>     follow us:   
> 
> 
> —
> A leading marketplace  for 
> electronic fixed income, derivatives and ETF trading
>  
> 
> 
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have received this e-mail in error) please 
> notify the sender immediately and destroy it. Any unauthorized copying, 
> disclosure or distribution of the material in this e-mail is strictly 
> forbidden. Tradeweb reserves the right to monitor all e-mail communications 
> through its networks. If you do not wish to receive marketing emails about 
> our products / services, please let us know by contacting us, either by email 
> at contac...@tradeweb.com  or by writing to us 
> at the registered office of Tradeweb in the UK, which is: Tradeweb Europe 
> Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To 
> see our privacy policy, visit our website @ www.tradeweb.com 
> .
> 

RE: C* 3 node issue -Urgent

[Jonathan Baynes]

I will also  mention I am on:

C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7

From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I've 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this

[cid:image004.png@01D31BF6.93C48130]

In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error
[cid:image005.png@01D31BF6.93C48130]

When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 

[cid:image003.jpg@01CD26AD.4165F110] 
-
A leading marketplace for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

C* 3 node issue -Urgent

[Jonathan Baynes]

Hi Everyone.

I  need the communities help here.

I have attempted this morning to turn on JMX authentication for Nodetool. I've 
gone into the Cassandra-env.sh file and updated the following:

LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS 
-Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"

Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)

Nodetool status shows this

[cid:image004.png@01D31BF4.B7FB3460]

In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.

as it stands right now..

115.63 is up, but I cannot connect to CQLSH
I get the following error
[cid:image005.png@01D31BF4.B7FB3460]

When I check the logs I on 115.64 and 115.65 I can see this

WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - 
CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - 
Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at 
org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
 ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
~[apache-cassandra-3.0.11.jar:3.0.11]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
consistency level LOCAL_ONE


I have no idea what to do, so any help would be amazing right now..

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 

[cid:image003.jpg@01CD26AD.4165F110] 
-
A leading marketplace for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: Bootstrapping a node fails because of compactions not keeping up

[Stefano Ortolani]

Hi Kurt,
sorry, I forgot to specify. I am on 3.0.14.

Cheers,
Stefano

On Wed, Aug 23, 2017 at 12:11 AM, kurt greaves  wrote:

> What version are you running? 2.2 has an improvement that will retain
> levels when streaming and this shouldn't really happen. If you're on 2.1
> best bet is to upgrade
>