Re: Cassandra causing OOM Killer to strike on new cluster running 3.4

[Satoshi Hikida]

Hi,

I'm looking forward to a  patch (file) for this bug(CASSANDRA-11344) to
apply C* version 2.2.5. Is there available patch for that version? I
watched link(https://issues.apache.org/jira/browse/CASSANDRA-11344) but
couldn't find patch file or something like that.  Or is there any
workaround to avoid this bug with version 2.2.5?

Regards,
Satoshi

On Tue, Mar 15, 2016 at 4:47 AM, Adam Plumb  wrote:

> OK so good news, I'm running with the patched jar file in my cluster and
> haven't seen any issues.  The bloom filter off-heap memory usage is between
> 1.5GB and 2GB per node, which is much more in-line with what I'm expecting!
>  (thumbsup)
>
> On Mon, Mar 14, 2016 at 9:42 AM, Adam Plumb  wrote:
>
>> Thanks for the link!  Luckily the cluster I'm running is not yet in
>> production and running with dummy data so I will throw that jar on the
>> nodes and I'll let you know how things shake out.
>>
>> On Sun, Mar 13, 2016 at 11:02 PM, Paulo Motta 
>> wrote:
>>
>>> You could be hitting CASSANDRA-11344 (
>>> https://issues.apache.org/jira/browse/CASSANDRA-11344).  If that's the
>>> case, you may try to replace your cassandra jar on an affected node with a
>>> version with this fix in place and force bloom filter regeneration to see
>>> if if it fixes your problem. You can build with "ant jar" from this branch:
>>> https://github.com/pauloricardomg/cassandra/tree/3.4-11344
>>>
>>> You can force bloom filter regeneration by either removing your
>>> *Filter.db files (make sure to backup them before for safety) or changing
>>> the bloom_filter_fp_chance before restarting affected nodes with the fixed
>>> jar.
>>>
>>> 2016-03-13 19:51 GMT-03:00 Adam Plumb :
>>>
 So it's looking like the bloom filter off heap memory usage is ramping
 up and up until the OOM killer kills the java process.  I relaunched on
 instances with 60GB of memory and the same thing is happening.  A node will
 start using more and more RAM until the process is killed, then another
 node will start using more and more until it is also killed.

 Is this the expected behavior?  It doesn't seem ideal to me.  Is there
 anything obvious that I'm doing wrong?

 On Fri, Mar 11, 2016 at 11:31 AM, Adam Plumb  wrote:

> Here is the creation syntax for the entire schema.  The xyz table has
> about 2.1 billion keys and the def table has about 230 million keys.  Max
> row size is about 3KB, mean row size is 700B.
>
> CREATE KEYSPACE abc WITH replication = {'class':
>> 'NetworkTopologyStrategy', 'us-east': 3};
>> CREATE TABLE xyz (
>>   id text,
>>   secondary_id int,
>>   data text,
>>   PRIMARY KEY(id)
>> )
>>   WITH
>>   compaction = { 'class': 'LeveledCompactionStrategy' }
>>   and compression = {'class': 'LZ4Compressor'};
>> CREATE INDEX secondary_id_index ON abc.xyz (secondary_id);
>> CREATE TABLE def (
>>   id text,
>>   secondary_id int,
>>   data text,
>>   PRIMARY KEY(id)
>> )
>>   WITH
>>   compaction = { 'class': 'LeveledCompactionStrategy' }
>>   and compression = {'class': 'LZ4Compressor'};
>> CREATE INDEX secondary_id_index_def ON abc.def (secondary_id);
>
>
> On Fri, Mar 11, 2016 at 11:24 AM, Jack Krupansky <
> jack.krupan...@gmail.com> wrote:
>
>> What is your schema and data like - in particular, how wide are your
>> partitions (number of rows and typical row size)?
>>
>> Maybe you just need (a lot) more heap for rows during the repair
>> process.
>>
>> -- Jack Krupansky
>>
>> On Fri, Mar 11, 2016 at 11:19 AM, Adam Plumb 
>> wrote:
>>
>>> These are brand new boxes only running Cassandra.  Yeah the kernel
>>> is what is killing the JVM, and this does appear to be a memory leak in
>>> Cassandra.  And Cassandra is the only thing running, aside from the 
>>> basic
>>> services needed for Amazon Linux to run.
>>>
>>> On Fri, Mar 11, 2016 at 11:17 AM, Sebastian Estevez <
>>> sebastian.este...@datastax.com> wrote:
>>>
 Sacrifice child in dmesg is your OS killing the process with the
 most ram. That means you're actually running out of memory at the Linux
 level outside of the JVM.

 Are you running anything other than Cassandra on this box?

 If so, does it have a memory leak?

 all the best,

 Sebastián
 On Mar 11, 2016 11:14 AM, "Adam Plumb"  wrote:

> I've got a new cluster of 18 nodes running Cassandra 3.4 that I
> just launched and loaded data into yesterday (roughly 2TB of total 
> storage)
> and am seeing runaway memory usage.  These nodes are EC2 c3.4xlarges 
> with
> 30GB RAM and the heap size is set to 8G with a new 

Cassandra 2.0.x OOM during bootstrap

[Michael Fong]

Hi, all,

We have recently encountered a Cassandra OOM issue when Cassandra is brought up 
sometimes (but not always) in our 4-node cluster test bed.

After analyzing the heap dump, we could find the Internal-Response thread pool 
(JMXEnabledThreadPoolExecutor) is filled with thounds of 
'org.apache.cassandra.net.MessageIn' objects, and occupy > 2 gigabytes of heap 
memory.

According to the documents on internet, it seems internal-response thread pool 
is somewhat related to schema-checking. Has anyone encountered similar issue 
before?

We are using Cassandra 2.0.17 and JDK 1.8. Thanks in advance!

Sincerely,

Michael Fong

Optional TLS CQL Encryption

[Jason J. W. Williams]

Hey Guys,

Is there a way to make TLS encryption optional for the CQL listener? We'd
like to be able to use for remote management connections but not for same
datacenter usage (since the build/up  tear down cost is too high for things
that don't use pools).

Right now it appears if we enable encryption it requires it for all
connections, which definitely is not what we want.

-J

Re: Proper use of COUNT

[Jack Krupansky]

Sylvain & Tyler, this Jira is for a user reporting a timeout for SELECT
COUNT(*) using 3.3:
https://issues.apache.org/jira/browse/CASSANDRA-11566

I'll let one of you guys follow up on that. I mean, I thought it was timing
out die to the amount of data, but you guys are saying that paging should
make that not a problem. Or is there a timeout in cqlsh simply because the
operation is slow - as opposed to the server reporting an internal timeout?

Thanks.



-- Jack Krupansky

On Tue, Apr 19, 2016 at 12:45 PM, Tyler Hobbs  wrote:

>
> On Tue, Apr 19, 2016 at 11:32 AM, Jack Krupansky  > wrote:
>
>>
>> Are the queries sent from the coordinator to other nodes sequencing
>> through partitions in token order and that's what allows the coordinator to
>> dedupe with just a single page at a time? IOW, if a target node responds
>> with a row from token t, then by definition there will be no further rows
>> returned from that node with a token less than t?
>>
>
> That's correct.  The internal paging for aggregation queries is exactly
> the same as the normal "client facing" paging.
>
>
>>
>> And if I understand all of this so far, this means that for 3.x COUNT
>> (and other aggregate functions) are "safe but may be slow" (paraphrasing
>> Sylvain.) Is this for 3.0 and later or some other 3.x (or even some 2.x)?
>>
>
> I think count(*) started using paging internally in 2.1, but I'm having
> trouble finding the jira ticket.  It could have been 2.0.
>
> The new aggregation functions in 2.2 utilize the same code path.
>
>
>>
>> There remains the question of recommended usage for COUNT. I think my two
>> proposed guidelines remain valid (ignoring the old timeout issue), with the
>> only remaining question about how large a row count is advisable for
>> "decent" request latency. 1,000? 10,000? Granted, it depends on the
>> specific data and hardware, but I'm thinking that the guidance should be
>> that you should only use COUNT(*) for no more than "low thousands" of rows
>> unless you are willing to accept it both being very slow and very
>> disruptive to normal cluster health. IOW, it's more like a batch analytics
>> operation than a real-time operation. An occasional administrative query to
>> measure table size should be okay, but common use for OLTP should be
>> restricted to relatively narrow slices or row counts... I think. Feedback
>> welcome.
>>
>> The upcoming support for 2GB partitions will be interesting, but the same
>> guidance should cover, I think. Maybe the numeric upper bound might be a
>> bit higher since only a single partition is involved, but if processing
>> many thousands of rows will remain time consuming, it sounds like that
>> should be treated more as a batch-style OLAP operation rather than a
>> real-time OLTP operation... I think.
>>
>
> I think this is decent guidance.  I'll also clarify that aggregation
> functions should only be used on single partitions if you expect to get a
> response back with reasonable latency.  Full table scans are still
> expensive, even when they're wrapped in an aggregation function.
>
> If count(*) is too slow, the standard alternatives are:
>  - counters
>  - a static count that's periodically refreshed by a batch/background
> process
>  - LWT increments on an int column
>  - an external datastore like redis
>
> Obviously, each of these has a different set of tradeoffs.
>
> --
> Tyler Hobbs
> DataStax 
>

nodetool -h fails Connection refused

[Alaa Zubaidi (PDF)]

Hi,

I am trying to run nodetool remotely. but its not working:
I am running Cassandra 2.2.5 on CentOS 6.
listen_address: is set to 
rpc_address: is set to 0.0.0.0
broadcast_rpc_address: is set to 

I changed the following in cassadnra-env.sh
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname="
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

"nodetool -h  -p  status" results in:
failed to connect to 'hostname' - Connection Exception: 'Connection refused"

netstat -nl | grep 7199
tcp00127.0.0.1:71990.0.0.0:*LISTEN

ONLY "nodetool -h localhost" works

Any idea how to fix it?

Thanks,
Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Discrepancy while paging through table, and static column updated inbetween

[Tyler Hobbs]

This sounds similar to https://issues.apache.org/jira/browse/CASSANDRA-10010,
but that only affected 2.x.  Can you open a Jira ticket with your table
schema, the problematic query, and the details you posted here?

On Tue, Apr 19, 2016 at 10:25 AM, Siddharth Verma <
verma.siddha...@snapdeal.com> wrote:

> Hi,
>
> We are using cassandra(dsc3.0.3) on production.
>
> For some purpose, we were doing a full table scan (setPagingState and
> getPagingState used on ResultSet in java program), and there has been some
> discrepancy when we ran the same job multiple times.
> Each time some new data was added to the output, and some was left out.
>
> Side Note 1 :
> Table structure
> col1, col2, col3, col4, col5, col6
> Primary key(col1, col2)
> col5 is static column
> col6 static column. Used to explicitly store updated time when col5 changed
>
>
> Sample Data
> 1,A,AA,AAA,STATIC,T1
> 1,B,BB,BBB,STATIC,T1
> 1,C,CC,CCC,STATIC,T1
> 1,D,DD,DDD,STATIC,T1
>
> For some key, sometime col6 was updated while the job was running, so some
> values were not printed for that partition key.
>
> Side Note 2 :
> we did -> select col6, writetime(col6) from ... where col1=... and col2=...
> For the data that was missed out to make sure that particular entry wasn't
> added later.
>
>
> Side Note 3:
> The above scenario that some col6 was updated while job was running,
> therefore some entry for that partition key was ignored, is an assumption
> from our end.
> We can't understand why some entries were not printed in the table scan.
>
>


-- 
Tyler Hobbs
DataStax 

Re: Proper use of COUNT

[Jack Krupansky]

Thanks, Tyler.

"Deduping (i.e. normal conflict resolution) happens per-page"

Are the queries sent from the coordinator to other nodes sequencing through
partitions in token order and that's what allows the coordinator to dedupe
with just a single page at a time? IOW, if a target node responds with a
row from token t, then by definition there will be no further rows returned
from that node with a token less than t?

And if I understand all of this so far, this means that for 3.x COUNT (and
other aggregate functions) are "safe but may be slow" (paraphrasing
Sylvain.) Is this for 3.0 and later or some other 3.x (or even some 2.x)?

There remains the question of recommended usage for COUNT. I think my two
proposed guidelines remain valid (ignoring the old timeout issue), with the
only remaining question about how large a row count is advisable for
"decent" request latency. 1,000? 10,000? Granted, it depends on the
specific data and hardware, but I'm thinking that the guidance should be
that you should only use COUNT(*) for no more than "low thousands" of rows
unless you are willing to accept it both being very slow and very
disruptive to normal cluster health. IOW, it's more like a batch analytics
operation than a real-time operation. An occasional administrative query to
measure table size should be okay, but common use for OLTP should be
restricted to relatively narrow slices or row counts... I think. Feedback
welcome.

The upcoming support for 2GB partitions will be interesting, but the same
guidance should cover, I think. Maybe the numeric upper bound might be a
bit higher since only a single partition is involved, but if processing
many thousands of rows will remain time consuming, it sounds like that
should be treated more as a batch-style OLAP operation rather than a
real-time OLTP operation... I think.

Thanks.


-- Jack Krupansky

On Tue, Apr 19, 2016 at 12:04 PM, Tyler Hobbs  wrote:

>
> On Tue, Apr 19, 2016 at 9:51 AM, Jack Krupansky 
> wrote:
>
>>
>> 1. Another clarification: All of the aggregate functions, AVG, SUM, MIN,
>> MAX are in exactly the same boat as COUNT, right?
>>
>
> Yes.
>
>
>>
>> 2. Is the paging for COUNT, et al, done within the coordinator node?
>>
>
> Yes.
>
>
>>
>> 3. Does dedupe on the coordinator node consume memory proportional to the
>> number of rows on all nodes? I mean, you can't dedupe using only partition
>> keys of the coordinator node, right? What I'm wondering is if the usability
>> of COUNT (et al) is memory limited as well as time.
>>
>
> Deduping (i.e. normal conflict resolution) happens per-page, so in the
> worst case the memory requirements for the coordinator are RF * page size.
>
>
>
>
> --
> Tyler Hobbs
> DataStax 
>

Re: Proper use of COUNT

[Tyler Hobbs]

On Tue, Apr 19, 2016 at 9:51 AM, Jack Krupansky 
wrote:

>
> 1. Another clarification: All of the aggregate functions, AVG, SUM, MIN,
> MAX are in exactly the same boat as COUNT, right?
>

Yes.


>
> 2. Is the paging for COUNT, et al, done within the coordinator node?
>

Yes.


>
> 3. Does dedupe on the coordinator node consume memory proportional to the
> number of rows on all nodes? I mean, you can't dedupe using only partition
> keys of the coordinator node, right? What I'm wondering is if the usability
> of COUNT (et al) is memory limited as well as time.
>

Deduping (i.e. normal conflict resolution) happens per-page, so in the
worst case the memory requirements for the coordinator are RF * page size.




-- 
Tyler Hobbs
DataStax 

Re: Traffic inconsistent across nodes

[Eric Stevens]

Glad that worked out, that was going to be my next suspicion, since
everything thought it was up and happy, I can't think of a way that
Cassandra and the driver could both consider the cluster happy if some
nodes were not transmitting at least some data (they have to at least for
gossip).

On Mon, Apr 18, 2016 at 10:02 PM Anishek Agarwal  wrote:

> Looks like some problem with our monitoring framework. Thanks for you help
> !
>
> On Mon, Apr 18, 2016 at 2:46 PM, Anishek Agarwal 
> wrote:
>
>> OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs
>> Cent OS 7)
>> All of them are running Cassandra 2.0.17
>>
>> output of the test :
>>
>> host ip: 10.124.114.113
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.108
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.110
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.118
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.125.138.59
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.97
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.105
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.98
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>>
>> On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens  wrote:
>>
>>> Thanks for that, that helps a lot.  The next thing to check might be
>>> whether or not your application actually has access to the other nodes.
>>> With that topology, and assuming all the nodes you included in your
>>> original graph are in the 'WDC' data center, I'd be inclined to look for a
>>> network issue of some kind.
>>>
>>> Also, it probably doesn't matter, but what OS / Distribution are you
>>> running the servers and clients on?
>>>
>>> Check with netcat or something that you can reach all the configured
>>> ports from your application server, but also the driver itself offers some
>>> introspection into its view of individual connection health.  This is a
>>> little bit ugly, but this is how we include information about connection
>>> status in an API for health monitoring from a Scala application using the
>>> Java driver; hopefully you can use it to see how to access information
>>> about the driver's view of host health from the application's perspective.
>>> Most importantly I'd suggest looking for host.isUp status and
>>> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
>>> in your target datacenter to be LOCAL.
>>>
>>> "hosts" -> {
>>>   val hosts: Map[String, Map[String, mutable.Set[Host]]] =
>>> connection.getMetadata
>>>   .getAllHosts.asScala
>>>   .groupBy(_.getDatacenter)
>>>   .mapValues(_.groupBy(_.getRack))
>>>   val lbp: LoadBalancingPolicy = 
>>> connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>>>   JsObject(hosts.map { case (dc: String, rackAndHosts) =>
>>> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: 
>>> mutable.Set[Host]) =>
>>>   rack -> JsArray(hosts.map { host =>
>>> Json.obj(
>>>   "address"  -> host.getAddress.toString,
>>>   "socketAddress"-> host.getSocketAddress.toString,
>>>   "cassandraVersion" -> host.getCassandraVersion.toString,
>>>   "isUp" -> host.isUp,
>>>   "hostDistance" -> lbp.distance(host).toString
>>> )
>>>   }.toSeq)
>>> }.toSeq)
>>>   }.toSeq)
>>> },
>>>
>>>
>>> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal 
>>> wrote:
>>>
 here is the output:  every node in a single DC is in the same rack.

 Datacenter: WDC5

 

 Status=Up/Down

 |/ State=Normal/Leaving/Joining/Moving

 --  Address Load   Tokens  Owns (effective)  Host ID
 Rack

 UN  10.125.138.33   299.22 GB  256 64.2%
 8aaa6015-d444-4551-a3c5-3257536df476  RAC1

 UN  10.125.138.125  329.38 GB  256 70.3%
 70be44a2-de17-41f1-9d3a-6a0be600eedf  RAC1

 UN  10.125.138.129  305.11 GB  256 65.5%
 0fbc7f44-7062-4996-9eba-2a05ae1a7032  RAC1

 Datacenter: WDC

 ===

 Status=Up/Down

 |/ State=Normal/Leaving/Joining/Moving

 --  Address Load   Tokens  Owns (effective)  Host ID
 Rack

 UN  10.124.114.105  151.09 GB  256 38.0%
 

Discrepancy while paging through table, and static column updated inbetween

[Siddharth Verma]

Hi,

We are using cassandra(dsc3.0.3) on production.

For some purpose, we were doing a full table scan (setPagingState and
getPagingState used on ResultSet in java program), and there has been some
discrepancy when we ran the same job multiple times.
Each time some new data was added to the output, and some was left out.

Side Note 1 :
Table structure
col1, col2, col3, col4, col5, col6
Primary key(col1, col2)
col5 is static column
col6 static column. Used to explicitly store updated time when col5 changed


Sample Data
1,A,AA,AAA,STATIC,T1
1,B,BB,BBB,STATIC,T1
1,C,CC,CCC,STATIC,T1
1,D,DD,DDD,STATIC,T1

For some key, sometime col6 was updated while the job was running, so some
values were not printed for that partition key.

Side Note 2 :
we did -> select col6, writetime(col6) from ... where col1=... and col2=...
For the data that was missed out to make sure that particular entry wasn't
added later.


Side Note 3:
The above scenario that some col6 was updated while job was running,
therefore some entry for that partition key was ignored, is an assumption
from our end.
We can't understand why some entries were not printed in the table scan.

Re: Proper use of COUNT

[Jack Krupansky]

Thanks for that clarification, Sylvain.

1. Another clarification: All of the aggregate functions, AVG, SUM, MIN,
MAX are in exactly the same boat as COUNT, right?

2. Is the paging for COUNT, et al, done within the coordinator node?

3. Does dedupe on the coordinator node consume memory proportional to the
number of rows on all nodes? I mean, you can't dedupe using only partition
keys of the coordinator node, right? What I'm wondering is if the usability
of COUNT (et al) is memory limited as well as time.

Thanks.


-- Jack Krupansky

On Tue, Apr 19, 2016 at 5:36 AM, Sylvain Lebresne 
wrote:

>
>> Accept for relatively small or narrow queries, it seems to have a
>> propensity for timing out.
>>
>
> For recent enough version of C*, it shouldn't since it pages internally
> (it will be slow and as always be, but it shouldn't time out if some decent
> page size is used, which should be the default). I suspect report of it
> timeouting are either using old versions (or are using unreasonable paging
> size values, but that sounds less likely since I'd assume users would
> easily find and fix their error in that case).
>
> But if the query is timeouting unreasonably for large partition in recent
> versions, then it's a bug and a JIRA can be open with reproduction steps.
>
> --
> Sylvain
>
>>
>

Re: Proper use of COUNT

[Sylvain Lebresne]

>
>
> Accept for relatively small or narrow queries, it seems to have a
> propensity for timing out.
>

For recent enough version of C*, it shouldn't since it pages internally (it
will be slow and as always be, but it shouldn't time out if some decent
page size is used, which should be the default). I suspect report of it
timeouting are either using old versions (or are using unreasonable paging
size values, but that sounds less likely since I'd assume users would
easily find and fix their error in that case).

But if the query is timeouting unreasonably for large partition in recent
versions, then it's a bug and a JIRA can be open with reproduction steps.

--
Sylvain

>

Re: Bootstrapping Question for Cassandra 2.0

[Carlos Alonso]

It will be possible after the tokens reassignment has finished.

Actually Cassandra itself will tell you. If you try the remove command and
is not possible it will fail with an error indicating that the operation is
not possible while there are nodes in Joining status.

Regards

Carlos Alonso | Software Engineer | @calonso 

On 18 April 2016 at 20:01, Mir Tanvir Hossain 
wrote:

> Hi, could anyone please tell me whether is it possible to remove an
> existing node from cluster while another node is joining the same cluster?
>
> Thanks,
> Mir
>