Re: Check out new features in K8ssandra and Mission Control

[Christopher Bradford]

Hey Jon,

* What aspects of Mission Control are dependent on using K8ssandra?
>

Mission Control bundles in K8ssandra for the core automation workflows
(lifecycle management, cluster operations, medusa &. reaper). In fact we
include the K8ssandraSpec in the top-level MissionControlCluster resource
verbatim.

 * Can Mission Control work without K8ssandra?


Not at this time, K8ssandra powers a significant portion of the C* side of
the stack. Mission Control provides additional functionality (web
interface, certificate coordination, observability stack, etc) and
applies some conventions to how K8ssandra objects are created / templated
out, but the actually K8ssandra operator present in MC is the same one
available via the Helm charts.

* Is mission control open source?
>

Not at this time. While the majority of the Kubernetes operators are open
source as part of K8ssandra, there are some pieces which are closed source.
I expect some of the components may move from closed source into K8ssandra
over time.

* I'm not familiar with Vector - does it require an agent?


Vector <https://vector.dev/> is a pretty neat project. We run a few of
their components as part of the stack. There is a DaemonSet which runs on
each worker to collect host level metrics and scrape logs being emitted by
containers, a sidecar for collecting logs from the C* container, and an
aggregator which performs some filtering and transformation before pushing
to an object store.

* Is Reaper deployed separately or integrated in?
>

Reaper is deployed as part of the cluster creation workflow. It is spun up
and configured to connect to the cluster automatically.

~Chris

Christopher Bradford



On Tue, Feb 27, 2024 at 6:55 PM Jon Haddad  wrote:

> Hey Chris - this looks pretty interesting!  It looks like there's a lot of
> functionality in here.
>
> * What aspects of Mission Control are dependent on using K8ssandra?
> * Can Mission Control work without K8ssandra?
> * Is mission control open source?
> * I'm not familiar with Vector - does it require an agent?
> * Is Reaper deployed separately or integrated in?
>
> Thanks!  Looking forward to trying this out.
> Jon
>
>
> On Tue, Feb 27, 2024 at 7:07 AM Christopher Bradford 
> wrote:
>
>> Hey C* folks,
>>
>> I'm excited to share that the DataStax team has just released Mission
>> Control <https://datastax.com/products/mission-control>, a new
>> operations platform for running Apache Cassandra and DataStax Enterprise.
>> Built around the open source core of K8ssandra <https://k8ssandra.io/>
>> we've been hard at work expanding multi-region capabilities. If you haven't
>> seen some of the new features coming in here are some highlights:
>>
>>
>>-
>>
>>Management API support in Reaper - no more JMX credentials, YAY
>>-
>>
>>Additional support for TLS across the stack- including operator to
>>node, Reaper to management API, etc
>>-
>>
>>Updated metrics pipeline - removal of collectd from nodes, Vector for
>>monitoring log files (goodbye tail -f)
>>-
>>
>>Deterministic node selection for cluster operations
>>-
>>
>>Top-level management tasks in the control plane (no more forced
>>connections to data planes to trigger a restart)
>>
>>
>> On top of this Mission Control offers:
>>
>>
>>-
>>
>>A single web-interface to monitor and manage your clusters wherever
>>they're deployed
>>-
>>
>>Automatic management of internode and operator to node certificates -
>>this includes integration with third party CAs and rotation of all
>>certificates, keys, and various Java stores
>>-
>>
>>Centralized metrics and logs aggregation, querying and storage with
>>the capability to split the pipeline allowing for exporting of streams to
>>other observability tools within your environment
>>-
>>
>>Per-node configuration (this is an edge case, but still something we
>>wanted to make possible)
>>
>>
>> While building our Mission Control, K8ssandra has seen a number of
>> releases with quite a few contributions from the community. From Helm chart
>> updates to operator tweaks we want to send out a huge THANK YOU to everyone
>> who has filed issues, opened pull requests, and helped us test bugfixes and
>> new functionality.
>>
>> If you've been sleeping on K8ssandra, now is a good time to check it out
>> <https://k8ssandra.io/get-started/>. It has all of the pieces needed to
>> run Cassandra in production. Looking for something out of the box instead
>> of putting the pieces together yourself, take Mission Control for a spin
>> and sign up for the trial
>> <https://datastax.com/products/mission-control/download>. I'm happy to
>> answer any K8ssandra or Mission Control questions you may have here or on
>> our Discord <https://discord.gg/qP5tAt6Uwt>.
>>
>> Cheers,
>>
>> ~Chris
>>
>> Christopher Bradford
>>
>>

Check out new features in K8ssandra and Mission Control

[Christopher Bradford]

Hey C* folks,

I'm excited to share that the DataStax team has just released Mission
Control <https://datastax.com/products/mission-control>, a new operations
platform for running Apache Cassandra and DataStax Enterprise. Built around
the open source core of K8ssandra <https://k8ssandra.io/> we've been hard
at work expanding multi-region capabilities. If you haven't seen some of
the new features coming in here are some highlights:


   -

   Management API support in Reaper - no more JMX credentials, YAY
   -

   Additional support for TLS across the stack- including operator to node,
   Reaper to management API, etc
   -

   Updated metrics pipeline - removal of collectd from nodes, Vector for
   monitoring log files (goodbye tail -f)
   -

   Deterministic node selection for cluster operations
   -

   Top-level management tasks in the control plane (no more forced
   connections to data planes to trigger a restart)


On top of this Mission Control offers:


   -

   A single web-interface to monitor and manage your clusters wherever
   they're deployed
   -

   Automatic management of internode and operator to node certificates -
   this includes integration with third party CAs and rotation of all
   certificates, keys, and various Java stores
   -

   Centralized metrics and logs aggregation, querying and storage with the
   capability to split the pipeline allowing for exporting of streams to other
   observability tools within your environment
   -

   Per-node configuration (this is an edge case, but still something we
   wanted to make possible)


While building our Mission Control, K8ssandra has seen a number of releases
with quite a few contributions from the community. From Helm chart updates
to operator tweaks we want to send out a huge THANK YOU to everyone who has
filed issues, opened pull requests, and helped us test bugfixes and new
functionality.

If you've been sleeping on K8ssandra, now is a good time to check it out
<https://k8ssandra.io/get-started/>. It has all of the pieces needed to run
Cassandra in production. Looking for something out of the box instead of
putting the pieces together yourself, take Mission Control for a spin and sign
up for the trial <https://datastax.com/products/mission-control/download>.
I'm happy to answer any K8ssandra or Mission Control questions you may have
here or on our Discord <https://discord.gg/qP5tAt6Uwt>.

Cheers,

~Chris

Christopher Bradford

Re: Reducing no. of nodes to 0 without losing data

[Christopher Bradford]

This isn’t really a scaling operation. It’s a start / stop operation. In my
mind scaling the cluster would change the tokens / replicas. Here the
request seems to be all or nothing.

On a side note scale down operations were merged 3 days ago in PR #265 (
https://github.com/datastax/cass-operator/pull/265) although there has yet
to be a release cut with this functionality.

On Thu, Oct 8, 2020 at 12:46 AM Erick Ramirez 
wrote:

> The cass-operator does not support scaling down at this point, only
> scaling up. So the operation you're after isn't possible. Cheers!
>
>> --

Christopher Bradford

Re: Reducing no. of nodes to 0 without losing data

[Christopher Bradford]

cass-operator has a parameter spec.stopped (
https://github.com/datastax/cass-operator/blob/master/operator/example-cassdc-yaml/cassandra-3.11.x/example-cassdc-full.yaml#L65)
when set to true the underlying stateful sets are scaled down to 0 while
keeping the persistent volumes. This allows you to reduce your k8s cluster
size while not losing data. When you want to bring the cluster back up
change the parameter back to false.

On Thu, Oct 8, 2020 at 12:40 AM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Thu, Oct 8, 2020 at 12:21 AM Manu Chadha 
> wrote:
>
>> Hi
>>
>>
>>
>> I have created a Cassandra cluster on Kubernetes using cass-operator on
>> gcp. It is for my personal experimentation. To avoid incurring cost, I
>> want to stop the cluster when I am not using it and start it when I need it
>> without losing data. Is there a way to do so? Would setting number of
>> size to 0 in example-cassdc-minimal.yaml stop the compute resources
>> without losing data? If I change the size to 3 again later, would the
>> existing data be picked?
>>
>
> Depending if the operator is going to accept the size as 0 at all, but
> most probably not with the following policy in your storage class, as in
> the example[1]:
>
> reclaimPolicy: Delete
>
> You need some persistent storage and a suitable reclaim policy.
>
> [1]:
> https://docs.datastax.com/en/cass-operator/doc/cass-operator/cassOperatorCloserLookConfiguration.html#CreateandapplyaStorageClass
>
>
>>
> Regards,
> --
> Alex
>
-- 

Christopher Bradford

Re: Cassandra on K8S

[Christopher Bradford]

In *most* k8s environments each Kubernetes worker receives its own
dedicated CIDR range from the cluster’s CIDR space for allocating pod IP
addresses. The issue described can occur when a k8s worker goes down then
comes back up and the pods are rescheduled where either pod starts up with
another pods previously used IP.

They don’t necessarily have to swap 1:1 (ie one pod could use the other’s
previous address while that pod receives a new address). Additionally it’s
not a race condition of which container starts first. The k8s scheduler and
kubelet daemon assign IPs to pods.

On Mon, Aug 3, 2020 at 11:14 PM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> I have started reading about how to deploy Cassandra with K8S. But as I
>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>> Some of the challenges which I feel are
>>
>> 1. POD IPs identification - If the pods go down and when they come up
>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>> nodes for internode as well client server communication.
>>
>
> Strictly safe to change an IP to an IP that is unused.
> Strictly unsafe to use an IP that's already in the cluster (so if two pods
> go down, having the first pod that comes up grab the IP of the second pod
> is strictly dangerous and will violate consistency and maybe lose data).
>
> *For point 2 (Strictly unsafe to use an IP) , if the first pod grabs the
> IP of the second node then it should not be able to join the cluster.*
> *So can IPs still be swapped?  *
> *When and how this IP swap can occur?*
>
> Regards
> Manish
>
> On Mon, Jul 6, 2020 at 10:40 PM Jeff Jirsa  wrote:
>
>>
>>
>> On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal <
>> manishkhandelwa...@gmail.com> wrote:
>>
>>> I have started reading about how to deploy Cassandra with K8S. But as I
>>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>>> Some of the challenges which I feel are
>>>
>>> 1. POD IPs identification - If the pods go down and when they come up
>>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>>> nodes for internode as well client server communication.
>>>
>>
>> Strictly safe to change an IP to an IP that is unused.
>> Strictly unsafe to use an IP that's already in the cluster (so if two
>> pods go down, having the first pod that comes up grab the IP of the second
>> pod is strictly dangerous and will violate consistency and maybe lose
>> data).
>>
>>
>>>
>>> 2. A K8S node can host  a single pod. This is being done so that even if
>>> the host goes down we have only one pod down case. With multiple pods on a
>>> single host there is a risk of traffic failures as consistency might not be
>>> achieved. But if we keep two pods of the same rack on a single host then
>>> are we safe or is there  any unknown risk?
>>>
>>
>> This sounds like you're asking if rack aware snitches protect you from
>> concurrent pods going down. Make sure you're using a rack aware snitch.
>>
>>
>>>
>>> 3. Seed discovery? Again as an extension of point 1, since IPs can
>>> change, how we can manage seeds.
>>>
>>
>> Either use DNS instead of static IPs, or use a seed provider that handles
>> IPs changing.
>>
>>
>>>
>>> 4. Also I read a lot of use of Cassandra operators for maintaining a
>>> Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a
>>> robot (automated admin) which works and acts like a norma admin will work.
>>> I want to understand that how important is Cassandra operator and what if
>>> we go on to production without one?
>>>
>>> Regards
>>> Manish
>>>
>> --

Christopher Bradford

Re: Cassandra container, Google Cloud and Kubernetes

[Christopher Bradford]

Hi Manu,

OSS Cassandra support in cass-operator is marked as a Technology Preview as
there is no integrated solution for repairs or backup / restore
functionality at this time. If you are comfortable managing these
operational tasks either manually or through other complementary tools
(Reaper and Medusa come to mind) then there should not be anything blocking
you from using this operator. As Erick mentioned there are other operators
available that may or may not handle these tasks for you and should be
considered.

~Chris

Christopher Bradford



On Thu, Jun 25, 2020 at 2:39 AM Manu Chadha  wrote:

> Thanks. One more concern popped up. It seems Cass is not recommended for
> production. However, I need this specifically for production. What is your
> take on this?
>
>
>
> “
>
> The use of Cass Operator with Cassandra 3.11.6 is intended as a *Technology
> Preview* only. Using Cass Operator with Cassandra is not recommended at
> this time for production environments.
>
> “
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Erick Ramirez 
> *Sent: *25 June 2020 07:25
> *To: *user@cassandra.apache.org
> *Subject: *Re: Cassandra container, Google Cloud and Kubernetes
>
>
>
> It seems that 3.11.4 is not supported. I am happy to move up to 3.11.6 but
> is 3.11.6 backward compatible with 3.11.4? I don’t want start changing my
> driver code in the application.
>
>
>
> There isn't a breaking change from a driver perspective between 3.11.4 and
> 3.11.6 so you don't need to rewrite your code. Cheers!
>
>
>

Re: Multi DC replication between different Cassandra versions

[Christopher Bradford]

It’s worth noting there can be issues with streaming between different
versions of C*. Note this excerpt from
https://thelastpickle.com/blog/2019/02/26/data-center-switch.html

Note that with an upgrade it’s important to keep in mind that *streaming in
a cluster running mixed versions of Casandra is not recommended*

Emphasis mine. With the approach you’re suggesting streaming would be
involved both during bootstrap and repair. Would it be possible to upgrade
to a more recent release prior to pursuing this course of action?

On Thu, Apr 16, 2020 at 1:02 AM Erick Ramirez 
wrote:

> I don't mean any disrespect but let me offer you a friendly advice --
> don't do it to yourself. I think you would have a very hard time finding
> someone who would recommend implementing a solution that involves mixed
> versions. If you run into issues, it would be hell trying to unscramble
> that egg.
>
> On top of that, Cassandra 3.0.9 is an ancient version released 4 years ago
> (September 2016). There are several pages of fixes deployed since then. So
> in the nicest possible way, what you're planning to do is not a good idea.
> I personally wouldn't do it. Cheers!
>
-- 

Christopher Bradford

Re: How to Protect Tracing Requests From Client Side

[Christopher Bradford]

`nodetool settraceeprobability` controls the *automated* tracing within a
single node based on the value set. It may be some or none, but it doesn't
effect queries which are explicitly marked for tracing by the driver within
your application. You can test this by running CQLSH and enabling TRACING
with "TRACING ON;". Now run a query and trace results will be returned.

To resolve your problem it may be worth checking to see if you could limit
the ability to modify tables within the system_traces keyspace via
permissions.

~Chris

On Thu, Mar 22, 2018 at 6:07 PM shalom sagges 
wrote:

> Thanks a lot Rahul! :-)
>
> On Thu, Mar 22, 2018 at 8:03 PM, Rahul Singh  > wrote:
>
>> Execute ‘nodetool settraceprobability 0’ on all nodes. It does zero
>> percentage of he tracing.
>>
>> --
>> Rahul Singh
>> rahul.si...@anant.us
>>
>> Anant Corporation
>>
>> On Mar 22, 2018, 11:10 AM -0500, shalom sagges ,
>> wrote:
>>
>> Hi All,
>>
>> Is there a way to protect C* on the server side from tracing commands
>> that are executed from clients?
>>
>> Thanks!
>>
>>
>

Re: nodetool repair uses option '-local' and '-pr' togather

[Christopher Bradford]

If each AZ has a different rack identifier and the keyspace uses
NetworkTopologyStrategy with a replication factor of 3 then the single host
in us-east-1d *will receive 100% of the data*. This is due
to NetworkTopologyStrategy's preference for placing replicas across
different racks before placing a second replica in a rack where data
already resides. Check it out with CCM:

> ccm node1 status

Datacenter: us-east-1
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens   Owns (effective)  Host ID
Rack
UN  127.0.0.1  98.31 KiB  140.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
UN  127.0.0.2  98.31 KiB  140.0%
30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
UN  127.0.0.3  98.3 KiB   140.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
UN  127.0.0.4  98.31 KiB  140.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
UN  127.0.0.5  98.31 KiB  140.0%
2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d

cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class':
'NetworkTopologyStrategy', 'us-east-1': 3};

> ccm node1 status replication_test
Datacenter: us-east-1
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens   Owns (effective)  Host ID
Rack
UN  127.0.0.1  88.38 KiB  180.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
UN  127.0.0.2  98.31 KiB  120.0%
30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
UN  127.0.0.3  98.3 KiB   180.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
UN  127.0.0.4  98.31 KiB  120.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
UN  127.0.0.5  98.31 KiB  1100.0%
 2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d

This can be tested further with a simple table and nodetool getendpoints.

> ccm node1 nodetool getendpoints replication_test sample bar

127.0.0.2
127.0.0.3
127.0.0.5

> ccm node1 nodetool getendpoints replication_test sample baz

127.0.0.1
127.0.0.3
127.0.0.5

> ccm node1 nodetool getendpoints replication_test sample bif

127.0.0.3
127.0.0.5
127.0.0.1

> ccm node1 nodetool getendpoints replication_test sample biz

127.0.0.2
127.0.0.3
127.0.0.5

On Fri, Sep 2, 2016 at 9:41 AM Paulo Motta  wrote:

> If I understand the way replication is done, the node in us-east-1d has
> all the (data) replicas, right?
>
> No, for this to be correct, you'd need to have one DC per AZ, which is not
> this case since you have a single DC encompassing multiple AZs. Right now,
> replicas will be spread in 3 distinct AZs, which are represented as racks
> in the single NTS DC if you are using EC2*Snitch. So your best bet is
> probably to run repair -pr in all nodes.
>
>
> 2016-09-01 14:28 GMT-03:00 Li, Guangxing :
>
>> Thanks for the info, Paulo.
>>
>> My cluster is in AWS, the keyspace has replication factor 3 with
>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
>> us-east-1c and 1 in us-east-1d. If I understand the way replication is
>> done, the node in us-east-1d has all the (data) replicas, right? If so, if
>> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
>> on the node in us-east-1d? In other words, does 'nodetool repair' started
>> on node in us-east-1d also cause repairs on replicas on other nodes? I am
>> seeing different answers in discussion like this
>> http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node
>> .
>>
>> Thanks again.
>>
>> George
>>
>> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
>> wrote:
>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-7450
>>>
>>> 2016-09-01 13:11 GMT-03:00 Li, Guangxing :
>>>
 Hi,

 I have a cluster running 2.0.9 with 2 data centers. I noticed that
 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
 node's data size is 39 GB and the largest SSTable size is like 7 GB so the
 column family is not huge, SizeTieredCompactionStrategy is used). Repairing
 a column family on a single node takes over 5 hours. So I am wondering if I
 can use option '-local' and '-pr' together, hoping to get some speed up.
 But according to documentation at
 https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsRepair.html
 '...Do not use -pr with this option to repair only a local data
 center...'. Can someone tell me the reason why we should not use options
 '-local' and '-pr' together?

 Thanks.

 George

>>>
>>>
>>
>

Re: High Bloom filter false ratio

[Christopher Bradford]

Does every record in the SSTable have a "d" column?

On Mon, Feb 22, 2016 at 2:14 AM Anishek Agarwal  wrote:

> Hey guys,
>
> Just did some more digging ... looks like DTCS is not removing old data
> completely, I used sstable2json for one such table and saw old data there.
> we have a value of 30 for  max_stable_age_days for the table.
>
> One of the columns showed data as :["2015-12-10 11\\:03+0530:",
> "56690ea2", 1449725602552000, "d"] what is the meaning of "d" in the last
> IS_MARKED_FOR_DELETE column ?
>
> I see data from 10 dec 2015 still there, looks like there are a few issues
> with DTCS, Operationally what choices do i have to rectify this, We are on
> version 2.0.15.
>
> thanks
> anishek
>
>
>
>
> On Mon, Feb 22, 2016 at 10:23 AM, Anishek Agarwal 
> wrote:
>
>> We are using DTCS have a 30 day window for them before they are cleaned
>> up. I don't think with DTCS we can do anything about table sizing. Please
>> do let me know if there are other ideas.
>>
>> On Sat, Feb 20, 2016 at 12:51 AM, Jaydeep Chovatia <
>> chovatia.jayd...@gmail.com> wrote:
>>
>>> To me following three looks on higher side:
>>> SSTable count: 1289
>>>
>>> In order to reduce SSTable count see if you are compacting of not (If
>>> using STCS). Is it possible to change this to LCS?
>>>
>>>
>>> Number of keys (estimate): 345137664 (345M partition keys)
>>>
>>> I don't have any suggestion about reducing this unless you partition
>>> your data.
>>>
>>>
>>> Bloom filter space used, bytes: 493777336 (400MB is huge)
>>>
>>> If number of keys are reduced then this will automatically reduce bloom
>>> filter size I believe.
>>>
>>>
>>>
>>> Jaydeep
>>>
>>> On Thu, Feb 18, 2016 at 7:52 PM, Anishek Agarwal 
>>> wrote:
>>>
 Hey all,

 @Jaydeep here is the cfstats output from one node.

 Read Count: 1721134722

 Read Latency: 0.04268825050756254 ms.

 Write Count: 56743880

 Write Latency: 0.014650376727851532 ms.

 Pending Tasks: 0

 Table: user_stay_points

 SSTable count: 1289

 Space used (live), bytes: 122141272262

 Space used (total), bytes: 224227850870

 Off heap memory used (total), bytes: 653827528

 SSTable Compression Ratio: 0.4959736121441446

 Number of keys (estimate): 345137664

 Memtable cell count: 339034

 Memtable data size, bytes: 106558314

 Memtable switch count: 3266

 Local read count: 1721134803

 Local read latency: 0.048 ms

 Local write count: 56743898

 Local write latency: 0.018 ms

 Pending tasks: 0

 Bloom filter false positives: 40664437

 Bloom filter false ratio: 0.69058

 Bloom filter space used, bytes: 493777336

 Bloom filter off heap memory used, bytes: 493767024

 Index summary off heap memory used, bytes: 91677192

 Compression metadata off heap memory used, bytes: 68383312

 Compacted partition minimum bytes: 104

 Compacted partition maximum bytes: 1629722

 Compacted partition mean bytes: 1773

 Average live cells per slice (last five minutes): 0.0

 Average tombstones per slice (last five minutes): 0.0


 @Tyler Hobbs

 we are using cassandra 2.0.15 so
 https://issues.apache.org/jira/browse/CASSANDRA-8525  shouldnt occur.
 Other problems looks like will be fixed in 3.0 .. we will mostly try and
 slot in an upgrade to 3.x version towards second quarter of this year.


 @Daemon

 Latencies seem to have higher ratios, attached is the graph.


 I am mostly trying to look at Bloom filters, because the way we do
 reads, we read data with non existent partition keys and it seems to be
 taking long to respond, like for 720 queries it takes 2 seconds, with all
 721 queries not returning anything. the 720 queries are done in
 sequence of 180 queries each with 180 of them running in parallel.


 thanks

 anishek



 On Fri, Feb 19, 2016 at 3:09 AM, Jaydeep Chovatia <
 chovatia.jayd...@gmail.com> wrote:

> How many partition keys exists for the table which shows this problem
> (or provide nodetool cfstats for that table)?
>
> On Thu, Feb 18, 2016 at 11:38 AM, daemeon reiydelle <
> daeme...@gmail.com> wrote:
>
>> The bloom filter buckets the values in a small number of buckets. I
>> have been surprised by how many cases I see with large cardinality where 
>> a
>> few values populate a given bloom leaf, resulting in high false 
>> positives,
>> and a surprising impact on latencies!
>>
>> Are you seeing 2:1 ranges between mean and worse case latencies
>> (allowing for gc times)?
>>
>> Daemeon Reiydelle
>> On Feb 18, 2016 8:57 AM, "Tyler Hobbs"  wrote:
>>
>>> You can try