Re: Cassandra on Kubernetes

[Akshit Jain]

Hi Jean
Thanks for replying. I had seen CassKop and the amount functionality it
provides is quite awesome as compared to other operators.

I would like to know how stable is kubernetes for stateful/database
applications right now?

I haven't read/heard any major production stateful application running on
k8s.


-Akshit




On Wed, 30 Oct, 2019, 8:12 PM Jean-Armel Luce,  wrote:

> Hi,
>
> We are currently developping CassKop, a Cassandra operator for K8S.
> This operator is developped in Go, based on the operator-sdk framework.
>
> At this time of the project, the goal is to deploy a Cassandra cluster in
> 1 Kubernetes datacenter, but this will change in next versions to deal with
> Kubernetes in multi-datacenters.
>
> The following features are already supported by CassKop:
> - Deployment of a C* cluster (rack or AZ aware)
> - Scaling up the cluster (with cleanup)
> - Scaling down the cluster (with decommission prior to Kubernetes scale
> down)
> - Pods operations (removenode, upgradesstable, cleanup, rebuild..)
> - Adding a Cassandra DC
> - Removing a Cassandra DC
> - Setting and modifying configuration files
> - Setting and modifying configuration parameters
> - Update of the Cassandra docker image
> - Rolling update of a Cassandra cluster
> - Update of Cassandra version (including upgradesstable in case of major
> upgrade)
> - Update of JVM
> - Update of configuration
> - Stopping a Kubernetes node for maintenance
> - Process a remove node (and create new Cassandra node on another
> Kubernetes node)
> - Process a replace address (of the old Cassandra node on another
> Kubernetes node)
> - Manage operations on pods through CassKop plugin (cleanup, rebuild,
> upgradesstable, removenode..)
> - Monitoring (using Instaclustr Prometheus exporter to Prometheus/Grafana)
> - Pause/Restart & rolling restart operations through CassKoP plugin.
>
> We use also Cassandra reaper for scheduling repair sessions.
>
>
> If you would like more informations about this operator, you may have a
> look here : https://github.com/Orange-OpenSource/cassandra-k8s-operator
>
> Please, feel free to download it and try it. We would be more than happy
> to receive your feedback
>
>
> If you have any question about this operator, feel free to contact us via
> our mailing-list: prj.casskop.supp...@list.orangeportails.net or on our
> slack https://casskop.slack.com
>
> Note : this operator is still in alpha version and works only in a mono
> region architecture for now. We are currently working hard for adding new
> features in order to run it in multi-regions architecture.
>
>
> Thanks.
>
>
>
> Le mer. 30 oct. 2019 à 13:56, Akshit Jain  a
> écrit :
>
>> Hi everyone,
>>
>> Is there anyone who is running Cassandra on K8s clusters. It would be
>> great if you can share your experience , the operator you are using and the
>> overall stability of stateful sets in Kubernetes
>>
>> -Akshit
>>
>

Cassandra on Kubernetes

[Akshit Jain]

Hi everyone,

Is there anyone who is running Cassandra on K8s clusters. It would be great
if you can share your experience , the operator you are using and the
overall stability of stateful sets in Kubernetes

-Akshit

Does repair handles indexes

[Akshit Jain]

Hi,
Does repair in cassandra handles secondary indexes repair/rebuild?
Has someone observed this/ is there any documentation related to it.

Regards

Re: Frequency of rebuild_index

[Akshit Jain]

Hi,
This looks good but when can the discrepancy in the index arise. Any
specific example?
Is there any documentation which says the index automatically
rebuilds/keeps itself up to date after updations and deletions. Also if
there anything specific to stratio-lucene-index.

Regards
Akshit Jain
9891724697

On Fri, Apr 27, 2018 at 9:59 AM, Anup Shirolkar <
anup.shirol...@instaclustr.com> wrote:

> Hi,
>
> The secondary indices in Cassandra are maintained continuously as data is
> written. Also index rebuilding is kicked off automatically when you create
> a new index. So, there is no good reason to schedule nodetool rebuild_index
> regularly.
>
> However, if you find any discrepancy in the index and data you should run
> it. Ideally, this should not happen but if it is required as a result of
> any major activity/failure you can use it.
>
> Talking about the load it puts on system, it depends upon the size of
> index itself. Although it will consume resources, it should not give a
> major performance hit to the system.
>
> Regards,
> Anup
>
> On 27 April 2018 at 13:46, Akshit Jain <akshit13...@iiitd.ac.in> wrote:
>
>> Hi,
>> How frequently one should run nodetool rebuild_index and what's its
>> impact on performance in terms of iops,cpu utilisation etc.
>>
>> Regards
>>
>>
>

Frequency of rebuild_index

[Akshit Jain]

Hi,
How frequently one should run nodetool rebuild_index and what's its impact
on performance in terms of iops,cpu utilisation etc.

Regards

Token range redistribution

[Akshit Jain]

Hi,
If i replace a node does it redistributes the token range or when the node
again joins will it be allocated a new token range.

Use case:
I have booted a C* on AWS. I terminated a node and then boot a new node
assigned it the same ip and made it join the cluster.

In this case would the token range be redistributed and the node will get
the new token range.
Would the process be different for seed nodes?

Regards
Akshit Jain

Re: Repair giving error

[Akshit Jain]

Hi alain
Thanks for the response.
I'm using cassandra 3.10
nodetool status  shows all the nodes up
No schema disaggrement
port 7000 is open

Regards
Akshit Jain
9891724697

On Thu, Jan 18, 2018 at 4:53 PM, Alain RODRIGUEZ <arodr...@gmail.com> wrote:

> Hello,
>
> I looks like a communication issue.
>
> What Cassandra version are you using?
> What's the result of 'nodetool status '?
> Any schema disagreement 'nodetool describecluster'?
> Is the port 7000 opened and the nodes communicating with each other?(Ping
> is not proving connection is up, even though it is good to know the machine
> is there and up :)).
> Any other errors you could see in the logs?
>
> You might want to consider this an open source project my coworkers have
> been working on (and are maintaining) called reaper that aims at making
> repairs more efficient and easy to manage as repair is one of the most
> tricky operation to handle for a Cassandra operator: http://cassandra-
> reaper.io/. I did not work on this project directly but we have good
> feedbacks and like this tool ourselves.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>
>
> 2018-01-14 7:47 GMT+00:00 Akshit Jain <akshit13...@iiitd.ac.in>:
>
>> ​I have a 10 node C* cluster with 4-5 keyspaces​.
>> I tried to perform nodetool repair one by one for each keyspace.
>> For some keyspaces the repair passed but for some it gave this error:
>> ​
>> I am not able to figure out what is causing this issue.The replica nodes
>> are up and I am able to ping them from this node.​
>> ​Any suggestions?​
>>
>> *Error I am getting on incremental repair:*
>>
>> *[2018-01-10 12:50:14,047] Did not get positive replies from all
>> endpoints. List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *-- StackTrace --java.lang.RuntimeException: Repair job has failed with
>> the error message: [2018-01-10 12:50:14,047] Did not get positive replies
>> from all endpoints. List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]at
>> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)at
>> org.apache.cassandra.utils.pro
>> <http://org.apache.cassandra.utils.pro>gress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)at
>> com.sun.jmx.remote.internal.Cl
>> <http://com.sun.jmx.remote.internal.Cl>ientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)at
>> com.sun.jmx.remote.internal.Cl
>> <http://com.sun.jmx.remote.internal.Cl>ientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)at
>> com.sun.jmx.remote.internal.Cl
>> <http://com.sun.jmx.remote.internal.Cl>ientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)at
>> com.sun.jmx.remote.internal.Cl
>> <http://com.sun.jmx.remote.internal.Cl>ientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)*
>>
>
>

Repair giving error

[Akshit Jain]

​I have a 10 node C* cluster with 4-5 keyspaces​.
I tried to perform nodetool repair one by one for each keyspace.
For some keyspaces the repair passed but for some it gave this error:
​
I am not able to figure out what is causing this issue.The replica nodes
are up and I am able to ping them from this node.​
​Any suggestions?​

*Error I am getting on incremental repair:*

*[2018-01-10 12:50:14,047] Did not get positive replies from all endpoints.
List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]*









*-- StackTrace --java.lang.RuntimeException: Repair job has failed with the
error message: [2018-01-10 12:50:14,047] Did not get positive replies from
all endpoints. List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]at
org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)*

Error while performing repair: "Did not get positive replies from all endpoints"

[Akshit Jain]

​I have a 10 node C* cluster with 4-5 keyspaces​.
I tried to perform nodetool repair one by one for each keyspace.
For some keyspaces the repair passed but for some it gave this error:
​
I am not able to figure out what is causing this issue.The replica nodes
are up and I am able to ping them from this node.​
​Any suggestions?​









*[2018-01-10 12:50:08,162] Starting repair command #4539, repairing
keyspace ​ with repair options (parallelism: parallel,
primary range: false, incremental: true, job threads: 1, ColumnFamilies:
[], dataCenters: [], hosts: [], # of ranges: 2243)[2018-01-10 12:50:14,047]
Did not get positive replies from all endpoints. List of failed
endpoint(s): [​a.b.c.d, ​e.f.g.h][2018-01-10 12:50:14,047] Repair command
#4539 finished with errorerror: Repair job has failed with the error
message: [2018-01-10 12:50:14,047] Did not get positive replies from all
endpoints. List of failed endpoint(s): *










*[​a.b.c.d, ​e.f.g.h]-- StackTrace --java.lang.RuntimeException: Repair job
has failed with the error message: [2018-01-10 12:50:14,047] Did not get
positive replies from all endpoints. List of failed
endpoint(s): [​a.b.c.d, ​e.f.g.h]at
org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)*

Right way to stress a cassandra cluster

[Akshit Jain]

Hi,
Is it right to load test the cassandra cluster using stress tool with
default keyspace, keyspace1 and having columns type as blob?

Stress test cassandr

[Akshit Jain]

Hi,
What is the best way to stress test the cassandra cluster with real life
workloads which is being followed currently?
Currently i am using cassandra stress-tool but it generated blob data /yaml
files provides the option to use custom keyspace.

But what are the different parameters values which can be set to test the
cassandra cluster in extreme environment?

Backup and Restore in cassandra

[Akshit Jain]

What is the correct process to backup and restore in cassandra?
Should we do backup node by node like first schema backup from all the
nodes then all other stuff?
In restore the schema should be restored on one node or all the nodes
again?It will give Already Exists Exception but still what's the correct
process which is followed in production?

Full repair use case

[Akshit Jain]

Is there any use case where we need full repair and incremental repair will
not help?
Actually i am performing incremental repair regularly is there any need to
run full repair?

Difference in ops/s for cassandra stress tool

[Akshit Jain]

hi,
Would there be any difference in the performance of the stress tool if the
keyspaces being populated are of type :-
One is of default keyspace1 standard1 having columns of blob type
Second one is of mixed type (string,int,text)etc.

Please tell in terms of ops/sec.

Stratio Lucene Index

[Akshit Jain]

Hi,
Does using lucene index affects the performance of the read/write in
cassandra.
How heavy is the operation of creating a lucene index on a table with some
GB's of data?

Re: Node failure

[Akshit Jain]

You replace it with a new node and bootstraping happens.The new node
receives data from other two nodes.
Rest depends on the scenerio u are asking for.

Regards
Akshit Jain
B-Tech,2013124
9891724697


On Fri, Oct 6, 2017 at 10:50 PM, Mark Furlong <mfurl...@ancestry.com> wrote:

> What happens when I have a 3 node cluster with RF 3 and a node fails that
> needs to be removed?
>
>
>
> *Mark Furlong*
>
> Sr. Database Administrator
>
> *mfurl...@ancestry.com <mfurl...@ancestry.com>*
> M: 801-859-7427
>
> O: 801-705-7115
>
> 1300 W Traverse Pkwy
> <https://maps.google.com/?q=1300+W+Traverse+Pkwy%0D+Lehi,+UT+84043=gmail=g>
>
> Lehi, UT 84043
> <https://maps.google.com/?q=1300+W+Traverse+Pkwy%0D+Lehi,+UT+84043=gmail=g>
>
>
>
>
>
> ​[image: http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]
>
>
>
>
>

Re: Nodetool Authentication

[Akshit Jain]

In this tutorial :
https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureNodetool.html
In note section it is written "In Cassandra 3.0.8 and later, a user
designated readonly access can run nodetool info so that cluster monitoring
is available. In earlier versions, the user must have readwritepermission."
So when i give a read only access to a user in jmxremote.access: ,they can
still run all the nodetool commands.Why is this so?

Regards
Akshit Jain
B-Tech,2013124
9891724697


On Fri, Oct 6, 2017 at 5:14 PM, Horia Mocioi <horia.moc...@ericsson.com>
wrote:

> Which exactly "read access" are you refering to? Can you point out in the
> tutorial?
>
> On fre, 2017-10-06 at 14:03 +0530, Akshit Jain wrote:
>
> Hi,
> For nodetool authentication I'm following this:
>  https://support.datastax.com/hc/en-us/articles/204226179-
> Step-by-step-instructions-for-securing-JMX-authentication-
> for-nodetool-utility-OpsCenter-and-JConsole
> <https://support.datastax.com/hc/en-us/articles/204226179-Step-by-step-instructions-for-securing-JMX-authentication-for-nodetool-utility-OpsCenter-and-JConsole>
>
> I want to know the read access to the user in nodetool authentication
> gives access to which all features available in nodetool?
>
>

Nodetool Authentication

[Akshit Jain]

Hi,
For nodetool authentication I'm following this:
 
https://support.datastax.com/hc/en-us/articles/204226179-Step-by-step-instructions-for-securing-JMX-authentication-for-nodetool-utility-OpsCenter-and-JConsole


I want to know the read access to the user in nodetool authentication gives
access to which all features available in nodetool?

Manual Compaction in cassandra

[Akshit Jain]

If Size Tiered Compaction Strategy is used, then on performing manual
compaction would there be always *1 sstable* at last irrespective of the
size?

Also would there be 2 sstables if repair has been performed?

Compaction task not available in dcos-cassandra-service

[Akshit Jain]

Hi, there isn't a compaction task feature in
mesosphere/dcos-cassandra-service like repair and cleanup.
Is anybody working on it or is there any plan to add in later releases?
Regards

Compaction in cassandra

[Akshit Jain]

Is it helpful to run nodetool compaction in cassandra?
or automatic compaction is just fine.
Regards

Rebalance a cassandra cluster

[Akshit Jain]

Suppose I have a cassandra cluster with the data that is skewed such that
one node have 40% more data than other nodes.Since while creating the
cassandra the tokens were distributed uniformly.
Now to make the data uniform I have to recalculate the tokens and assign
them to nodes in the cluster. Then run repair and cleanup.
The question is How to recalculate the tokens and assign them to
nodes(Keeping cost ,distance between nodes and data movement in mind).

Re: Rebalance a cassandra cluster

[Akshit Jain]

Suppose I have a cassandra cluster with the data that is skewed such that
one node have 40% more data than other nodes.Since while creating the
cassandra the tokens were distributed uniformly.
Now to make the data uniform I have to recalculate the tokens and assign
them to nodes in the cluster. Then run repair and cleanup.
The question is How to recalculate the tokens and assign them to
nodes(Keeping cost ,distance between nodes and data movement in mind)

Regards
Akshit Jain
B-Tech,2013124
9891724697


On Wed, Sep 13, 2017 at 11:54 AM, Hannu Kröger <hkro...@gmail.com> wrote:

> Hi,
>
> you should make sure that token range is evenly distributed if you have a
> single token configured per node. You can use e.g. this tool to calculate
> tokens:
> https://www.geroba.com/cassandra/cassandra-token-calculator/
>
> Also, make sure that none of the partitions in your data model are
> hotspots that contain a lot more data than on average. Check also
> materialized views if you use them.
>
> Also, due to way the compactions work, it’s normal that the disk usage
> goes up and down. Since nodes often do that in different rhythms, you
> always see that some node(s) are using more disk space than others if some
> point of time especially if you do updates and not just inserts.
>
> Cheers,
> Hannu
>
> On 13 September 2017 at 07:47:09, Akshit Jain (akshit13...@iiitd.ac.in)
> wrote:
>
> Hi,
> Can a cassandra cluster be unbalanced in terms of data?
> If yes then how to rebalance a cassandra cluster.
>
>

Rebalance a cassandra cluster

[Akshit Jain]

Hi,
Can a cassandra cluster be unbalanced in terms of data?
If yes then how to rebalance a cassandra cluster.

Stop writes to a cassandra node for maintenance purpose & enable them after some time

[Akshit Jain]

Hi,
I want to stop writes to a cassandra node for some time and then start
them.Is there any way to do that?
Regards

Stress tool command

[Akshit Jain]

Hi,
Is there any way to set the *gc_grace_seconds* parameter in the stress tool
command?

Regards

How to check if repair is actually successful

[Akshit Jain]

Hi,
I am performing repair on cassandra cluster.
After getting repair status as successful, How to figure out if it is
successful actually?
Is there any way to test it?

Test repair command

[Akshit Jain]

​Hi everyone,
I'm new to cassandra.I was checking the nodetool repair command. I ran the
command and got success but I am not able to figure out how to check repair
has actually happened or not?
It would be a great help if somebody can suggest a way to do that in terms
of data check etc.