Re: Using K8s to Manage Cassandra in Production

[Hassaan Pasha]

 On Wed, May 23, 2018 at 7:35 AM Tom Petracca 
>>>>> wrote:
>>>>>
>>>>>> Using a statefulset should get you pretty far, though will likely be
>>>>>> less effective than a coreos-style “operator”. Some random points:
>>>>>>
>>>>>>- For scale-up: a node shouldn’t report “ready” until it’s in the
>>>>>>NORMAL state; this will prevent multiple nodes from bootstrapping at 
>>>>>> once.
>>>>>>- For scale-down: as of now there isn’t a mechanism to know if a
>>>>>>pod is getting decommissioned because you’ve permanently lowered 
>>>>>> replica
>>>>>>count, or because it’s just getting bounced/re-scheduled, thus knowing
>>>>>>whether or not to decommission is basically impossible. Relevant 
>>>>>> issue:
>>>>>>kubernetes/kubernetes#1462
>>>>>><https://github.com/kubernetes/kubernetes/issues/1462>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From: *Pradeep Chhetri 
>>>>>> *Reply-To: *"user@cassandra.apache.org" 
>>>>>> *Date: *Friday, May 18, 2018 at 10:20 AM
>>>>>> *To: *"user@cassandra.apache.org" 
>>>>>> *Subject: *Re: Using K8s to Manage Cassandra in Production
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hello Hassaan,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We use cassandra helm chart[0] for deploying cassandra over
>>>>>> kubernetes in production. We have around 200GB cas data. It works really
>>>>>> well. You can scale up nodes easily (I haven't tested scaling down).
>>>>>>
>>>>>>
>>>>>>
>>>>>> I would say that if you are worried about running cassandra over k8s
>>>>>> in production, maybe you should first try setting it for your
>>>>>> staging/preproduction and gain confidence over time.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have tested situations where i have killed the host running
>>>>>> cassandra container and have seen that container moves to a different 
>>>>>> node
>>>>>> and joins cluster properly. So from my experience its pretty good. No
>>>>>> issues till yet.
>>>>>>
>>>>>>
>>>>>>
>>>>>> [0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra
>>>>>> [github.com]
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_charts_tree_master_incubator_cassandra=DwMFaQ=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=1oh1YI8i5eJD1DFTwooO7U92fFi2fjan6lqP61yAiyo=dupKDpZi0lkjFkqaSd6XaEj5nuY1T5UObgAcXCBqo7A=0WTYStEM1zvh2BQKvnVLRpukxgr0aDLyGffyE1V2xik=>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Pradeep
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко 
>>>>>> wrote:
>>>>>>
>>>>>> Hi, Hassaan! For example we are using C* in k8s in production for our
>>>>>> video surveillance system. Moreover, we are using Ceph RBD as our storage
>>>>>> for cassandra. Today we have 8 C* nodes each manages 2Tb of data.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha  wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am trying to craft a deployment strategy for deploying and
>>>>>> maintaining a C* cluster. I was wondering if there are actual production
>>>>>> deployments of C* using K8s as the orchestration layer.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have been given the impression that K8s managing a C* cluster can
>>>>>> be a recipe for disaster, especially if you aren't well versed with the
>>>>>> intricacies of a scale-up/down event. I know use cases where people are
>>>>>> using Mesos or a custom tool built with terraform/chef etc to run their
>>>>>> production clusters but have yet to find a real K8s use case.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Questions?*
>>>>>>
>>>>>> Is K8s a reasonable choice for managing a production C* cluster?
>>>>>>
>>>>>> Are there documented use cases for this?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any help would be greatly appreciated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Hassaan Pasha*
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Regrads,
>>>>>>
>>>>>> Pavel Sapezhko
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Ben Bromhead
>>>>> CTO | Instaclustr <https://www.instaclustr.com/>
>>>>> +1 650 284 9692 <(650)%20284-9692>
>>>>> Reliability at Scale
>>>>> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>>>>>
>>>>
>>>> --
>>> Ben Bromhead
>>> CTO | Instaclustr <https://www.instaclustr.com/>
>>> +1 650 284 9692 <(650)%20284-9692>
>>> Reliability at Scale
>>> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>>>
>>
>> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>



-- 
Regards,

*Hassaan Pasha*
Mobile: 03347767442

Re: Using K8s to Manage Cassandra in Production

[Ben Bromhead]

> once.
>>>>>- For scale-down: as of now there isn’t a mechanism to know if a
>>>>>pod is getting decommissioned because you’ve permanently lowered 
>>>>> replica
>>>>>count, or because it’s just getting bounced/re-scheduled, thus knowing
>>>>>whether or not to decommission is basically impossible. Relevant issue:
>>>>>kubernetes/kubernetes#1462
>>>>><https://github.com/kubernetes/kubernetes/issues/1462>
>>>>>
>>>>>
>>>>>
>>>>> *From: *Pradeep Chhetri <prad...@stashaway.com>
>>>>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>>>> *Date: *Friday, May 18, 2018 at 10:20 AM
>>>>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>>>> *Subject: *Re: Using K8s to Manage Cassandra in Production
>>>>>
>>>>>
>>>>>
>>>>> Hello Hassaan,
>>>>>
>>>>>
>>>>>
>>>>> We use cassandra helm chart[0] for deploying cassandra over kubernetes
>>>>> in production. We have around 200GB cas data. It works really well. You 
>>>>> can
>>>>> scale up nodes easily (I haven't tested scaling down).
>>>>>
>>>>>
>>>>>
>>>>> I would say that if you are worried about running cassandra over k8s
>>>>> in production, maybe you should first try setting it for your
>>>>> staging/preproduction and gain confidence over time.
>>>>>
>>>>>
>>>>>
>>>>> I have tested situations where i have killed the host running
>>>>> cassandra container and have seen that container moves to a different node
>>>>> and joins cluster properly. So from my experience its pretty good. No
>>>>> issues till yet.
>>>>>
>>>>>
>>>>>
>>>>> [0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra
>>>>> [github.com]
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_charts_tree_master_incubator_cassandra=DwMFaQ=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=1oh1YI8i5eJD1DFTwooO7U92fFi2fjan6lqP61yAiyo=dupKDpZi0lkjFkqaSd6XaEj5nuY1T5UObgAcXCBqo7A=0WTYStEM1zvh2BQKvnVLRpukxgr0aDLyGffyE1V2xik=>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Pradeep
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко <amelius0...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi, Hassaan! For example we are using C* in k8s in production for our
>>>>> video surveillance system. Moreover, we are using Ceph RBD as our storage
>>>>> for cassandra. Today we have 8 C* nodes each manages 2Tb of data.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha <hpa...@an10.io> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I am trying to craft a deployment strategy for deploying and
>>>>> maintaining a C* cluster. I was wondering if there are actual production
>>>>> deployments of C* using K8s as the orchestration layer.
>>>>>
>>>>>
>>>>>
>>>>> I have been given the impression that K8s managing a C* cluster can be
>>>>> a recipe for disaster, especially if you aren't well versed with the
>>>>> intricacies of a scale-up/down event. I know use cases where people are
>>>>> using Mesos or a custom tool built with terraform/chef etc to run their
>>>>> production clusters but have yet to find a real K8s use case.
>>>>>
>>>>>
>>>>>
>>>>> *Questions?*
>>>>>
>>>>> Is K8s a reasonable choice for managing a production C* cluster?
>>>>>
>>>>> Are there documented use cases for this?
>>>>>
>>>>>
>>>>>
>>>>> Any help would be greatly appreciated.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>> *Hassaan Pasha*
>>>>>
>>>>> --
>>>>>
>>>>> Regrads,
>>>>>
>>>>> Pavel Sapezhko
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Ben Bromhead
>>>> CTO | Instaclustr <https://www.instaclustr.com/>
>>>> +1 650 284 9692 <(650)%20284-9692>
>>>> Reliability at Scale
>>>> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>>>>
>>>
>>> --
>> Ben Bromhead
>> CTO | Instaclustr <https://www.instaclustr.com/>
>> +1 650 284 9692 <(650)%20284-9692>
>> Reliability at Scale
>> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Using K8s to Manage Cassandra in Production

[vincent gromakowski]

Thanks ! Do you have some pointers on the available features ? I am more
afraid of the lack of custom controller integration, for instance the code
generator...

2018-05-23 17:17 GMT+02:00 Ben Bromhead <b...@instaclustr.com>:

> The official Kubernetes Java driver is actually pretty feature complete,
> if not exactly idiomatic Java...  it's only missing full examples to get it
> to GOLD compatibility levels iirc.
>
> A few reasons we went down the Java path:
>
>- Cassandra community engagement was the primary concern. If you are a
>developer in the Cassandra community you have a base level of Java
>knowledge, so it means if you want to work on the Kubernetes operator you
>only have to learn 1 thing, Kubernetes. If the operator was in Go, you
>would then have two things to learn, Go and Kubernetes :)
>- We actually wrote an initial PoC in Go (based off the etcd operator,
>you can find it here https://github.com/benbromhead/cassandra-
>operator-old ), but because it was in Go we ended up making
>architectural decisions simply because Go doesn't do JMX, so it felt like
>we were just fighting different ecosystems just to be part of the cool
>group.
>
> Some other less important points weighed the decision in Java's favour:
>
>- The folk at Instaclustr all know Java, and are productive in it from
>day 1. Go is fun and relatively simple, but not our forte.
>-  Mature package management, Generics/inability to write DRY
>code, a million if err statements  (:
>- Some other awesome operators/controllers are written in JVM based
>languages. The sparkKubernetes resource manager (which is a k8s controller)
>is written in Scala.
>
>
> On Wed, May 23, 2018 at 10:04 AM vincent gromakowski <
> vincent.gromakow...@gmail.com> wrote:
>
>> Why did you choose java for the operator implementation when everybody
>> seems to use the go client (probably for greater functionalities) ?
>>
>> 2018-05-23 15:39 GMT+02:00 Ben Bromhead <b...@instaclustr.com>:
>>
>>> You can get a good way with StatefulSets, but as Tom mentioned there are
>>> still some issues with this, particularly around scaling up and down.
>>>
>>> We are working on an Operator for Apache Cassandra, you can find it here
>>> https://github.com/instaclustr/cassandra-operator. This is a joint
>>> project between Instaclustr, Pivotal and a few other folk.
>>>
>>> Currently it's a work in progress, but we would love any or all early
>>> feedback/PRs/issues etc. Our first GA release will target the following
>>> capabilities:
>>>
>>>- Safe scaling up and down (including decommissioning)
>>>- Backup/restore workflow (snapshots only initially)
>>>- Built in prometheus integration and discovery
>>>
>>> Other features like repair, better PV support, maybe even a nice
>>> dashboard will be on the way.
>>>
>>>
>>> On Wed, May 23, 2018 at 7:35 AM Tom Petracca <tpetra...@palantir.com>
>>> wrote:
>>>
>>>> Using a statefulset should get you pretty far, though will likely be
>>>> less effective than a coreos-style “operator”. Some random points:
>>>>
>>>>- For scale-up: a node shouldn’t report “ready” until it’s in the
>>>>NORMAL state; this will prevent multiple nodes from bootstrapping at 
>>>> once.
>>>>- For scale-down: as of now there isn’t a mechanism to know if a
>>>>pod is getting decommissioned because you’ve permanently lowered replica
>>>>count, or because it’s just getting bounced/re-scheduled, thus knowing
>>>>whether or not to decommission is basically impossible. Relevant issue:
>>>>kubernetes/kubernetes#1462
>>>><https://github.com/kubernetes/kubernetes/issues/1462>
>>>>
>>>>
>>>>
>>>> *From: *Pradeep Chhetri <prad...@stashaway.com>
>>>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>>> *Date: *Friday, May 18, 2018 at 10:20 AM
>>>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>>> *Subject: *Re: Using K8s to Manage Cassandra in Production
>>>>
>>>>
>>>>
>>>> Hello Hassaan,
>>>>
>>>>
>>>>
>>>> We use cassandra helm chart[0] for deploying cassandra over kubernetes
>>>> in production. We have around 200GB cas data. It works really well. You can
>>>> scale up nodes 

Re: Using K8s to Manage Cassandra in Production

[Ben Bromhead]

The official Kubernetes Java driver is actually pretty feature complete, if
not exactly idiomatic Java...  it's only missing full examples to get it to
GOLD compatibility levels iirc.

A few reasons we went down the Java path:

   - Cassandra community engagement was the primary concern. If you are a
   developer in the Cassandra community you have a base level of Java
   knowledge, so it means if you want to work on the Kubernetes operator you
   only have to learn 1 thing, Kubernetes. If the operator was in Go, you
   would then have two things to learn, Go and Kubernetes :)
   - We actually wrote an initial PoC in Go (based off the etcd operator,
   you can find it here
   https://github.com/benbromhead/cassandra-operator-old ), but because it
   was in Go we ended up making architectural decisions simply because
   Go doesn't do JMX, so it felt like we were just fighting different
   ecosystems just to be part of the cool group.

Some other less important points weighed the decision in Java's favour:

   - The folk at Instaclustr all know Java, and are productive in it from
   day 1. Go is fun and relatively simple, but not our forte.
   -  Mature package management, Generics/inability to write DRY
   code, a million if err statements  (:
   - Some other awesome operators/controllers are written in JVM based
   languages. The sparkKubernetes resource manager (which is a k8s controller)
   is written in Scala.


On Wed, May 23, 2018 at 10:04 AM vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> Why did you choose java for the operator implementation when everybody
> seems to use the go client (probably for greater functionalities) ?
>
> 2018-05-23 15:39 GMT+02:00 Ben Bromhead <b...@instaclustr.com>:
>
>> You can get a good way with StatefulSets, but as Tom mentioned there are
>> still some issues with this, particularly around scaling up and down.
>>
>> We are working on an Operator for Apache Cassandra, you can find it here
>> https://github.com/instaclustr/cassandra-operator. This is a joint
>> project between Instaclustr, Pivotal and a few other folk.
>>
>> Currently it's a work in progress, but we would love any or all early
>> feedback/PRs/issues etc. Our first GA release will target the following
>> capabilities:
>>
>>- Safe scaling up and down (including decommissioning)
>>- Backup/restore workflow (snapshots only initially)
>>- Built in prometheus integration and discovery
>>
>> Other features like repair, better PV support, maybe even a nice
>> dashboard will be on the way.
>>
>>
>> On Wed, May 23, 2018 at 7:35 AM Tom Petracca <tpetra...@palantir.com>
>> wrote:
>>
>>> Using a statefulset should get you pretty far, though will likely be
>>> less effective than a coreos-style “operator”. Some random points:
>>>
>>>- For scale-up: a node shouldn’t report “ready” until it’s in the
>>>NORMAL state; this will prevent multiple nodes from bootstrapping at 
>>> once.
>>>- For scale-down: as of now there isn’t a mechanism to know if a pod
>>>is getting decommissioned because you’ve permanently lowered replica 
>>> count,
>>>or because it’s just getting bounced/re-scheduled, thus knowing whether 
>>> or
>>>not to decommission is basically impossible. Relevant issue:
>>>kubernetes/kubernetes#1462
>>><https://github.com/kubernetes/kubernetes/issues/1462>
>>>
>>>
>>>
>>> *From: *Pradeep Chhetri <prad...@stashaway.com>
>>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>> *Date: *Friday, May 18, 2018 at 10:20 AM
>>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>> *Subject: *Re: Using K8s to Manage Cassandra in Production
>>>
>>>
>>>
>>> Hello Hassaan,
>>>
>>>
>>>
>>> We use cassandra helm chart[0] for deploying cassandra over kubernetes
>>> in production. We have around 200GB cas data. It works really well. You can
>>> scale up nodes easily (I haven't tested scaling down).
>>>
>>>
>>>
>>> I would say that if you are worried about running cassandra over k8s in
>>> production, maybe you should first try setting it for your
>>> staging/preproduction and gain confidence over time.
>>>
>>>
>>>
>>> I have tested situations where i have killed the host running cassandra
>>> container and have seen that container moves to a different node and joins
>>> cluster properly. So from my experience its pretty good. No issues till yet.

Re: Using K8s to Manage Cassandra in Production

[vincent gromakowski]

Why did you choose java for the operator implementation when everybody
seems to use the go client (probably for greater functionalities) ?

2018-05-23 15:39 GMT+02:00 Ben Bromhead <b...@instaclustr.com>:

> You can get a good way with StatefulSets, but as Tom mentioned there are
> still some issues with this, particularly around scaling up and down.
>
> We are working on an Operator for Apache Cassandra, you can find it here
> https://github.com/instaclustr/cassandra-operator. This is a joint
> project between Instaclustr, Pivotal and a few other folk.
>
> Currently it's a work in progress, but we would love any or all early
> feedback/PRs/issues etc. Our first GA release will target the following
> capabilities:
>
>- Safe scaling up and down (including decommissioning)
>- Backup/restore workflow (snapshots only initially)
>- Built in prometheus integration and discovery
>
> Other features like repair, better PV support, maybe even a nice dashboard
> will be on the way.
>
>
> On Wed, May 23, 2018 at 7:35 AM Tom Petracca <tpetra...@palantir.com>
> wrote:
>
>> Using a statefulset should get you pretty far, though will likely be less
>> effective than a coreos-style “operator”. Some random points:
>>
>>- For scale-up: a node shouldn’t report “ready” until it’s in the
>>NORMAL state; this will prevent multiple nodes from bootstrapping at once.
>>- For scale-down: as of now there isn’t a mechanism to know if a pod
>>is getting decommissioned because you’ve permanently lowered replica 
>> count,
>>or because it’s just getting bounced/re-scheduled, thus knowing whether or
>>not to decommission is basically impossible. Relevant issue:
>>kubernetes/kubernetes#1462
>><https://github.com/kubernetes/kubernetes/issues/1462>
>>
>>
>>
>> *From: *Pradeep Chhetri <prad...@stashaway.com>
>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Date: *Friday, May 18, 2018 at 10:20 AM
>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Subject: *Re: Using K8s to Manage Cassandra in Production
>>
>>
>>
>> Hello Hassaan,
>>
>>
>>
>> We use cassandra helm chart[0] for deploying cassandra over kubernetes in
>> production. We have around 200GB cas data. It works really well. You can
>> scale up nodes easily (I haven't tested scaling down).
>>
>>
>>
>> I would say that if you are worried about running cassandra over k8s in
>> production, maybe you should first try setting it for your
>> staging/preproduction and gain confidence over time.
>>
>>
>>
>> I have tested situations where i have killed the host running cassandra
>> container and have seen that container moves to a different node and joins
>> cluster properly. So from my experience its pretty good. No issues till yet.
>>
>>
>>
>> [0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra
>> [github.com]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_charts_tree_master_incubator_cassandra=DwMFaQ=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=1oh1YI8i5eJD1DFTwooO7U92fFi2fjan6lqP61yAiyo=dupKDpZi0lkjFkqaSd6XaEj5nuY1T5UObgAcXCBqo7A=0WTYStEM1zvh2BQKvnVLRpukxgr0aDLyGffyE1V2xik=>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Pradeep
>>
>>
>>
>> On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко <amelius0...@gmail.com>
>> wrote:
>>
>> Hi, Hassaan! For example we are using C* in k8s in production for our
>> video surveillance system. Moreover, we are using Ceph RBD as our storage
>> for cassandra. Today we have 8 C* nodes each manages 2Tb of data.
>>
>>
>>
>> On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha <hpa...@an10.io> wrote:
>>
>> Hi,
>>
>>
>>
>> I am trying to craft a deployment strategy for deploying and maintaining
>> a C* cluster. I was wondering if there are actual production deployments of
>> C* using K8s as the orchestration layer.
>>
>>
>>
>> I have been given the impression that K8s managing a C* cluster can be a
>> recipe for disaster, especially if you aren't well versed with the
>> intricacies of a scale-up/down event. I know use cases where people are
>> using Mesos or a custom tool built with terraform/chef etc to run their
>> production clusters but have yet to find a real K8s use case.
>>
>>
>>
>> *Questions?*
>>
>> Is K8s a reasonable choice for managing a production C* cluster?
>>
>> Are there documented use cases for this?
>>
>>
>>
>> Any help would be greatly appreciated.
>>
>>
>>
>> --
>>
>> Regards,
>>
>>
>>
>> *Hassaan Pasha*
>>
>> --
>>
>> Regrads,
>>
>> Pavel Sapezhko
>>
>>
>>
> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>

Re: Using K8s to Manage Cassandra in Production

[Ben Bromhead]

You can get a good way with StatefulSets, but as Tom mentioned there are
still some issues with this, particularly around scaling up and down.

We are working on an Operator for Apache Cassandra, you can find it here
https://github.com/instaclustr/cassandra-operator. This is a joint project
between Instaclustr, Pivotal and a few other folk.

Currently it's a work in progress, but we would love any or all early
feedback/PRs/issues etc. Our first GA release will target the following
capabilities:

   - Safe scaling up and down (including decommissioning)
   - Backup/restore workflow (snapshots only initially)
   - Built in prometheus integration and discovery

Other features like repair, better PV support, maybe even a nice dashboard
will be on the way.


On Wed, May 23, 2018 at 7:35 AM Tom Petracca <tpetra...@palantir.com> wrote:

> Using a statefulset should get you pretty far, though will likely be less
> effective than a coreos-style “operator”. Some random points:
>
>- For scale-up: a node shouldn’t report “ready” until it’s in the
>NORMAL state; this will prevent multiple nodes from bootstrapping at once.
>- For scale-down: as of now there isn’t a mechanism to know if a pod
>is getting decommissioned because you’ve permanently lowered replica count,
>or because it’s just getting bounced/re-scheduled, thus knowing whether or
>not to decommission is basically impossible. Relevant issue:
>kubernetes/kubernetes#1462
><https://github.com/kubernetes/kubernetes/issues/1462>
>
>
>
> *From: *Pradeep Chhetri <prad...@stashaway.com>
> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Date: *Friday, May 18, 2018 at 10:20 AM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *Re: Using K8s to Manage Cassandra in Production
>
>
>
> Hello Hassaan,
>
>
>
> We use cassandra helm chart[0] for deploying cassandra over kubernetes in
> production. We have around 200GB cas data. It works really well. You can
> scale up nodes easily (I haven't tested scaling down).
>
>
>
> I would say that if you are worried about running cassandra over k8s in
> production, maybe you should first try setting it for your
> staging/preproduction and gain confidence over time.
>
>
>
> I have tested situations where i have killed the host running cassandra
> container and have seen that container moves to a different node and joins
> cluster properly. So from my experience its pretty good. No issues till yet.
>
>
>
> [0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra
> [github.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_charts_tree_master_incubator_cassandra=DwMFaQ=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=1oh1YI8i5eJD1DFTwooO7U92fFi2fjan6lqP61yAiyo=dupKDpZi0lkjFkqaSd6XaEj5nuY1T5UObgAcXCBqo7A=0WTYStEM1zvh2BQKvnVLRpukxgr0aDLyGffyE1V2xik=>
>
>
>
>
>
> Regards,
>
> Pradeep
>
>
>
> On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко <amelius0...@gmail.com>
> wrote:
>
> Hi, Hassaan! For example we are using C* in k8s in production for our
> video surveillance system. Moreover, we are using Ceph RBD as our storage
> for cassandra. Today we have 8 C* nodes each manages 2Tb of data.
>
>
>
> On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha <hpa...@an10.io> wrote:
>
> Hi,
>
>
>
> I am trying to craft a deployment strategy for deploying and maintaining a
> C* cluster. I was wondering if there are actual production deployments of
> C* using K8s as the orchestration layer.
>
>
>
> I have been given the impression that K8s managing a C* cluster can be a
> recipe for disaster, especially if you aren't well versed with the
> intricacies of a scale-up/down event. I know use cases where people are
> using Mesos or a custom tool built with terraform/chef etc to run their
> production clusters but have yet to find a real K8s use case.
>
>
>
> *Questions?*
>
> Is K8s a reasonable choice for managing a production C* cluster?
>
> Are there documented use cases for this?
>
>
>
> Any help would be greatly appreciated.
>
>
>
> --
>
> Regards,
>
>
>
> *Hassaan Pasha*
>
> --
>
> Regrads,
>
> Pavel Sapezhko
>
>
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Using K8s to Manage Cassandra in Production

[Tom Petracca]

Using a statefulset should get you pretty far, though will likely be less 
effective than a coreos-style “operator”. Some random points:
For scale-up: a node shouldn’t report “ready” until it’s in the NORMAL state; 
this will prevent multiple nodes from bootstrapping at once.
For scale-down: as of now there isn’t a mechanism to know if a pod is getting 
decommissioned because you’ve permanently lowered replica count, or because 
it’s just getting bounced/re-scheduled, thus knowing whether or not to 
decommission is basically impossible. Relevant issue: kubernetes/kubernetes#1462
 

From: Pradeep Chhetri <prad...@stashaway.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Friday, May 18, 2018 at 10:20 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Using K8s to Manage Cassandra in Production

 

Hello Hassaan, 

 

We use cassandra helm chart[0] for deploying cassandra over kubernetes in 
production. We have around 200GB cas data. It works really well. You can scale 
up nodes easily (I haven't tested scaling down).

 

I would say that if you are worried about running cassandra over k8s in 
production, maybe you should first try setting it for your 
staging/preproduction and gain confidence over time. 

 

I have tested situations where i have killed the host running cassandra 
container and have seen that container moves to a different node and joins 
cluster properly. So from my experience its pretty good. No issues till yet.

 

[0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra 
[github.com]

 

 

Regards,

Pradeep

 

On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко <amelius0...@gmail.com> wrote:

Hi, Hassaan! For example we are using C* in k8s in production for our video 
surveillance system. Moreover, we are using Ceph RBD as our storage for 
cassandra. Today we have 8 C* nodes each manages 2Tb of data. 

 

On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha <hpa...@an10.io> wrote:

Hi, 

 

I am trying to craft a deployment strategy for deploying and maintaining a C* 
cluster. I was wondering if there are actual production deployments of C* using 
K8s as the orchestration layer.

 

I have been given the impression that K8s managing a C* cluster can be a recipe 
for disaster, especially if you aren't well versed with the intricacies of a 
scale-up/down event. I know use cases where people are using Mesos or a custom 
tool built with terraform/chef etc to run their production clusters but have 
yet to find a real K8s use case.

 

Questions?

Is K8s a reasonable choice for managing a production C* cluster?

Are there documented use cases for this?

 

Any help would be greatly appreciated.

 

-- 

Regards, 

 

Hassaan Pasha

-- 

Regrads,
Pavel Sapezhko
 



smime.p7s
Description: S/MIME cryptographic signature

Re: Using K8s to Manage Cassandra in Production

[Pradeep Chhetri]

Hello Hassaan,

We use cassandra helm chart[0] for deploying cassandra over kubernetes in
production. We have around 200GB cas data. It works really well. You can
scale up nodes easily (I haven't tested scaling down).

I would say that if you are worried about running cassandra over k8s in
production, maybe you should first try setting it for your
staging/preproduction and gain confidence over time.

I have tested situations where i have killed the host running cassandra
container and have seen that container moves to a different node and joins
cluster properly. So from my experience its pretty good. No issues till yet.

[0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra


Regards,
Pradeep

On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко 
wrote:

> Hi, Hassaan! For example we are using C* in k8s in production for our
> video surveillance system. Moreover, we are using Ceph RBD as our storage
> for cassandra. Today we have 8 C* nodes each manages 2Tb of data.
>
> On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha  wrote:
>
>> Hi,
>>
>> I am trying to craft a deployment strategy for deploying and maintaining
>> a C* cluster. I was wondering if there are actual production deployments of
>> C* using K8s as the orchestration layer.
>>
>> I have been given the impression that K8s managing a C* cluster can be a
>> recipe for disaster, especially if you aren't well versed with the
>> intricacies of a scale-up/down event. I know use cases where people are
>> using Mesos or a custom tool built with terraform/chef etc to run their
>> production clusters but have yet to find a real K8s use case.
>>
>> *Questions?*
>> Is K8s a reasonable choice for managing a production C* cluster?
>> Are there documented use cases for this?
>>
>> Any help would be greatly appreciated.
>>
>> --
>> Regards,
>>
>>
>> *Hassaan Pasha*
>>
> --
>
> Regrads,
>
> Pavel Sapezhko
>
>

Re: Using K8s to Manage Cassandra in Production

[Павел Сапежко]

Hi, Hassaan! For example we are using C* in k8s in production for our video
surveillance system. Moreover, we are using Ceph RBD as our storage for
cassandra. Today we have 8 C* nodes each manages 2Tb of data.

On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha  wrote:

> Hi,
>
> I am trying to craft a deployment strategy for deploying and maintaining a
> C* cluster. I was wondering if there are actual production deployments of
> C* using K8s as the orchestration layer.
>
> I have been given the impression that K8s managing a C* cluster can be a
> recipe for disaster, especially if you aren't well versed with the
> intricacies of a scale-up/down event. I know use cases where people are
> using Mesos or a custom tool built with terraform/chef etc to run their
> production clusters but have yet to find a real K8s use case.
>
> *Questions?*
> Is K8s a reasonable choice for managing a production C* cluster?
> Are there documented use cases for this?
>
> Any help would be greatly appreciated.
>
> --
> Regards,
>
>
> *Hassaan Pasha*
>
-- 

Regrads,

Pavel Sapezhko