Re: Cassandra image for Kubernetes

[Joseph Lynch]

On Fri, Sep 20, 2019 at 8:09 AM Ben Bromhead  wrote:
>
> Providing an official docker image is a little tricky, as despite what
> container marketing would tell you, containers need to make assumptions
> about outside orchestration/management methods. Folks in this thread have
> already identified differences in kubernetes distro's let alone other
> container schedulers.

Just to clarify the proposal Vinay and I made at the summit, I don't
think that we can provide a single image that works for every single
use case just like Cassandra's out of the box debian package does not
work perfectly out of the box for many configuration management /
orchestration systems, nor did I intend to propose we provide an
official image for kubernetes, marathon, DCOS, or whichever new
scheduler is popular these days.

I do think we can offer two generally configurable and reasonable base
Cassandra containers, one for testing and one for production. Both
containers must provide the requisite pluggability seams for common
use cases just like the debian and rpm packages do. For example, a
pluggable configuration file (I usually offer overrides via
environment variables in my images), ability to call a user provided
bash script before starting the daemon, a way to change jvm options,
etc ... These seams would then be documented so that people can easily
plug in their needed functionality. The testing image would optimize
for fast startup and single node clusters (e.g. turning off vnodes,
skipping integrity checks, etc ...). The production image would
naturally not turn these things off.

If a user cannot plug-in their functionality then they can raise a bug
report explaining the difficulty and we can either add the needed seam
or say "sounds like you need a custom image". It would be nice if the
community contributed documentation for "here is how you take the
production image and run it on kubernetes" but I don't think the
Cassandra developers need to maintain this integration.

Yes, these will not satisfy every use case, but they are still very
valuable even if only as a starting point for the community.

-Joey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Cassandra image for Kubernetes

[Ben Bromhead]

Providing an official docker image is a little tricky, as despite what
container marketing would tell you, containers need to make assumptions
about outside orchestration/management methods. Folks in this thread have
already identified differences in kubernetes distro's let alone other
container schedulers.

For example do you provide configuration via environment variables, docker
configs, kubernetes configmap/secret mechanism etc? Does your container
environment allow you to run containers as root or specify kernel
capabilities? C* has some explicit requirements around what it expects of
the OS in that regard for stability and performance.

What base image and userland tools do you use?

On top of that, many operators (folks that run C*, not the k8s concept)
need to embed their own sidecars and agents. For example embedding the
jolokia agent vs using JMX vs using a prometheus exporter etc.

A quick survey across the ecosystem shows that there are wildly different
opinions about what a cassandra container should look like and include:

   - https://github.com/docker-library/cassandra
   - https://github.com/datastax/docker-images
   - https://github.com/thelastpickle/docker-cassandra-bootstrap
   - https://github.com/bitnami/bitnami-docker-cassandra
   -
   
https://github.com/instaclustr/cassandra-operator/tree/master/docker/cassandra

That's not to say it can't be done, it's just it creates an explicit and
rigid contract in the way containers are configured, what's included and
how things are managed. Don't take this commentary as me being against the
idea, just based on my experience, it will be hard to meet even a slim
majority of users production requirements for an image :)

On Thu, Sep 19, 2019 at 5:38 PM Dimo Velev  wrote:

> Sidecar would be fine I guess provided that it does not require a lot of
> resources - which kind of eliminates more or less any java implementation.
> Unsure whether jvm metrics etc are available via virtual tables though.
>
> Also forgot an important point about the image for local usage - ability
> to load cql scripts upon start only once. Most databases offer these so
> that you can add initialization stuff like creating keyspaces, users,
> tables upon first start.
>
> > On 19. Sep 2019, at 23:02, Jon Haddad  wrote:
> >
> > I recently had a side conversation about including a Prometheus endpoint
> in
> > the sidecar project, which would query virtual tables exclusively.  Is it
> > reasonable to require running the sidecar in addition to Cassandra when
> > deploying on K8?  My container knowledge is pretty limited.
> >
> > If that's not a problem, I think it shouldn't be too hard to add, it's
> just
> > a matter of someone finding the time.
> >
> > On Thu, Sep 19, 2019 at 1:39 PM Dimo Velev  wrote:
> >
> >> Hi,
> >>
> >> A docker image for Cassandra would be the first step in the right
> >> direction. What one would really want is a kubernetes Operator that can
> >> deploy, upgrade, etc. a Cassandra cluster (
> >> https://kubernetes.io/docs/concepts/extend-kubernetes/operator).
> >>
> >> A base image with placeholders would be neat for people to quickly start
> >> using Cassandra. We have built our own image (actually a few of them
> over
> >> the time) and also use it e.g. for cqlsh (docker run -it —rm —entrypoint
> >> cqlsh  —help). What one would probably do in the end is building
> >> their own image based on the official one adding a layer with the
> >> configuration file, certificates, etc (or mounting them from a
> configmap in
> >> kubernetes).
> >>
> >> Some things to consider for the image:
> >> • Special care must be taken to make the image usable in OpenShift as it
> >> does not run the processes in the containers as root.
> >> • Logging should be changed to stdout so that it is easier to use and
> >> automatically picked up by log indexers of kubernetes clusters.
> >> • As for metrics, exposing Prometheus endpoint with them would be great
> as
> >> this makes scraping them just a matter of meta configuration of the pods
> >> (telling Prometheus the actual endpoint to scrape)
> >> • I had issues with Cassandra binding on the wrong IP in a container
> >>
> >> Would be happy to help with the image part / testing on OpenShift.
> >>
> >> Cheers,
> >> Dimo
> >>
> >>> On 19. Sep 2019, at 18:32, Nate McCall  wrote:
> >>>
> >>> Hi Cyril,
> >>> Thanks for bringing this topic up. I think it would be a good idea for
> us
> >>> to have an "official" docker file in the source tree.
> >>>
> >>> There are, however, some caveats:
> >>>
> >>
> https://issues.apache.org/jira/browse/LEGAL-270?focusedCommentId=15524446=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15524446
> >>>
> >>> As long as we could adhere to those guidelines (which I don't see as
> >> hard)
> >>> we could do this.
> >>>
> >>> In looking through your specific image (thanks for posting this, btw),
> I
> >>> would personally prefer something with a lot fewer dependencies
> >> 

Re: Cassandra image for Kubernetes

[Dimo Velev]

Sidecar would be fine I guess provided that it does not require a lot of 
resources - which kind of eliminates more or less any java implementation. 
Unsure whether jvm metrics etc are available via virtual tables though.

Also forgot an important point about the image for local usage - ability to 
load cql scripts upon start only once. Most databases offer these so that you 
can add initialization stuff like creating keyspaces, users, tables upon first 
start.

> On 19. Sep 2019, at 23:02, Jon Haddad  wrote:
> 
> I recently had a side conversation about including a Prometheus endpoint in
> the sidecar project, which would query virtual tables exclusively.  Is it
> reasonable to require running the sidecar in addition to Cassandra when
> deploying on K8?  My container knowledge is pretty limited.
> 
> If that's not a problem, I think it shouldn't be too hard to add, it's just
> a matter of someone finding the time.
> 
> On Thu, Sep 19, 2019 at 1:39 PM Dimo Velev  wrote:
> 
>> Hi,
>> 
>> A docker image for Cassandra would be the first step in the right
>> direction. What one would really want is a kubernetes Operator that can
>> deploy, upgrade, etc. a Cassandra cluster (
>> https://kubernetes.io/docs/concepts/extend-kubernetes/operator).
>> 
>> A base image with placeholders would be neat for people to quickly start
>> using Cassandra. We have built our own image (actually a few of them over
>> the time) and also use it e.g. for cqlsh (docker run -it —rm —entrypoint
>> cqlsh  —help). What one would probably do in the end is building
>> their own image based on the official one adding a layer with the
>> configuration file, certificates, etc (or mounting them from a configmap in
>> kubernetes).
>> 
>> Some things to consider for the image:
>> • Special care must be taken to make the image usable in OpenShift as it
>> does not run the processes in the containers as root.
>> • Logging should be changed to stdout so that it is easier to use and
>> automatically picked up by log indexers of kubernetes clusters.
>> • As for metrics, exposing Prometheus endpoint with them would be great as
>> this makes scraping them just a matter of meta configuration of the pods
>> (telling Prometheus the actual endpoint to scrape)
>> • I had issues with Cassandra binding on the wrong IP in a container
>> 
>> Would be happy to help with the image part / testing on OpenShift.
>> 
>> Cheers,
>> Dimo
>> 
>>> On 19. Sep 2019, at 18:32, Nate McCall  wrote:
>>> 
>>> Hi Cyril,
>>> Thanks for bringing this topic up. I think it would be a good idea for us
>>> to have an "official" docker file in the source tree.
>>> 
>>> There are, however, some caveats:
>>> 
>> https://issues.apache.org/jira/browse/LEGAL-270?focusedCommentId=15524446=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15524446
>>> 
>>> As long as we could adhere to those guidelines (which I don't see as
>> hard)
>>> we could do this.
>>> 
>>> In looking through your specific image (thanks for posting this, btw), I
>>> would personally prefer something with a lot fewer dependencies
>> (basically
>>> from just our source tree) and a lot more replacement properties
>> available
>>> for config files.
>>> 
>>> Curious what other folks think?
>>> 
>>> Cheers,
>>> -Nate
>>> 
 On Wed, Sep 18, 2019 at 6:43 AM Cyril Scetbon 
>> wrote:
 
 Hey guys,
 
 I heard that at the last summit there were discussions about providing
>> an
 official docker image to run Cassandra on Kubernetes. Is it something
>> that
 you’ve started to work on ? We have our own at
 https://github.com/Orange-OpenSource/cassandra-image <
 https://github.com/Orange-OpenSource/cassandra-image> but I think
 providing an official image makes sense. As long as we can easily do
 everything we do today. We could also collaborate.
 
 Thank you
 —
 Cyril Scetbon
 
 
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Cassandra image for Kubernetes

[Jon Haddad]

I recently had a side conversation about including a Prometheus endpoint in
the sidecar project, which would query virtual tables exclusively.  Is it
reasonable to require running the sidecar in addition to Cassandra when
deploying on K8?  My container knowledge is pretty limited.

If that's not a problem, I think it shouldn't be too hard to add, it's just
a matter of someone finding the time.

On Thu, Sep 19, 2019 at 1:39 PM Dimo Velev  wrote:

> Hi,
>
> A docker image for Cassandra would be the first step in the right
> direction. What one would really want is a kubernetes Operator that can
> deploy, upgrade, etc. a Cassandra cluster (
> https://kubernetes.io/docs/concepts/extend-kubernetes/operator).
>
> A base image with placeholders would be neat for people to quickly start
> using Cassandra. We have built our own image (actually a few of them over
> the time) and also use it e.g. for cqlsh (docker run -it —rm —entrypoint
> cqlsh  —help). What one would probably do in the end is building
> their own image based on the official one adding a layer with the
> configuration file, certificates, etc (or mounting them from a configmap in
> kubernetes).
>
> Some things to consider for the image:
> • Special care must be taken to make the image usable in OpenShift as it
> does not run the processes in the containers as root.
> • Logging should be changed to stdout so that it is easier to use and
> automatically picked up by log indexers of kubernetes clusters.
> • As for metrics, exposing Prometheus endpoint with them would be great as
> this makes scraping them just a matter of meta configuration of the pods
> (telling Prometheus the actual endpoint to scrape)
> • I had issues with Cassandra binding on the wrong IP in a container
>
> Would be happy to help with the image part / testing on OpenShift.
>
> Cheers,
> Dimo
>
> > On 19. Sep 2019, at 18:32, Nate McCall  wrote:
> >
> > Hi Cyril,
> > Thanks for bringing this topic up. I think it would be a good idea for us
> > to have an "official" docker file in the source tree.
> >
> > There are, however, some caveats:
> >
> https://issues.apache.org/jira/browse/LEGAL-270?focusedCommentId=15524446=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15524446
> >
> > As long as we could adhere to those guidelines (which I don't see as
> hard)
> > we could do this.
> >
> > In looking through your specific image (thanks for posting this, btw), I
> > would personally prefer something with a lot fewer dependencies
> (basically
> > from just our source tree) and a lot more replacement properties
> available
> > for config files.
> >
> > Curious what other folks think?
> >
> > Cheers,
> > -Nate
> >
> >> On Wed, Sep 18, 2019 at 6:43 AM Cyril Scetbon 
> wrote:
> >>
> >> Hey guys,
> >>
> >> I heard that at the last summit there were discussions about providing
> an
> >> official docker image to run Cassandra on Kubernetes. Is it something
> that
> >> you’ve started to work on ? We have our own at
> >> https://github.com/Orange-OpenSource/cassandra-image <
> >> https://github.com/Orange-OpenSource/cassandra-image> but I think
> >> providing an official image makes sense. As long as we can easily do
> >> everything we do today. We could also collaborate.
> >>
> >> Thank you
> >> —
> >> Cyril Scetbon
> >>
> >>
>

Re: Cassandra image for Kubernetes

[Dimo Velev]

Hi,

A docker image for Cassandra would be the first step in the right direction. 
What one would really want is a kubernetes Operator that can deploy, upgrade, 
etc. a Cassandra cluster 
(https://kubernetes.io/docs/concepts/extend-kubernetes/operator).

A base image with placeholders would be neat for people to quickly start using 
Cassandra. We have built our own image (actually a few of them over the time) 
and also use it e.g. for cqlsh (docker run -it —rm —entrypoint cqlsh  
—help). What one would probably do in the end is building their own image based 
on the official one adding a layer with the configuration file, certificates, 
etc (or mounting them from a configmap in kubernetes).

Some things to consider for the image:
• Special care must be taken to make the image usable in OpenShift as it does 
not run the processes in the containers as root.
• Logging should be changed to stdout so that it is easier to use and 
automatically picked up by log indexers of kubernetes clusters.
• As for metrics, exposing Prometheus endpoint with them would be great as this 
makes scraping them just a matter of meta configuration of the pods (telling 
Prometheus the actual endpoint to scrape)
• I had issues with Cassandra binding on the wrong IP in a container

Would be happy to help with the image part / testing on OpenShift.

Cheers,
Dimo

> On 19. Sep 2019, at 18:32, Nate McCall  wrote:
> 
> Hi Cyril,
> Thanks for bringing this topic up. I think it would be a good idea for us
> to have an "official" docker file in the source tree.
> 
> There are, however, some caveats:
> https://issues.apache.org/jira/browse/LEGAL-270?focusedCommentId=15524446=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15524446
> 
> As long as we could adhere to those guidelines (which I don't see as hard)
> we could do this.
> 
> In looking through your specific image (thanks for posting this, btw), I
> would personally prefer something with a lot fewer dependencies (basically
> from just our source tree) and a lot more replacement properties available
> for config files.
> 
> Curious what other folks think?
> 
> Cheers,
> -Nate
> 
>> On Wed, Sep 18, 2019 at 6:43 AM Cyril Scetbon  wrote:
>> 
>> Hey guys,
>> 
>> I heard that at the last summit there were discussions about providing an
>> official docker image to run Cassandra on Kubernetes. Is it something that
>> you’ve started to work on ? We have our own at
>> https://github.com/Orange-OpenSource/cassandra-image <
>> https://github.com/Orange-OpenSource/cassandra-image> but I think
>> providing an official image makes sense. As long as we can easily do
>> everything we do today. We could also collaborate.
>> 
>> Thank you
>> —
>> Cyril Scetbon
>> 
>> 

Re: Cassandra image for Kubernetes

[Nate McCall]

Hi Cyril,
Thanks for bringing this topic up. I think it would be a good idea for us
to have an "official" docker file in the source tree.

There are, however, some caveats:
https://issues.apache.org/jira/browse/LEGAL-270?focusedCommentId=15524446=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15524446

As long as we could adhere to those guidelines (which I don't see as hard)
we could do this.

In looking through your specific image (thanks for posting this, btw), I
would personally prefer something with a lot fewer dependencies (basically
from just our source tree) and a lot more replacement properties available
for config files.

Curious what other folks think?

Cheers,
-Nate

On Wed, Sep 18, 2019 at 6:43 AM Cyril Scetbon  wrote:

> Hey guys,
>
> I heard that at the last summit there were discussions about providing an
> official docker image to run Cassandra on Kubernetes. Is it something that
> you’ve started to work on ? We have our own at
> https://github.com/Orange-OpenSource/cassandra-image <
> https://github.com/Orange-OpenSource/cassandra-image> but I think
> providing an official image makes sense. As long as we can easily do
> everything we do today. We could also collaborate.
>
> Thank you
> —
> Cyril Scetbon
>
>

Cassandra image for Kubernetes

[Cyril Scetbon]

Hey guys,

I heard that at the last summit there were discussions about providing an 
official docker image to run Cassandra on Kubernetes. Is it something that 
you’ve started to work on ? We have our own at 
https://github.com/Orange-OpenSource/cassandra-image 
 but I think providing an 
official image makes sense. As long as we can easily do everything we do today. 
We could also collaborate.

Thank you
—
Cyril Scetbon