[
https://issues.apache.org/jira/browse/CASSANDRA-15717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eddy Truyen updated CASSANDRA-15717:
------------------------------------
Attachment: (was: kube-adm-cassandra-nodetool-tablestats)
> Benchmark performance difference between Docker and Kubernetes when running
> Cassandra:2.2.16 official Docker image
> ------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15717
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15717
> Project: Cassandra
> Issue Type: Bug
> Components: Test/benchmark
> Reporter: Eddy Truyen
> Priority: Normal
>
> This is my first JIRA issue. Sorry if I do something wrong in the reporting.
> I experienced a performance degradation when running a single Cassandra
> Docker container inside Kubernetes in comparison with running the Docker
> container stand-alone. I used the following image decomads/cassandra:2.2.16,
> which uses cassandra:2.2.16 as base image and adds a readinessProbe to it.
> I used identical Docker configuration parameters by ensuring that the output
> of docker inspect is as much as possible the same. First we got the ycsb
> benchmark in a container that is co-located with the cassandra container in
> one pod. Kubernetes starts these containers then with network mode
> "net=container:... This is a separate container that link up the ycsb and
> cassandra containers within the same network space so they can talk via
> localhost – by this we hope to avoid network plugin interference from the CNI
> plugin.
> We ran the docker-only container within the Kubernetes node using the default
> bridge network
> Experiment (repeated on minikube+VirtualBox (12GB, 4 CPU cores, 30 GB) on
> physical laptop with 4 cores/8 logical processors and 16GB RAM on and
> Openstack VM Ubuntu 16:04 (4GB, 4 CPU cores, 50GB), that runs on a physical
> nodes with 16 CPU cores. Storage is Ceph.
> * A write-only workload (YCSB benchmark workload A - Load phase) using the
> following user table:
> cqlsh> create keyspace ycsb
> WITH REPLICATION = \{'class' : 'SimpleStrategy', 'replication_factor': 1 }
> ;
> cqlsh> USE ycsb;
> cqlsh> create table usertable (
> y_id varchar primary key,
> field0 varchar,
> field1 varchar,
> field2 varchar,
> field3 varchar,
> field4 varchar,
> field5 varchar,
> field6 varchar,
> field7 varchar,
> field8 varchar,
> field9 varchar);
> * And using the following script: python ./bin/ycsb load cassandra2-cql -P
> workloads/workloada -p recordcount=1500000 -p operationcount=1500000 -p
> measurementtype=raw -p cassandra.connecttimeoutmillis=60000 -p
> cassandra.readtimeoutmillis=60000 -target 1500 -threads 20 -p hosts=localhost
> >
> results/cassandra-docker/cassandra-docker-load-workloada-1-records-1500000-rnd-1762034446.txt
> sleep 15
> Observations (On Ubuntu-OpenStack)
> * Docker:
> ** Mean average response latency YCSB benchmark: 1,5 ms-1.7ms
> * Kubernetes
> ** Mean average response latency YCSB benchmark: 2.7 ms-3ms
> * CPU usage of the Cassandra Daemon JVM is way lower than Kubernetes (see my
> position paper: [https://lirias.kuleuven.be/2788169?limo=0)]:
> Possible causes:
> * Network overhead of virtual bridge in container orchestrator is not the
> cause of the problem in our opinion
> ** We repeated the experiment where we ran the Docker-Only containers inside
> a Kubernetes node and we linked the containers using the --net=container:
> mode mechanisms as similar as possible as we could. The YCSB latency stayed
> the same.
> * Disk/io bottleneck: Nodetool tablestats are very similar
> ** Write latency is very stable over multiple runs
> *** Kubernetes for ycsb user table: 0.0167 ms.
> *** Write latency Docker for ycsb usertable: 0.0150 ms.
> ** Compaction_history/compaction_in_progress is also very similar (as
> opposed to earlier versions of the issue – sorry for the confusion!)
> *
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
