[
https://issues.apache.org/jira/browse/CASSANDRA-15717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eddy Truyen updated CASSANDRA-15717:
------------------------------------
Description:
This is my first JIRA issue. Sorry if I do something wrong in the reporting.
I experienced a performance degradation when running a single Cassandra Docker
container inside Kubernetes in comparison with running the Docker container
stand-alone. I used the following image decomads/cassandra:2.2.16, which uses
cassandra:2.2.16 as base image and adds a readinessProbe to it.
I used identical Docker configuration parameters by ensuring that the output of
docker inspect is as much as possible the same. Docker runs in bridged mode
Experiment (repeated on minikube+VirtualBox (12GB, 4 CPU cores, 30 GB) on
physical laptop with 4 cores/8 logical processors and 16GB RAM on and Openstack
VM Ubuntu 16:04 (4GB, 4 CPU cores, 50GB), that runs on a physical nodes with
16 CPU cores. Storage is Ceph.
* A write-only workload (YCSB benchmark workload A - Load phase) using the
following user table:
cqlsh> create keyspace ycsb
WITH REPLICATION = \{'class' : 'SimpleStrategy', 'replication_factor': 1 }
;
cqlsh> USE ycsb;
cqlsh> create table usertable (
y_id varchar primary key,
field0 varchar,
field1 varchar,
field2 varchar,
field3 varchar,
field4 varchar,
field5 varchar,
field6 varchar,
field7 varchar,
field8 varchar,
field9 varchar);
* And using the following script: python ./bin/ycsb load cassandra2-cql -P
workloads/workloada -p recordcount=1500000 -p operationcount=1500000 -p
measurementtype=raw -p cassandra.connecttimeoutmillis=60000 -p
cassandra.readtimeoutmillis=60000 -target 1500 -threads 20 -p hosts=localhost >
results/cassandra-docker/cassandra-docker-load-workloada-1-records-1500000-rnd-1762034446.txt
sleep 15
Observations:
* Docker:
** Mean average response latency: 1500
** Average CPU usage of cassandra instances (wrt 2 cores): 42
* Kubernetes
** Mean average response latency: 2700
** Average CPU usage of cassandra instance (wrt 2 cores): 32%
* Nodetool tablestats
** There are little difference for the usertable, with an almost identical
write latency (difference < 0.002 ms).
** However for the system keyspace there are quite some differences in
read/write count and read latency (difference = 2.5 ms). More specifically,
compaction history (see attachment the 2 tablestats output)
*** Table: compaction_history Kubernetes
SSTable count: 1
Space used (live): 12049
Space used (total): 12049
Space used by snapshots (total): 0
Off heap memory used (total): 108
SSTable Compression Ratio: 0.25466231166368136
Number of keys (estimate): 54
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 12
*** Table: compaction_history Docker
SSTable count: 1
Space used (live): 8921
Space used (total): 8921
Space used by snapshots (total): 0
Off heap memory used (total): 76
SSTable Compression Ratio: 0.2603359822955437
Number of keys (estimate): 25
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
* Cassandra Logs:
*
|Docker|Kubernetes|
|ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
Memtable-compaction' docker-cassandra-logs
27
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-size'
docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c ' Writing
Memtable-sstable' docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
Memtable-usertable' docker-cassandra-logs
45
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
Memtable-compactions_in_progress' docker-cassandra-logs
26
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-peers'
docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
Memtable-schema' docker-cassandra-logs
24
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-local'
docker-cassandra-logs
6|ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
Memtable-compaction' kubeadm-cassandra-logs \| more
32
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-size'
kubeadm-cassandra-logs \| more
7
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c ' Writing
Memtable-sstable' kubeadm-cassandra-logs \| more
7
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
Memtable-usertable' kubeadm-cassandra-logs \| more
45
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
Memtable-compactions_in_progress' kubeadm-cassandra-logs \| more
26
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-peers'
kubeadm-cassandra-logs \| more
2
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-schema'
kubeadm-cassandra-logs
17
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-local'
kubeadm-cassandra-logs
5|
Can the higher compaction rate of Kubernetes-based Cassandra instance be the
cause of the latency for YCSB benchmark? And how could this be resolved?
was:
This is my first JIRA issue. Sorry if I do something wrong in the reporting.
I experienced a performance degradation when running a single Cassandra Docker
container inside Kubernetes in comparison with running the Docker container
stand-alone. I used the following image decomads/cassandra:2.2.16, which uses
cassandra:2.2.16 as base image and adds a readinessProbe to it.
I used identical Docker configuration parameters by ensuring that the output of
docker inspect is as much as possible the same. Docker runs in bridged mode
Experiment (repeated on minikube+VirtualBox (12GB, 4 CPU cores, 30 GB) on
physical laptop with 4 cores/8 logical processors and 16GB RAM on and Openstack
VM Ubuntu 16:04 (4GB, 4 CPU cores, 50GB), that runs on a physical nodes with
16 CPU cores. Storage is Ceph.
* A write-only workload (YCSB benchmark workload A - Load phase) using the
following user table:
cqlsh> create keyspace ycsb
WITH REPLICATION = \{'class' : 'SimpleStrategy', 'replication_factor': 3 }
;
cqlsh> USE ycsb;
cqlsh> create table usertable (
y_id varchar primary key,
field0 varchar,
field1 varchar,
field2 varchar,
field3 varchar,
field4 varchar,
field5 varchar,
field6 varchar,
field7 varchar,
field8 varchar,
field9 varchar);
* And using the following script: python ./bin/ycsb load cassandra2-cql -P
workloads/workloada -p recordcount=1500000 -p operationcount=1500000 -p
measurementtype=raw -p cassandra.connecttimeoutmillis=60000 -p
cassandra.readtimeoutmillis=60000 -target 1500 -threads 20 -p hosts=localhost >
results/cassandra-docker/cassandra-docker-load-workloada-1-records-1500000-rnd-1762034446.txt
sleep 15
Observations:
* Docker:
** Mean average response latency: 1500
** Average CPU usage of cassandra instances (wrt 2 cores): 42
* Kubernetes
** Mean average response latency: 2700
** Average CPU usage of cassandra instance (wrt 2 cores): 32%
* Nodetool tablestats
** There are little difference for the usertable, with an almost identical
write latency (difference < 0.002 ms).
** However for the system keyspace there are quite some differences in
read/write count and read latency (difference = 2.5 ms). More specifically,
compaction history (see attachment the 2 tablestats output)
*** Table: compaction_history Kubernetes
SSTable count: 1
Space used (live): 12049
Space used (total): 12049
Space used by snapshots (total): 0
Off heap memory used (total): 108
SSTable Compression Ratio: 0.25466231166368136
Number of keys (estimate): 54
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 12
*** Table: compaction_history Docker
SSTable count: 1
Space used (live): 8921
Space used (total): 8921
Space used by snapshots (total): 0
Off heap memory used (total): 76
SSTable Compression Ratio: 0.2603359822955437
Number of keys (estimate): 25
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
* Cassandra Logs:
*
|Docker|Kubernetes|
|ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
Memtable-compaction' docker-cassandra-logs
27
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-size'
docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c ' Writing
Memtable-sstable' docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
Memtable-usertable' docker-cassandra-logs
45
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
Memtable-compactions_in_progress' docker-cassandra-logs
26
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-peers'
docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-schema'
docker-cassandra-logs
24
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-local'
docker-cassandra-logs
6|ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
Memtable-compaction' kubeadm-cassandra-logs \| more
32
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-size'
kubeadm-cassandra-logs \| more
7
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c ' Writing
Memtable-sstable' kubeadm-cassandra-logs \| more
7
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
Memtable-usertable' kubeadm-cassandra-logs \| more
45
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
Memtable-compactions_in_progress' kubeadm-cassandra-logs \| more
26
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-peers'
kubeadm-cassandra-logs \| more
2
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-schema'
kubeadm-cassandra-logs
17
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-local'
kubeadm-cassandra-logs
5|
Can the higher compaction rate of Kubernetes-based Cassandra instance be the
cause of the latency for YCSB benchmark? And how could this be resolved?
> Benchmark performance difference between Docker and Kubernetes when running
> Cassandra:2.2.16 official Docker image
> ------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15717
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15717
> Project: Cassandra
> Issue Type: Bug
> Components: Test/benchmark
> Reporter: Eddy Truyen
> Priority: Normal
> Attachments: docker-cassandra-nodetool-tablestats,
> kube-adm-cassandra-nodetool-tablestats
>
>
> This is my first JIRA issue. Sorry if I do something wrong in the reporting.
> I experienced a performance degradation when running a single Cassandra
> Docker container inside Kubernetes in comparison with running the Docker
> container stand-alone. I used the following image decomads/cassandra:2.2.16,
> which uses cassandra:2.2.16 as base image and adds a readinessProbe to it.
> I used identical Docker configuration parameters by ensuring that the output
> of docker inspect is as much as possible the same. Docker runs in bridged
> mode
> Experiment (repeated on minikube+VirtualBox (12GB, 4 CPU cores, 30 GB) on
> physical laptop with 4 cores/8 logical processors and 16GB RAM on and
> Openstack VM Ubuntu 16:04 (4GB, 4 CPU cores, 50GB), that runs on a physical
> nodes with 16 CPU cores. Storage is Ceph.
> * A write-only workload (YCSB benchmark workload A - Load phase) using the
> following user table:
> cqlsh> create keyspace ycsb
> WITH REPLICATION = \{'class' : 'SimpleStrategy', 'replication_factor': 1 }
> ;
> cqlsh> USE ycsb;
> cqlsh> create table usertable (
> y_id varchar primary key,
> field0 varchar,
> field1 varchar,
> field2 varchar,
> field3 varchar,
> field4 varchar,
> field5 varchar,
> field6 varchar,
> field7 varchar,
> field8 varchar,
> field9 varchar);
> * And using the following script: python ./bin/ycsb load cassandra2-cql -P
> workloads/workloada -p recordcount=1500000 -p operationcount=1500000 -p
> measurementtype=raw -p cassandra.connecttimeoutmillis=60000 -p
> cassandra.readtimeoutmillis=60000 -target 1500 -threads 20 -p hosts=localhost
> >
> results/cassandra-docker/cassandra-docker-load-workloada-1-records-1500000-rnd-1762034446.txt
> sleep 15
> Observations:
> * Docker:
> ** Mean average response latency: 1500
> ** Average CPU usage of cassandra instances (wrt 2 cores): 42
> * Kubernetes
> ** Mean average response latency: 2700
> ** Average CPU usage of cassandra instance (wrt 2 cores): 32%
> * Nodetool tablestats
> ** There are little difference for the usertable, with an almost identical
> write latency (difference < 0.002 ms).
> ** However for the system keyspace there are quite some differences in
> read/write count and read latency (difference = 2.5 ms). More specifically,
> compaction history (see attachment the 2 tablestats output)
> *** Table: compaction_history Kubernetes
> SSTable count: 1
> Space used (live): 12049
> Space used (total): 12049
> Space used by snapshots (total): 0
> Off heap memory used (total): 108
> SSTable Compression Ratio: 0.25466231166368136
> Number of keys (estimate): 54
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 12
> *** Table: compaction_history Docker
> SSTable count: 1
> Space used (live): 8921
> Space used (total): 8921
> Space used by snapshots (total): 0
> Off heap memory used (total): 76
> SSTable Compression Ratio: 0.2603359822955437
> Number of keys (estimate): 25
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 1
> * Cassandra Logs:
> *
> |Docker|Kubernetes|
> |ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
> Memtable-compaction' docker-cassandra-logs
> 27
> ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
> Memtable-size' docker-cassandra-logs
> 1
> ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c ' Writing
> Memtable-sstable' docker-cassandra-logs
> 1
> ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
> Memtable-usertable' docker-cassandra-logs
> 45
> ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
> Memtable-compactions_in_progress' docker-cassandra-logs
> 26
> ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
> Memtable-peers' docker-cassandra-logs
> 1
> ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
> Memtable-schema' docker-cassandra-logs
> 24
> ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing
> Memtable-local' docker-cassandra-logs
> 6|ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
> Memtable-compaction' kubeadm-cassandra-logs \| more
> 32
> ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-size'
> kubeadm-cassandra-logs \| more
> 7
> ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c ' Writing
> Memtable-sstable' kubeadm-cassandra-logs \| more
> 7
> ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
> Memtable-usertable' kubeadm-cassandra-logs \| more
> 45
> ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
> Memtable-compactions_in_progress' kubeadm-cassandra-logs \| more
> 26
> ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
> Memtable-peers' kubeadm-cassandra-logs \| more
> 2
> ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing
> Memtable-schema' kubeadm-cassandra-logs
> 17
> ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-local'
> kubeadm-cassandra-logs
> 5|
> Can the higher compaction rate of Kubernetes-based Cassandra instance be the
> cause of the latency for YCSB benchmark? And how could this be resolved?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
