Hello All,
I have a 3 node cassandra cluster with Replication factor as 2 and
read-write consistency set to QUORUM. We are using Spring data cassandra.
All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then
we try to modify/update one of the record using save method of repo, like
below:
ChunkMeta *tmpRec* = chunkMetaRepository.*save*(chunkMeta);
After execution of above statement we never see any exception or error. But
still this update state goes silent/fail intermittently. That is at times
the record in the db gets updated successfully where as other time it
fails. Also in the above query when we print *tmpRec* it contains the
updated and correct value every time. Still in the db these updated values
doesn't get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the
our queries are getting logged there and are being executed also with out
any error or exception.
Now another weird observation is this all thing works erfectly fine if I am
using single cassandra node (in kubernetes) or if we deploy above infra
using ansible (even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment
of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
I have a 3 node cassandra cluster with Replication factor as 2 and
read-write consistency set to QUORUM. We are using Spring data cassandra.
All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then
we try to modify/update one of the record using save method of repo, like
below:
ChunkMeta tmpRec = chunkMetaRepository.*save*(chunkMeta);
After execution of above statement we never see any exception or error. But
still this update fail intermittently. That is when we check the record in
the db sometime it gets updated successfully where as other time it fails.
Also in the above query when we print *tmpRec* it contains the updated and
correct value. Still in the db these updated values doesnt get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the
our queries are getting logged there and are being executed also.
Now another weird observation is this all thing works if I am using single
cassandra node (in kubernetes) or if we deploy above infra using ansible
(even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment
of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
Below are the contents of my cassandra Docker file:
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils
net-tools && apt-get clean && \
addgroup testuser && useradd -g testuser testuser && usermod
--password testuser testuser;
RUN mkdir -p /opt/test && \
mkdir -p /opt/test/data;
ADD jre8.tar.gz /opt/test/
ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
RUN chmod 755 -R /opt/test/jre && \
ln -s /opt/test/jre/bin/java /usr/bin/java && \
mv /opt/test/apache-cassandra* /opt/test/cassandra;
RUN mkdir -p /opt/test/cassandra/logs;
ENV JAVA_HOME /opt/test/jre
RUN export JAVA_HOME
COPY version.txt /opt/test/cassandra/version.txt
WORKDIR /opt/test/cassandra/bin/
RUN mkdir -p /opt/test/data/saved_caches && \
mkdir -p /opt/test/data/commitlog && \
mkdir -p /opt/test/data/hints && \
chown -R testuser:testuser /opt/test/data && \
chown -R testuser:testuser /opt/test;
USER testuser
CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e
's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg;
s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml && rm
../conf/conf.yml && ./cassandra -f
Please note conf.yml is basically cassandra.yml file having properties
related to cassandra.
Thanks,
Mahesh Daksha
