Re: Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException:
Hi, how much data you are trying to read in the single query? is it large in size or normal text data. Looking at the exception it seems the node is unable to deliver data within stipulated time. I have faced similar issue with the response data in huge in size (some binary data). But it was solved as we spread the data across multiple rows. Thanks, Mahesh Daksha On Wed, Apr 17, 2019 at 11:42 AM Krishnanand Khambadkone wrote: > Hi, I have a single instance cassandra server. I am trying to execute a > query with ALLOW FILTERING option. When I run this same query from cqlsh > it runs fine but when I try to execute the query through the java driver it > throws this exception. I have increased all the timeouts in cassandra.yaml > file and also included the read timeout option in the SimpleStetement query > I am running. Any idea how I can fix this issue. > > Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: > Cassandra timeout during read query at consistency LOCAL_ONE (1 responses > were required but only 0 replica responded) > >
Re: Getting Consistency level TWO when it is requested LOCAL_ONE
Hi Jean, I want to understand how you are setting the write consistency level as LOCAL ONE. That is with every query you mentioning consistency level or you have set the spring cassandra config with provided consistency level. Like this: cluster.setQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.valueOf(cassandraConsistencyLevel))); The only possibility i see of such behavior is its getting overridden from some where. Thanks, Mahesh Daksha On Thu, Apr 11, 2019 at 1:43 PM Jean Carlo wrote: > Hello everyone, > > I have a case where the developers are using spring data framework for > Cassandra. We are writing batches setting consistency level at LOCAL_ONE > but we got a timeout like this > > *Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: > Cassandra timeout during BATCH_LOG write query at consistency TWO (2 > replica were required but only 1 acknowledged the write)* > > Is it Cassandra that somehow writes to the *system.batchlog* using > consistency TWO or is it spring data that makes some dirty things behind > the scenes ? > (I want to believe it is the second one) > > Cheers > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay >
Cassandra client side versus server side query timestamp.
Hello all, As per my knowledge spring data cassandra (recent version) uses by default cassandra client side query timestamp. I am just curious to know which once is more preferable and recommended to have out of client side or server side query timestamp. Also if any logical reason for the same. Thanks, Mahesh Daksha
Re: [EXTERNAL] Issue while updating a record in 3 node cassandra cluster deployed using kubernetes
Thank you Sean for your response. We are also suspecting the same and analyzing/troubleshooting it around queries associated timestamp. Thanks, Mahesh Daksha On Tue, Apr 9, 2019 at 7:08 PM Durity, Sean R wrote: > My first suspicion would be to look at the server times in the cluster. It > looks like other cases where a write occurs (with no errors) but the data > is not retrieved as expected. If the write occurs with an earlier timestamp > than the existing data, this is the behavior you would see. The write would > occur, but it would not be the latest data to be retrieved. The write looks > like it fails silently, but it actually does exactly what it is designed to > do. > > > > Sean Durity > > > > *From:* Mahesh Daksha > *Sent:* Tuesday, April 09, 2019 9:10 AM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] Issue while updating a record in 3 node cassandra > cluster deployed using kubernetes > > > > Hello All, > > I have a 3 node cassandra cluster with Replication factor as 2 and > read-write consistency set to QUORUM. We are using Spring data cassandra. > All infrastructure is deployed using kubernetes. > > Now in normal use case many records gets inserted to cassandra table. Then > we try to modify/update one of the record using save method of repo, like > below: > > ChunkMeta *tmpRec* = chunkMetaRepository.*save*(chunkMeta); > > After execution of above statement we never see any exception or error. > But still this update state goes silent/fail intermittently. That is at > times the record in the db gets updated successfully where as other time it > fails. Also in the above query when we print *tmpRec* it contains the > updated and correct value every time. Still in the db these updated values > doesn't get reflected. > > We check the the cassandra transport TRACE logs on all nodes and found the > our queries are getting logged there and are being executed also with out > any error or exception. > > Now another weird observation is this all thing works erfectly fine if I > am using single cassandra node (in kubernetes) or if we deploy above infra > using ansible (even works for 3 nodes for Ansible). > > It looks some issue is specifically with the kubernetes 3 node deployment > of cassandra. Primarily looks like replication among nodes causing this. > > Please suggest. > > > > > I have a 3 node cassandra cluster with Replication factor as 2 and > read-write consistency set to QUORUM. We are using Spring data cassandra. > All infrastructure is deployed using kubernetes. > > Now in normal use case many records gets inserted to cassandra table. Then > we try to modify/update one of the record using save method of repo, like > below: > > ChunkMeta tmpRec = chunkMetaRepository.*save*(chunkMeta); > > After execution of above statement we never see any exception or error. > But still this update fail intermittently. That is when we check the record > in the db sometime it gets updated successfully where as other time it > fails. Also in the above query when we print *tmpRec* it contains the > updated and correct value. Still in the db these updated values doesnt get > reflected. > > We check the the cassandra transport TRACE logs on all nodes and found the > our queries are getting logged there and are being executed also. > > Now another weird observation is this all thing works if I am using single > cassandra node (in kubernetes) or if we deploy above infra using ansible > (even works for 3 nodes for Ansible). > > It looks some issue is specifically with the kubernetes 3 node deployment > of cassandra. Primarily looks like replication among nodes causing this. > > Please suggest. > > Below are the contents of my cassandra Docker file: > > FROM ubuntu:16.04 > > > > RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils > net-tools && apt-get clean && \ > > addgroup testuser && useradd -g testuser testuser && usermod --password > testuser testuser; > > > > RUN mkdir -p /opt/test && \ > > mkdir -p /opt/test/data; > > > > ADD jre8.tar.gz /opt/test/ > > ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/ > > > > RUN chmod 755 -R /opt/test/jre && \ > > ln -s /opt/test/jre/bin/java /usr/bin/java && \ > > mv /opt/test/apache-cassandra* /opt/test/cassandra; > > > > RUN mkdir -p /opt/test/cassandra/logs; > > > > ENV JAVA_HOME /opt/test/jre > > RUN export JAVA_HOME > > > > COPY version.txt /opt/test/cassandra/version.txt > > > > WORKDIR /opt/test/cassandra/bin/ > > > > RUN mkdir -p /opt/t
Issue while updating a record in 3 node cassandra cluster deployed using kubernetes
Hello All,
I have a 3 node cassandra cluster with Replication factor as 2 and
read-write consistency set to QUORUM. We are using Spring data cassandra.
All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then
we try to modify/update one of the record using save method of repo, like
below:
ChunkMeta *tmpRec* = chunkMetaRepository.*save*(chunkMeta);
After execution of above statement we never see any exception or error. But
still this update state goes silent/fail intermittently. That is at times
the record in the db gets updated successfully where as other time it
fails. Also in the above query when we print *tmpRec* it contains the
updated and correct value every time. Still in the db these updated values
doesn't get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the
our queries are getting logged there and are being executed also with out
any error or exception.
Now another weird observation is this all thing works erfectly fine if I am
using single cassandra node (in kubernetes) or if we deploy above infra
using ansible (even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment
of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
I have a 3 node cassandra cluster with Replication factor as 2 and
read-write consistency set to QUORUM. We are using Spring data cassandra.
All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then
we try to modify/update one of the record using save method of repo, like
below:
ChunkMeta tmpRec = chunkMetaRepository.*save*(chunkMeta);
After execution of above statement we never see any exception or error. But
still this update fail intermittently. That is when we check the record in
the db sometime it gets updated successfully where as other time it fails.
Also in the above query when we print *tmpRec* it contains the updated and
correct value. Still in the db these updated values doesnt get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the
our queries are getting logged there and are being executed also.
Now another weird observation is this all thing works if I am using single
cassandra node (in kubernetes) or if we deploy above infra using ansible
(even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment
of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
Below are the contents of my cassandra Docker file:
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils
net-tools && apt-get clean && \
addgroup testuser && useradd -g testuser testuser && usermod
--password testuser testuser;
RUN mkdir -p /opt/test && \
mkdir -p /opt/test/data;
ADD jre8.tar.gz /opt/test/
ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
RUN chmod 755 -R /opt/test/jre && \
ln -s /opt/test/jre/bin/java /usr/bin/java && \
mv /opt/test/apache-cassandra* /opt/test/cassandra;
RUN mkdir -p /opt/test/cassandra/logs;
ENV JAVA_HOME /opt/test/jre
RUN export JAVA_HOME
COPY version.txt /opt/test/cassandra/version.txt
WORKDIR /opt/test/cassandra/bin/
RUN mkdir -p /opt/test/data/saved_caches && \
mkdir -p /opt/test/data/commitlog && \
mkdir -p /opt/test/data/hints && \
chown -R testuser:testuser /opt/test/data && \
chown -R testuser:testuser /opt/test;
USER testuser
CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e
's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg;
s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml && rm
../conf/conf.yml && ./cassandra -f
Please note conf.yml is basically cassandra.yml file having properties
related to cassandra.
Thanks,
Mahesh Daksha
Re: ***UNCHECKED*** Query regarding cassandra column write time set by client Timestamp Generator
Thank you Ben and Varun. Will try these approaches. On Tue, Apr 9, 2019 at 3:12 PM Varun Barala wrote: > I'm not sure about the use cases. But other approaches can also be > considered:- > > * Every mutation will have the timestamp in the commitlog [So taking > backup of the commitlogs will give you this functionality] > * At client side, you fetch the existing writetime for those columns from > the db and also log the actual timestamp which is associated with the > current update/insert statements > https://docs.datastax.com/en/drivers/java/3.6/com/datastax/driver/core/Statement.html#getDefaultTimestamp-- > (though this should only be used for debugging purposes!) > > > Thanks! > > On Tue, Apr 9, 2019 at 5:31 PM Ben Slater > wrote: > >> Maybe stabledump can help you? >> https://cassandra.apache.org/doc/4.0/tools/sstable/sstabledump.html >> >> --- >> >> >> *Ben Slater* >> *Chief Product Officer* >> >> >> <https://www.facebook.com/instaclustr> >> <https://twitter.com/instaclustr> >> <https://www.linkedin.com/company/instaclustr> >> >> Read our latest technical blog posts here >> <https://www.instaclustr.com/blog/>. >> >> This email has been sent on behalf of Instaclustr Pty. Limited >> (Australia) and Instaclustr Inc (USA). >> >> This email and any attachments may contain confidential and legally >> privileged information. If you are not the intended recipient, do not copy >> or disclose its content, but please reply to this email immediately and >> highlight the error to the sender and then immediately delete the message. >> >> >> On Tue, 9 Apr 2019 at 19:26, Mahesh Daksha wrote: >> >>> Thanks Ben for your response. >>> WRITETIME gives the information of about the column value already >>> residing int the table. We intend to know the timestamp of the record >>> which is about to apply/update. >>> This is needed to understand the timestamp difference of the data >>> residing in table with the one going to overwite the same. >>> >>> This all information is needed as out update statements going silent >>> (not reflecting any changes) in database. Not even returning any error or >>> exception. >>> >>> Thanks, >>> Mahesh Daksha >>> >>> On Tue, Apr 9, 2019 at 2:46 PM Ben Slater >>> wrote: >>> >>>> Not in the logs but I think you should be able to use the WRITETIME >>>> function to view via CQL (see >>>> https://cassandra.apache.org/doc/latest/cql/dml.html#select) >>>> >>>> Cheers >>>> Ben >>>> >>>> --- >>>> >>>> >>>> *Ben Slater* >>>> *Chief Product Officer* >>>> >>>> >>>> <https://www.facebook.com/instaclustr> >>>> <https://twitter.com/instaclustr> >>>> <https://www.linkedin.com/company/instaclustr> >>>> >>>> Read our latest technical blog posts here >>>> <https://www.instaclustr.com/blog/>. >>>> >>>> This email has been sent on behalf of Instaclustr Pty. Limited >>>> (Australia) and Instaclustr Inc (USA). >>>> >>>> This email and any attachments may contain confidential and legally >>>> privileged information. If you are not the intended recipient, do not copy >>>> or disclose its content, but please reply to this email immediately and >>>> highlight the error to the sender and then immediately delete the message. >>>> >>>> >>>> On Tue, 9 Apr 2019 at 16:51, Mahesh Daksha wrote: >>>> >>>>> Hello, >>>>> >>>>> I have configured the timestamp generator at cassandra client as below: >>>>> >>>>> cluster.setTimestampGenerator(new AtomicMonotonicTimestampGenerator()); >>>>> >>>>> >>>>> My cassandra client inserting and updating few of the rows in a table. >>>>> My query is where in the cassandra debug logs I can see the query >>>>> write time associated by with updated columns in the update query (sent by >>>>> cient). Or if there is any other way I can log the same at client >>>>> itself. >>>>> >>>>> Basically I want to see the write time sent by client to cassandra >>>>> cluster. >>>>> >>>>> Thanks, >>>>> Mahesh Daksha >>>>> >>>>
Re: ***UNCHECKED*** Query regarding cassandra column write time set by client Timestamp Generator
Thanks Ben for your response. WRITETIME gives the information of about the column value already residing int the table. We intend to know the timestamp of the record which is about to apply/update. This is needed to understand the timestamp difference of the data residing in table with the one going to overwite the same. This all information is needed as out update statements going silent (not reflecting any changes) in database. Not even returning any error or exception. Thanks, Mahesh Daksha On Tue, Apr 9, 2019 at 2:46 PM Ben Slater wrote: > Not in the logs but I think you should be able to use the WRITETIME > function to view via CQL (see > https://cassandra.apache.org/doc/latest/cql/dml.html#select) > > Cheers > Ben > > --- > > > *Ben Slater* > *Chief Product Officer* > > > <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> ><https://www.linkedin.com/company/instaclustr> > > Read our latest technical blog posts here > <https://www.instaclustr.com/blog/>. > > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > > > On Tue, 9 Apr 2019 at 16:51, Mahesh Daksha wrote: > >> Hello, >> >> I have configured the timestamp generator at cassandra client as below: >> >> cluster.setTimestampGenerator(new AtomicMonotonicTimestampGenerator()); >> >> My cassandra client inserting and updating few of the rows in a table. >> My query is where in the cassandra debug logs I can see the query write >> time associated by with updated columns in the update query (sent by >> cient). Or if there is any other way I can log the same at client itself. >> >> Basically I want to see the write time sent by client to cassandra >> cluster. >> >> Thanks, >> Mahesh Daksha >> >
***UNCHECKED*** Query regarding cassandra column write time set by client Timestamp Generator
Hello, I have configured the timestamp generator at cassandra client as below: cluster.setTimestampGenerator(new AtomicMonotonicTimestampGenerator()); My cassandra client inserting and updating few of the rows in a table. My query is where in the cassandra debug logs I can see the query write time associated by with updated columns in the update query (sent by cient). Or if there is any other way I can log the same at client itself. Basically I want to see the write time sent by client to cassandra cluster. Thanks, Mahesh Daksha
