Re: ReadStage filling up and leading to Read Timeouts

Thank you Jeff for the link. Please do comment on the G1GC settings,if they are ok for the cluster. Also comment on reducing the concurrent reads to 32 on all nodes in the cluster. As has earlier lead to reads getting dropped. Will adding nodes to the cluster be helpful. Thanks, Rajsekhar Mallick

Re: ReadStage filling up and leading to Read Timeouts

https://docs.datastax.com/en/developer/java-driver/3.2/manual/paging/ -- Jeff Jirsa > On Feb 5, 2019, at 11:33 PM, Rajsekhar Mallick > wrote: > > Hello Jeff, > > Thanks for the reply. > We do have GC logs enabled. > We do observe gc pauses upto 2 seconds but quite often we see this issue

Re: ReadStage filling up and leading to Read Timeouts

Hello Jeff, Thanks for the reply. We do have GC logs enabled. We do observe gc pauses upto 2 seconds but quite often we see this issue even when the gc log reads good and clear. JVM Flags related to G1GC: Xms: 48G Xmx:48G Maxgcpausemillis=200 Parallels gc threads=32 Concurrent gc threads= 10

Re: ReadStage filling up and leading to Read Timeouts

What you're potentially seeing is the GC impact of reading a large partition - do you have GC logs or StatusLogger output indicating you're pausing? What are you actual JVM flags you're using? Given your heap size, the easiest mitigation may be significantly increasing your key cache size (up to

ReadStage filling up and leading to Read Timeouts

Hello Team, Cluster Details: 1. Number of Nodes in cluster : 7 2. Number of CPU cores: 48 3. Swap is enabled on all nodes 4. Memory available on all nodes : 120GB 5. Disk space available : 745GB 6. Cassandra version: 2.1 7. Active tables are using size-tiered compaction strategy 8. Read

Re: Read timeouts when performing rolling restart

ication = {'class': 'SimpleStrategy', >>>>> 'replication_factor': '1'} >>>>> >>>>> system_auth has a replication factor of 1 and even if one node is down >>>>> it may impact the system because of the replication factor. >>>>> >

Re: Read timeouts when performing rolling restart

impact the system because of the replication factor. >>>> >>>> >>>> >>>> On Wed, 12 Sep 2018 at 09:46, Steinmaurer, Thomas < >>>> thomas.steinmau...@dynatrace.com> wrote: >>>> >>>>> Hi, >>>>

Re: Read timeouts when performing rolling restart

;> >>>> >>>> I remember something that a client using the native protocol gets >>>> notified too early by Cassandra being ready due to the following issue: >>>> >>>> https://issues.apache.org/jira/browse/CASSANDRA-8236

Re: Read timeouts when performing rolling restart

using the native protocol gets >>> notified too early by Cassandra being ready due to the following issue: >>> >>> https://issues.apache.org/jira/browse/CASSANDRA-8236 >>> >>> >>> >>> which looks similar, but above was marked as fixed in 2.2.

Re: Read timeouts when performing rolling restart

gt;> https://issues.apache.org/jira/browse/CASSANDRA-8236 >> >> >> >> which looks similar, but above was marked as fixed in 2.2. >> >> >> >> Thomas >> >> >> >> *From:* Riccardo Ferrari >> *Sent:* Mittwoch, 12. September 20

Re: Read timeouts when performing rolling restart

; > > which looks similar, but above was marked as fixed in 2.2. > > > > Thomas > > > > *From:* Riccardo Ferrari > *Sent:* Mittwoch, 12. September 2018 18:25 > *To:* user@cassandra.apache.org > *Subject:* Re: Read timeouts when performing rolling restart

RE: Read timeouts when performing rolling restart

: Mittwoch, 12. September 2018 18:25 To: user@cassandra.apache.org Subject: Re: Read timeouts when performing rolling restart Hi Alain, Thank you for chiming in! I was thinking to perform the 'start_native_transport=false' test as well and indeed the issue is not showing up. Starting the/a node

Re: Read timeouts when performing rolling restart

Hi Alain, Thank you for chiming in! I was thinking to perform the 'start_native_transport=false' test as well and indeed the issue is not showing up. Starting the/a node with native transport disabled and letting it cool down lead to no timeout exceptions no dropped messages, simply a crystal

Re: Read timeouts when performing rolling restart

r 5 nodes is negligible. > The node is generally out of service just for the time of the restart (ie. > cassandra.yml update) > > When the node comes back up and switch on the native transport I start see > lots of read timeouts in our various services: > > com.datasta

Read timeouts when performing rolling restart

for the time of the restart (ie. cassandra.yml update) When the node comes back up and switch on the native transport I start see lots of read timeouts in our various services: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_ONE (1

Re: Read timeouts

Thank you Jeff. We are at Cassandra 3.0.10 Will look forward to upgrade or enable driver logging. > On May 16, 2017, at 11:44 AM, Jeff Jirsa <jji...@apache.org> wrote: > > > > On 2017-05-16 08:53 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote: >>

Re: Read timeouts

On 2017-05-16 08:53 (-0700), Nitan Kainth <ni...@bamlabs.com> wrote: > Hi, > > We see read timeouts intermittently. Mostly after they have occurred. > Timeouts are not consistent and does not occur in 100s at a moment. > > 1. Does read timeout considered as Dropped

Read timeouts

Hi, We see read timeouts intermittently. Mostly after they have occurred. Timeouts are not consistent and does not occur in 100s at a moment. 1. Does read timeout considered as Dropped Mutation? 2. What is best way to nail down exact issue of scattered timeouts? Thank you

Re: Read timeouts on primary key queries

: >> dclocal_read_repair_chance = 0.0 >> Is there any reason to do that or is it just the old (pre 2.0.9) default >> configuration? >> >> The cell count is the number of triplets: (name, value, timestamp) >> >> Also, I see that you have set sstabl

Re: Read timeouts on primary key queries

__ From: Joseph Tech <jaalex.t...@gmail.com> Sent: Wednesday, August 31, 2016 11:16 PM Subject: Re: Read timeouts on primary key queries To: <user@cassandra.apache.org> Patrick, The desc table is below (only col names changed) :  CREATE TABLE db.tbl (    id1 text,

Re: Read timeouts on primary key queries

jaalex.t...@gmail.com> a > écrit : > > > Hi Ryan, > > Attached are the cfhistograms run within few mins of each other. On the > surface, don't see anything which indicates too much skewing (assuming > skewing ==keys spread across many SSTables) . Please confirm. Related

Re: Read timeouts on primary key queries

: Have you looked at cfhistograms/tablehistograms your data maybe just skewed (most likely explanation is probably the correct one here) Regard, Ryan Svihla _ From: Joseph Tech <jaalex.t...@gmail.com> Sent: Wednesday, August 31, 2016 11:16 PM Subject: Re:

Re: Read timeouts on primary key queries

s) . Please confirm. Related to >> this, what does the "cell count" metric indicate ; didn't find a clear >> explanation in the documents. >> >> Thanks, >> Joseph >> >> >> On Thu, Sep 1, 2016 at 6:30 PM, Ryan Svihla <r...@foundev.pro>

Re: Read timeouts on primary key queries

count" metric indicate ; didn't find a clear > explanation in the documents. > > Thanks, > Joseph > > > On Thu, Sep 1, 2016 at 6:30 PM, Ryan Svihla <r...@foundev.pro> wrote: > > Have you looked at cfhistograms/tablehistograms your data maybe just > skewed (

Re: Read timeouts on primary key queries

is probably the correct one here) Regard, Ryan Svihla _ From: Joseph Tech <jaalex.t...@gmail.com> Sent: Wednesday, August 31, 2016 11:16 PM Subject: Re: Read timeouts on primary key queries To: <user@cassandra.apache.org> Patrick, The desc table is below (on

Re: Read timeouts on primary key queries

ou looked at cfhistograms/tablehistograms your data maybe just > skewed (most likely explanation is probably the correct one here) > > Regard, > > Ryan Svihla > > _ > From: Joseph Tech <jaalex.t...@gmail.com> > Sent: Wednesday, August 31, 2016 11:16 PM > Su

Re: Read timeouts on primary key queries

ur data maybe just skewed (most likely explanation is probably the correct one here) Regard, Ryan Svihla _ From: Joseph Tech <jaalex.t...@gmail.com> Sent: Wednesday, August 31, 2016 11:16 PM Subject: Re: Read timeouts on primary key queries To: <user@cassandra.apach

Re: Read timeouts on primary key queries

> Regard, > > Ryan Svihla > > _ > From: Joseph Tech <jaalex.t...@gmail.com> > Sent: Wednesday, August 31, 2016 11:16 PM > Subject: Re: Read timeouts on primary key queries > To: <user@cassandra.apache.org> > > > > Patrick, > > The

Re: Read timeouts on primary key queries

ubject: Re: Read timeouts on primary key queries To: <user@cassandra.apache.org> Patrick, The desc table is below (only col names changed) :  CREATE TABLE db.tbl (    id1 text,    id2 text,    id3 text,    id4 text,    f1 text,    f2 map<text, text>,    f3 map<text, text&g

Re: Read timeouts on primary key queries

Patrick, The desc table is below (only col names changed) : CREATE TABLE db.tbl ( id1 text, id2 text, id3 text, id4 text, f1 text, f2 map, f3 map, created timestamp, updated timestamp, PRIMARY KEY (id1, id2, id3, id4) ) WITH

Re: Read timeouts on primary key queries

If you are getting a timeout on one table, then a mismatch of RF and node count doesn't seem as likely. Time to look at your query. You said it was a 'select * from table where key=?' type query. I would next use the trace facility in cqlsh to investigate further. That's a good way to find hard

Re: Read timeouts on primary key queries

On further analysis, this issue happens only on 1 table in the KS which has the max reads. @Atul, I will look at system health, but didnt see anything standing out from GC logs. (using JDK 1.8_92 with G1GC). @Patrick , could you please elaborate the "mismatch on node count + RF" part. On Tue,

Re: Read timeouts on primary key queries

There could be many reasons for this if it is intermittent. CPU usage + I/O wait status. As read are I/O intensive, your IOPS requirement should be met that time load. Heap issue if CPU is busy for GC only. Network health could be the reason. So better to look system health during that time when

Re: Read timeouts on primary key queries

Hi Patrick, The nodetool status shows all nodes up and normal now. From OpsCenter "Event Log" , there are some nodes reported as being down/up etc. during the timeframe of timeout, but these are Search workload nodes from the remote (non-local) DC. The RF is 3 and there are 9 nodes per DC.

Re: Read timeouts on primary key queries

You aren't achieving quorum on your reads as the error is explains. That means you either have some nodes down or your topology is not matching up. The fact you are using LOCAL_QUORUM might point to a datacenter mis-match on node count + RF. What does your nodetool status look like? Patrick On

Read timeouts on primary key queries

Hi, We recently started getting intermittent timeouts on primary key queries (select * from table where key=) The error is : com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica a

Re: Consistent read timeouts for bursts of reads

Emils, We believe we've tracked it down to the following issue: https://issues.apache.org/jira/browse/CASSANDRA-11302, introduced in 2.1.5. We are running a build of 2.2.5 with that patch and so far have not seen any more timeouts. Mike On Fri, Mar 4, 2016 at 3:14 AM, Emīls Šolmanis

Re: Consistent read timeouts for bursts of reads

Mike, Is that where you've bisected it to having been introduced? I'll see what I can do, but doubt it, since we've long since upgraded prod to 2.2.4 (and stage before that) and the tests I'm running were for a new feature. On Fri, 4 Mar 2016 03:54 Mike Heffner, wrote: >

Re: Consistent read timeouts for bursts of reads

Emils, I realize this may be a big downgrade, but are you timeouts reproducible under Cassandra 2.1.4? Mike On Thu, Feb 25, 2016 at 10:34 AM, Emīls Šolmanis wrote: > Having had a read through the archives, I missed this at first, but this > seems to be *exactly* like

Re: Consistent read timeouts for bursts of reads

We have had similar issues sometimes. Usually the problem was that failing queries where reading the same partition that another query still running and that partition is too big. The fact that is reading the same partition is why your query works upon retry. The fact that the partition (or the

Re: Consistent read timeouts for bursts of reads

Having had a read through the archives, I missed this at first, but this seems to be *exactly* like what we're experiencing. http://www.mail-archive.com/user@cassandra.apache.org/msg46064.html Only difference is we're getting this for reads and using CQL, but the behaviour is identical. On Thu,

Consistent read timeouts for bursts of reads

Hello, We're having a problem with concurrent requests. It seems that whenever we try resolving more than ~ 15 queries at the same time, one or two get a read timeout and then succeed on a retry. We're running Cassandra 2.2.4 accessed via the 2.1.9 Datastax driver on AWS. What we've found while

Read timeouts with ALLOW FILTERING turned on

Hi all, Allow me to rephrase a question I asked last week. I am performing some queries with ALLOW FILTERING and getting consistent read timeouts like the following: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses

Re: Read timeouts with ALLOW FILTERING turned on

On Tue, Aug 5, 2014 at 10:01 AM, Clint Kelly clint.ke...@gmail.com wrote: Allow me to rephrase a question I asked last week. I am performing some queries with ALLOW FILTERING and getting consistent read timeouts like the following: ALLOW FILTERING should be renamed PROBABLY TIMEOUT in order

Re: Read timeouts with ALLOW FILTERING turned on

...@eventbrite.com: On Tue, Aug 5, 2014 at 10:01 AM, Clint Kelly clint.ke...@gmail.com wrote: Allow me to rephrase a question I asked last week. I am performing some queries with ALLOW FILTERING and getting consistent read timeouts like the following: ALLOW FILTERING should be renamed PROBABLY TIMEOUT

Re: Read timeouts with ALLOW FILTERING turned on

consistent read timeouts like the following: ALLOW FILTERING should be renamed PROBABLY TIMEOUT in order to properly describe its typical performance. As a general statement, if you have to ALLOW FILTERING, you are probably Doing It Wrong in terms of schema design. A correctly operated

Re: Read timeouts with ALLOW FILTERING turned on

performing some queries with ALLOW FILTERING and getting consistent read timeouts like the following: ALLOW FILTERING should be renamed PROBABLY TIMEOUT in order to properly describe its typical performance. As a general statement, if you have to ALLOW FILTERING, you are probably Doing It Wrong

Re: Read timeouts with ALLOW FILTERING turned on

On Tue, Aug 5, 2014 at 11:53 AM, Clint Kelly clint.ke...@gmail.com wrote: Ah FWIW I was able to reproduce the problem by reducing range_request_timeout_in_ms. This is great since I want to increase the timeout for batch jobs where we scan a large set of rows, but leave the timeout for

Re: Occasional read timeouts seen during row scans

: Saturday, August 2, 2014 7:04 AM To: user@cassandra.apache.org Subject: Re: Occasional read timeouts seen during row scans Hi Clint, is time correctly synchronized between your nodes? Ciao, Duncan. On 02/08/14 02:12, Clint Kelly wrote: BTW a few other details, sorry for omitting

Re: Occasional read timeouts seen during row scans

occasional read timeouts during multi-row queries, but I'm having difficulty reproducing them or understanding what the problem is. First, some background: Our team wrote a custom MapReduce InputFormat that looks pretty similar to the DataStax InputFormat except that it allows

Re: Occasional read timeouts seen during row scans

To: user@cassandra.apache.org Subject: Re: Occasional read timeouts seen during row scans Hi Clint, is time correctly synchronized between your nodes? Ciao, Duncan. On 02/08/14 02:12, Clint Kelly wrote: BTW a few other details, sorry for omitting these: * We are using version 2.0.4 of the Java

Occasional read timeouts seen during row scans

Hi everyone, I am seeing occasional read timeouts during multi-row queries, but I'm having difficulty reproducing them or understanding what the problem is. First, some background: Our team wrote a custom MapReduce InputFormat that looks pretty similar to the DataStax InputFormat except

Re: Occasional read timeouts seen during row scans

was observing the timeout) Best regards, Clint On Fri, Aug 1, 2014 at 5:02 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I am seeing occasional read timeouts during multi-row queries, but I'm having difficulty reproducing them or understanding what the problem is. First, some

Re: Cassandra 2.0.2 - Frequent Read timeouts and delays in replication on 3-node cluster in AWS VPC

Looks like the read timeouts were a result of a bug that will be fixed in 2.0.3. I found this question on the Datastax Java Driver mailing list: https://groups.google.com/a/lists.datastax.com/forum/#!topic/java-driver-user/ao1ohSLpjRM which led me to: https://issues.apache.org/jira/browse

Re: Cassandra 2.0.2 - Frequent Read timeouts and delays in replication on 3-node cluster in AWS VPC

: Hi all, I am attempting to bring up our new app on a 3-node cluster and am having problems with frequent read timeouts and slow inter-node replication. Initially, these errors were mostly occurring in our app server, affecting 0.02%-1.0% of our queries in an otherwise unloaded cluster

Re: Cassandra 2.0.2 - Frequent Read timeouts and delays in replication on 3-node cluster in AWS VPC

It seems that with NTP properly configured, the replication is now working as expected, but there are still a lot of read timeouts. The troubleshooting continues... On Tue, Nov 19, 2013 at 8:53 AM, Steven A Robenalt srobe...@stanford.eduwrote: Thanks Michael, I will try that out. On Tue

Cassandra 2.0.2 - Frequent Read timeouts and delays in replication on 3-node cluster in AWS VPC

Hi all, I am attempting to bring up our new app on a 3-node cluster and am having problems with frequent read timeouts and slow inter-node replication. Initially, these errors were mostly occurring in our app server, affecting 0.02%-1.0% of our queries in an otherwise unloaded cluster

Increased read timeouts during rolling upgrade to C* 1.2

Hello, I have isolated one of our data centers to simulate a rolling restart upgrade from C* 1.1.10 to 1.2.10. We replayed our production traffic to the C* nodes during the upgrade and observed an increased number of read timeouts during the upgrade process. I executed nodetool drain before

Re: Increased read timeouts during rolling upgrade to C* 1.2

don't think it was a network problem. The number of read timeouts diminished as the number of upgraded nodes increased, until it reached stability. The logs were showing the following messages periodically: ... Two similar issues were reported, but without satisfactory responses: - http

Re: kswapd0 causing read timeouts

On Mon, 18 Jun 2012 11:57:17 -0700, Gurpreet Singh wrote: Thanks for all the information Holger. Will do the jvm updates, kernel updates will be slow to come by. I see that with disk access mode standard, the performance is stable and better than in mmap mode, so i will probably stick to

Re: kswapd0 causing read timeouts

JNA is installed. swappiness was 0. vfs_cache_pressure was 100. 2 questions on this.. 1. Is there a way to find out if mlockall really worked other than just the mlockall successful log message? 2. Does cassandra only mlock the jvm heap or also the mmaped memory? I disabled mmap completely, and

Re: kswapd0 causing read timeouts

Upgrade java (version 1.6.21 have memleaks) to latest 1.6.32. Its abnormally that on 80Gigs you have 15Gigs of index vfs_cache_pressure - used for inodes and dentrys Also to check that you have memleaks use drop_cache sysctl 2012/6/14 Gurpreet Singh gurpreet.si...@gmail.com: JNA is

Re: kswapd0 causing read timeouts

2012/6/14 Gurpreet Singh gurpreet.si...@gmail.com: JNA is installed. swappiness was 0. vfs_cache_pressure was 100. 2 questions on this.. 1. Is there a way to find out if mlockall really worked other than just the mlockall successful log message? yes you must see something like this (from our

Re: kswapd0 causing read timeouts

Soory i mistaken,here is right string INFO [main] 2012-06-14 02:03:14,520 CLibrary.java (line 109) JNA mlockall successful 2012/6/15 ruslan usifov ruslan.usi...@gmail.com: 2012/6/14 Gurpreet Singh gurpreet.si...@gmail.com: JNA is installed. swappiness was 0. vfs_cache_pressure was 100. 2

Re: kswapd0 causing read timeouts

Alright, here it goes again... Even with mmap_index_only, once the RES memory hit 15 gigs, the read latency went berserk. This happens in 12 hours if diskaccessmode is mmap, abt 48 hrs if its mmap_index_only. only reads happening at 50 reads/second row cache size: 730 mb, row cache hit ratio:

Re: kswapd0 causing read timeouts

Hm, it's very strange what amount of you data? You linux kernel version? Java version? PS: i can suggest switch diskaccessmode to standart in you case PS:PS also upgrade you linux to latest, and javahotspot to 1.6.32 (from oracle site) 2012/6/13 Gurpreet Singh gurpreet.si...@gmail.com: Alright,

Re: kswapd0 causing read timeouts

I would check /etc/sysctl.conf and get the values of /proc/sys/vm/swappiness and /proc/sys/vm/vfs_cache_pressure. If you don't have JNA enabled (which Cassandra uses to fadvise) and swappiness is at its default of 60, the Linux kernel will happily swap out your heap for cache space. Set

kswapd0 causing read timeouts

Hi, I am testing cassandra 1.1 on a 1 node cluster. 8 core, 16 gb ram, 6 data disks raid0, no swap configured cassandra 1.1.1 heap size: 8 gigs key cache size in mb: 800 (used only 200mb till now) memtable_total_space_in_mb : 2048 I am running a read workload.. about 30 reads/second. no writes

Re: kswapd0 causing read timeouts

disk_access_mode: mmap?? set to disk_access_mode: mmap_index_only in cassandra yaml 2012/6/8 Gurpreet Singh gurpreet.si...@gmail.com: Hi, I am testing cassandra 1.1 on a 1 node cluster. 8 core, 16 gb ram, 6 data disks raid0, no swap configured cassandra 1.1.1 heap size: 8 gigs key cache

Re: kswapd0 causing read timeouts

Thanks Ruslan. I will try the mmap_index_only. Is there any guideline as to when to leave it to auto and when to use mmap_index_only? /G On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov ruslan.usi...@gmail.comwrote: disk_access_mode: mmap?? set to disk_access_mode: mmap_index_only in cassandra

Re: kswapd0 causing read timeouts

2012/6/8 aaron morton aa...@thelastpickle.com: Ruslan, Why did you suggest changing the disk_access_mode ? Because this bring problems on empty seat, in any case for me mmap bring similar problem and i doesn't have find any solution to resolve it, only change disk_access_mode:-((. For me also

Re: kswapd0 causing read timeouts

Aaron, Ruslan, I changed the disk access mode to mmap_index_only, and it has been stable ever since, well at least for the past 20 hours. Previously, in abt 10-12 hours, as soon as the resident memory was full, the client would start timing out on all its reads. It looks fine for now, i am going

Fwd: read timeouts in cassandra 0.6.5

Hi, I have a simple keyspace: Keyspace Name=reef-test ColumnFamily Name =Meas CompareWith=LongType / ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy ReplicationFactor1/ReplicationFactor