Re: ONE has much higher latency than LOCAL_ONE

Yes, as I mentioned in my other thread, LOCAL_ONE does not allow the retry policy to take action if all local nodes are down. Yes, I am using withLocalDc(); Here's the code (Scala): def getClusterBuilder: Builder = { val pool = new PoolingOptions

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed on there, which you might investigate, as well. -- Kind regards, Michael On 03/22/2017 11:21 AM, Daniel Miranda wrote: > Greetings, > > Recently I've

Re: ONE has much higher latency than LOCAL_ONE

I understand all that, but it doesn't explain why the latency increases. The requests are not going to a remote DC. I know this because currently all requests are going to the client in one particular DC. The read request rate of the Cassandra nodes in the other DC remained flat (near zero) the

Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

Greetings, Recently I've started to see the an assertion (traceback follows at the end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all nodes. Repairs run fine without indicating any errors. I can't seem to

Re: ONE has much higher latency than LOCAL_ONE

Yes, your request from the client is going to the LocalDC that you've defined for the data center aware load balancing policy, but with a consistency level of ONE, there is a chance for the coordinator (the node your client has connected to) to route the request across DC's. Please see:

Most used token or partitions

Hi, I am looking for a way to retrieve some statistic about the usage of the data on my Cassandra cluster. Ideally I would like to retrieve a list of the most used token ranges or partitions over the time through JMX or other similar ways. I found that nodetool as the option "toppartition" to

Re: [Cassandra 3.0.9] Cannot allocate memory

Is/are the Cassandra server(s) shared? E.g. do they run mesos + spark? From: Abhishek Kumar Maheshwari Date: Wednesday, March 22, 2017 at 12:45 AM To: "user@cassandra.apache.org" Subject: [Cassandra 3.0.9] Cannot allocate memory

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

Thank you for the pointer Michael, I'll try to investigate if this is the same bug I am seeing. I am afraid it might not be, since I'm observing the error periodically, not just during compactions, and the traceback seems different. Regards, Daniel On Wed, Mar 22, 2017 at 1:27 PM Michael Shuler

Re: [Cassandra 3.0.9] Cannot allocate memory

Just a shot in the dark, but what is your setting of vm.max_map_count in /etc/sysctl.conf ? It is recommended to set it to: vm.max_map_count = 1048575 Source: https://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettingsLinux.html We saw a similar problem in the past where mmap

Re: A Single Dropped Node Fails Entire Read Queries

Upgrading to 3.0.12 solved the issue. Thanks a lot for the help Joel! Shalom Sagges DBA T: +972-74-700-4035 We Create Meaningful Connections

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

I found out the problem is conditioned to having the row cache enabled. Whenever a query would return an empty result set in a particular table, it would fail instead with the exception being thrown in all all nodes. Disabling the row cache for that particular table fixes the problem. Re-enabling

Re: [Cassandra 3.0.9] Cannot allocate memory

On Thu, Mar 23, 2017 at 11:18 AM, Abhishek Kumar Maheshwari < abhishek.maheshw...@timesinternet.in> wrote: > JVM config is as below: > > > > -Xms16G > > -Xmx16G > > -Xmn3000M > > > I don't think it is the cause, but you need to remove Xmn when using G1GC.

RE: [Cassandra 3.0.9] Cannot allocate memory

JVM config is as below: -Xms16G -Xmx16G -Xmn3000M What I need to check in dmesg? From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com] Sent: 23 March 2017 03:39 To: Abhishek Kumar Maheshwari ; user@cassandra.apache.org Subject: RE: [Cassandra 3.0.9]

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

On Wed, Mar 22, 2017 at 4:34 PM, Daniel Miranda wrote: > I found out the problem is conditioned to having the row cache enabled. > Whenever a query would return an empty result set in a particular table, it > would fail instead with the exception being thrown in all all nodes. >

RE: [Cassandra 3.0.9] Cannot allocate memory

The exception is as below: INFO 17:42:37 Index build of til_lineitem_productsku_item_id,til_lineitem_productsku_status complete WARN 17:43:28 G1 Old Generation GC in 2906ms. G1 Eden Space: 1560281088 -> 0; G1 Old Gen: 3033393144 -> 1127339984; G1 Survivor Space : 150994944 -> 0; INFO

RE: [Cassandra 3.0.9] Cannot allocate memory

Hi Abhishek, In sysctl.conf file we have below setting: vm.swappiness = 10 vm.dirty_ratio = 60 vm.dirty_background_ratio = 2 vm.max_map_count = 1048575 so I need to apply patch for same? From: Abhishek Verma [mailto:ve...@uber.com] Sent: 22 March 2017 23:04 To: user@cassandra.apache.org Cc:

RE: [Cassandra 3.0.9] Cannot allocate memory

And what is the configured max heap? Sometimes you may also be able to see some useful messages in "dmesg" output. Jayesh From: Abhishek Kumar Maheshwari Sent: Wednesday, March 22, 2017 5:05:14 PM To: Thakrar, Jayesh;

Altering of types is not allowed

Hi, I’m fairly new to this and am attempting a pretty basic “change column type” example. I’m receiving an error saying InvalidRequest: Error from server: code=2200 [Invalid query] message="Altering of types is not allowed" upon attempting to change any column type. For example, from the

RE: [Cassandra 3.0.9] Cannot allocate memory

No only Cassandra is running on these servers. From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com] Sent: 22 March 2017 22:27 To: Abhishek Kumar Maheshwari ; user@cassandra.apache.org Subject: Re: [Cassandra 3.0.9] Cannot allocate memory Is/are the