Problems with large partitions and compaction

I have a table that uses LCS and has wound up with partitions upwards of 700 MB. I am seeing lots of the large partition warnings. Client requests are subsequently failing. The driver is not reporting timeout exception, just NoHostAvailableExceptions (in the logs I have reviewed so far). I know

Re: Does Too many GC pauses can cause cassandra service DOWN.

GC can absolutely cause a server to get marked down by a peer. See https://support.datastax.com/hc/en-us/articles/204226199-Common-Causes-of-GC-pauses As for tuning again we use CMS but this thread has some good G1 info that I looked at while evaluating it:

Re: inconsistent results

Change your consistency levels in the cqlsh shell while you query, from ONE to QUORUM to ALL. If you see your results change that's a consistency issue. (Assuming these are simple inserts, if there's deletes, potentially update collections, etc. in the mix then things get a bit more complex.) To

Digest mismatch

I’m running into a situation where I’m seeing a lot of Digest errors in the debug log. I looked at this post: http://stackoverflow.com/questions/39765813/datastax-mismatch-for-key-issue and verified that read_repair_chance is set to 0. We are using DateTieredCompactionStrategy for our time

Re: inconsistent results

I suspect this is true, but it has proven to be significantly harder to track down. Either cassandra is tickling some bug that nothing else does or something strange is going on internally. On an otherwise quiet system, I'd see instant results most of the time intermixed with queries (reads)

Does Too many GC pauses can cause cassandra service DOWN.

On couple of servers when cassandra service is down all i see only GC pauses and load on these servers is not high when pauses happend, i haven't found any thing else is that can be the reason or do i need to look at something else ?? We are using G1GC any reference on how to adjust G1GC

Re: inconsistent results

I'm sorry, yes. The primary key is (foo_prefix, foo), with foo_prefix being the partition key. The query is: select * from table WHERE foo_prefix='blah'; -JE

Re: inconsistent results

Well, if it's the primary key there should only ever be one result. Is this the partition key and you also have a clustering key? On Tue, Feb 14, 2017 at 1:43 PM Josh England wrote: > Super simple: > select * from table WHERE primary_key='foo'; > > -JE > > > On Tue, Feb 14, 2017

Re: inconsistent results

If you're getting a lot of timeouts you will almost certainly end up with consistency issues. You're going to need to fix the root cause, your cluster instability, or this sort of issue will be commonplace. On Tue, Feb 14, 2017 at 1:43 PM Josh England wrote: > I'll try it the

Re: inconsistent results

I'll try it the repair. Using quorum tends to lead to too many timeout problems though. :( -JE On Tue, Feb 14, 2017 at 1:39 PM, Oskar Kjellin wrote: > Repair might help. But you will end up in this situation again unless you > read/write using quorum (may be local)

Re: inconsistent results

Super simple: select * from table WHERE primary_key='foo'; -JE On Tue, Feb 14, 2017 at 1:38 PM, sfesc...@gmail.com wrote: > What is your query? I've seen this once when using secondary indices as it > has to reach out to all nodes for the answer. If a node doesn't respond

Re: inconsistent results

Repair might help. But you will end up in this situation again unless you read/write using quorum (may be local) Sent from my iPhone > On 14 Feb 2017, at 22:37, Josh England wrote: > > All client interactions are from python (python-driver 3.7.1) using default > consistency

Re: inconsistent results

What is your query? I've seen this once when using secondary indices as it has to reach out to all nodes for the answer. If a node doesn't respond in time those records seem to get dropped. On Tue, Feb 14, 2017 at 1:37 PM Josh England wrote: > All client interactions are from

Re: inconsistent results

All client interactions are from python (python-driver 3.7.1) using default consistency (LOCAL_ONE I think). Should I try repairing all nodes to make sure all data is consistent? -JE On Tue, Feb 14, 2017 at 1:32 PM, Oskar Kjellin wrote: > What consistency levels are

Re: inconsistent results

What consistency levels are you using for reads/writes? Sent from my iPhone > On 14 Feb 2017, at 22:27, Josh England wrote: > > I'm running Cassandra 3.9 on CentOS 6.7 in a 6-node cluster. I've got a > situation where the same query sometimes returns 2 records (correct), and

Re: sstableloader making no progress

Adding to the above, each host shows the following log messages that, despite being at INFO level, appear like stack traces to me: 2017-02-13 15:09:22,166 INFO [STREAM-INIT-/10.128.X.Y:60306] StreamResultFuture.java:116 - [Stream #afe548d0-f230-11e6-bc5d-8f99f25bfcf7, ID#0] Received streaming

AbstractQueryPager in debug.log

Hi, I was looking through our logs today, an some thing that caught my eye are many debug logs like this one: DEBUG [SharedPool-Worker-8] 2017-02-14 12:05:39,330 AbstractQueryPager.java:112 - Got result (1) smaller than page size (5000), considering pager exhausted Those line got logged very