Debugging high tail read latencies (internal timeout)

2016-07-06 Thread Nimi Wariboko Jr
Hi, I've begun experiencing very high tail latencies across my clusters. While Cassandra's internal metrics report <1ms read latencies, measuring responses from within the driver in my applications (roundtrips of query/execute frames), have 90% round trip times of up to a second for very basic

Re: Cassandra 3.6: Compaction/Write Requests fail with org.xerial.snappy.SnappyError: [NOT_A_DIRECT_BUFFER] destination is not a direct buffer

2016-06-07 Thread Nimi Wariboko Jr
I also found CASSANDRA-9548, "Doing a bit more digging into this, the SnappyCompressor does only support OFF_HEAP buffers.", however I already am using offheap_buffers. I also got this error with offheap_objects. On Tue, Jun 7, 2016 at 4:58 PM, Nimi Wariboko Jr <n...@channelmet

Cassandra 3.6: Compaction/Write Requests fail with org.xerial.snappy.SnappyError: [NOT_A_DIRECT_BUFFER] destination is not a direct buffer

2016-06-07 Thread Nimi Wariboko Jr
Has anyone seen this issue since upgrading to 3.6? I don't have a good idea what is going on yet, as my logs have just been spewing the following error. WARN [SharedPool-Worker-174] 2016-06-07 16:54:07,651 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread

IF NOT EXISTS with multiple static columns confusion

2016-02-23 Thread Nimi Wariboko Jr
I have a table with 2 static columns, and I write to either one of them, if I then write to the other one using IF NOT EXISTS, it fails even though it has never been written too before. Is it the case that all static columns share the same "written too" marker? Given a table like so: CREATE

NullPointerException when trying to compact under 3.2

2016-01-25 Thread Nimi Wariboko Jr
Hi, I recently upgraded from 2.1.12 to 3.2, and one issue I'm having is I can no longer read certain rows from a table. A simple SELECT * FROM `table` times out, only when the bad partition keys are reached. Trying to query the affected partition keys directly also causes a timeout. I think the

Performance tuning for read throughput w/ token scans?

2015-06-19 Thread Nimi Wariboko Jr
[Cassandra 2.1.5] I'm trying to explore my options for increasing read throughput with token scans (SELECT * FROM x WHERE token(y) L AND token(y) L). So far I've started by reading an entire virtual token range from a single node. Currently on a single query I can read about 57,286.03 rows/s

Re: Rebalancing vnodes cluster

2013-09-19 Thread Nimi Wariboko Jr
than the others (which is leading to hot spotting as well). I guess this has to do with the unfinished shuffle? Are there any remedies for this? On Thursday, September 19, 2013 at 9:50 AM, Robert Coli wrote: On Wed, Sep 18, 2013 at 4:26 PM, Nimi Wariboko Jr nimiwaribo...@gmail.com

Rebalancing vnodes cluster

2013-09-18 Thread Nimi Wariboko Jr
Hi, When I started with cassandra I had originally set it up to use tokens. I then migrated to vnodes (using shuffle), but my cluster isn't balanced (http://imgur.com/73eNhJ3). What steps can I take to balance my cluster? Thanks, Nimi

Re: Rebalancing vnodes cluster

2013-09-18 Thread Nimi Wariboko Jr
to display that node on the ring. So a vnode cluster will always appear unbalanced in OpsCenter. Your cluster is probably balanced fine, but 'nodetoo status' should indicate that. Better vnode support is coming up in a future OpsCenter release. On Wed, Sep 18, 2013 at 6:26 PM, Nimi Wariboko

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Nimi Wariboko Jr
How can I recover that data? Can I assume they are still in the sstables? Would doing a sstable2json then reading and reinserting be an optimal solution? On Monday, June 10, 2013 at 9:18 AM, Tyler Hobbs wrote: On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr nimiwaribo...@gmail.com

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Nimi Wariboko Jr
they were before the switches, since the old data is still there. Either way both recovery options are long,painful, and a good amount of manual steps. I would not want to do either one. On Mon, Jun 10, 2013 at 12:39 PM, Nimi Wariboko Jr nimiwaribo...@gmail.com (mailto:nimiwaribo

Re: Data Loss/Missing With Cassandra

2013-06-09 Thread Nimi Wariboko Jr
it). On Sat, Jun 8, 2013 at 8:56 PM, Nimi Wariboko Jr nimiwaribo...@gmail.com (mailto:nimiwaribo...@gmail.com) wrote: Hi, We are seeing an issue where data that was written to the cluster is no longer accessible after trying to expand the size of the cluster. I will try and provide

Data Loss/Missing With Cassandra

2013-06-08 Thread Nimi Wariboko Jr
Hi, We are seeing an issue where data that was written to the cluster is no longer accessible after trying to expand the size of the cluster. I will try and provide as much information as possible, I am just starting at with Cassandra and I'm not entirely sure what data is relevant. All