high level of MemtablePostFlusher pending events

2012-03-13 Thread David Hawthorne
5 node cluster running 1.0.2, doing about 1300 reads and 1300 writes/sec into 3 column families in the same keyspace. 2 client machines, doing about the same amount of reads/writes, but one has an average response time in the 4-40ms range and the other in the 200-800ms range. Both running

Re: Replicate On Write behavior

2011-09-09 Thread David Hawthorne
On Thu, Sep 8, 2011 at 9:48 PM, David Hawthorne dha...@gmx.3crowd.com wrote: It was exactly due to 2890, and the fact that the first replica is always the one with the lowest value IP address. I patched cassandra to pick a random node out of the replica set in StorageProxy.java findSuitableEndpoint

Re: Disable hector stats

2011-09-09 Thread David Hawthorne
I do it with a log4j properties file: log4j.appender.null=org.apache.log4j.varia.NullAppender log4j.category.me.prettyprint.cassandra.hector.TimingLogger=INFO, null log4j.additivity.me.prettyprint.cassandra.hector.TimingLogger=false On Sep 9, 2011, at 2:07 PM, Daning wrote: Hi, How to

Re: Replicate On Write behavior

2011-09-08 Thread David Hawthorne
It was exactly due to 2890, and the fact that the first replica is always the one with the lowest value IP address. I patched cassandra to pick a random node out of the replica set in StorageProxy.java findSuitableEndpoint: Random rng = new Random(); return

Solved: NoSuchMethodError with google guava/collections starting embedded cassandra service

2011-09-06 Thread David Hawthorne
I ran into this problem today. It's common enough that it shows up in google, but not common enough to have a documented resolution, so here's one. [junit] com.google.common.collect.ImmutableSet.copyOf(Ljava/util/Collection;)Lcom/google/common/collect/ImmutableSet; [junit]

Re: Replicate On Write behavior

2011-09-02 Thread David Hawthorne
stayed steady (inserts/sec) with a relatively low average ms/insert. Each row is now significantly shorter as a result of this change. On Sep 2, 2011, at 12:30 AM, Sylvain Lebresne wrote: On Thu, Sep 1, 2011 at 8:52 PM, David Hawthorne dha...@gmx.3crowd.com wrote: I'm curious... digging

JMX TotalReadLatencyMicros sanity check

2011-09-02 Thread David Hawthorne
I've graphed the rate of change of the TotalReadLatencyMicros counter over the last 12 hours, and divided by 1,000,000 to get it in seconds. I'm grabbing it every 10 seconds, so I divided by another 10 to get per-second rates. The result is that I have a CF doing 10 seconds of read *every

Re: Replicate On Write behavior

2011-09-02 Thread David Hawthorne
Does it always pick the node with the lowest IP address? All of my hosts are in the same /24. The fourth node in the 5 node cluster has the lowest value in the 4th octet (54). I erased the cluster and rebuilt it from scratch as a 3 node cluster using the first 3 nodes, and now the

Replicate On Write behavior

2011-09-01 Thread David Hawthorne
I'm curious... digging through the source, it looks like replicate on write triggers a read of the entire row, and not just the columns/supercolumns that are affected by the counter update. Is this the case? It would certainly explain why my inserts/sec decay over time and why the average

Re: nodetool tpstats feature request

2011-08-31 Thread David Hawthorne
watch doesn't calculate diffs On Aug 31, 2011, at 2:29 PM, Peter Sanford wrote: I use `watch` to do this: watch -n 5 nodetool -h localhost tpstats -psanford On Wed, Aug 31, 2011 at 1:59 PM, David Hawthorne dha...@gmx.3crowd.com wrote: It would be very useful to be able to get

randompartitioner cluster unbalanced

2011-08-31 Thread David Hawthorne
$ ./nodetool -h localhost ring Address DC RackStatus State LoadOwns Token 136112946768375385385349842972707284580 10.0.0.57

ReplicateOnWrite issues

2011-07-12 Thread David Hawthorne
This is now my fourth attempt to get the message through. Apologies if you see multiple copies. I've tried to give as much relevant data as I can think of, but please let me know if you need any other info. I spent the day getting jmxtrans to talk to statsd with the cassandra JMX data, so I

Re: ReplicateOnWrite issues

2011-07-12 Thread David Hawthorne
On Mon, Jul 11, 2011 at 11:57 PM, David Hawthorne dha...@gmx.3crowd.com wrote: This is now my fourth attempt to get the message through. Apologies if you see multiple copies. I've tried to give as much relevant data as I can think of, but please let me know if you need any other

Re: ReplicateOnWrite issues

2011-07-12 Thread David Hawthorne
that, but for now you'll have to stagger your writes. I'm more surprised that tasks on the replicate on write stage would stack up at QUORUM, unless you're using a very large number of clients. -- Sylvain On Tue, Jul 12, 2011 at 8:57 AM, David Hawthorne dha...@gmx.3crowd.com wrote

Re: ReplicateOnWrite issues

2011-07-12 Thread David Hawthorne
On Jul 12, 2011, at 3:02 PM, Sylvain Lebresne wrote: On Tue, Jul 12, 2011 at 11:42 PM, David Hawthorne dha...@gmx.3crowd.com wrote: Well, I was using a large number of clients: I tried configuring a hector pool of 20-200 to see what affect that had on throughput. There's definitely

possible 'coming back to life' bug with counters

2011-06-10 Thread David Hawthorne
Please take a look at this thread over in the hector-users mailing list: http://groups.google.com/group/hector-users/browse_thread/thread/99835159b9ea1766 It looks as if the deleted columns are coming back to life when they shouldn't be. I don't want to open a bug on something if it's already

Re: RTG/MRTG/Cricket replacement using Cassandra?

2011-05-23 Thread David Hawthorne
Have you looked at graphite? It would be very cool to see graphite using cassandra as a backend, and then to have cacti feeding data into cassandra. On May 20, 2011, at 4:48 PM, Edward Capriolo wrote: The first love of my open life was cacti. I am going to discuss with them porting some of

Re: RTG/MRTG/Cricket replacement using Cassandra?

2011-03-31 Thread David Hawthorne
I know cloudkick is doing something like this, and we're developing our own in-house method, but it would be nice for there to be a generically-available package that would do this. Lately I've been wishing that someone would take graphite (written in python) and put the frontend on top of

questions about cassandra-1072/1546

2010-12-06 Thread David Hawthorne
Can we get an update? After reading through the comments on 1072, it looks like this is getting close to finished, but it's hard for someone not knee-deep in the project to tell. I'm primarily interested in the timeline you foresee for getting the increment support into trunk for 0.7, and