can you make it so that the client restarts the connection every 30m or so ? It could be an issue in thrift or something with long-lived connections.
On Dec 15, 2009, at 10:16 AM, Brian Burruss wrote: > i agree. i don't know anything about thrift, and i don't know how it keeps > connections open or manages resources from a client or server perspective, > but this situation suggests that maybe killing the clients is forcing the > server to free something. > > how's that sound :) > > ________________________________________ > From: Jonathan Ellis [[email protected]] > Sent: Monday, December 14, 2009 3:12 PM > To: [email protected] > Subject: Re: read latency creaping up > > hmm, me neither > > but, I can't think how restarting the client would, either :) > > On Mon, Dec 14, 2009 at 4:59 PM, Brian Burruss <[email protected]> wrote: >> Well not sure how that would affect he latency as reported by the Cassandra >> server using nodeprobe cfstats >> >> Jonathan Ellis <[email protected]> wrote: >> >> >> possibly the clients are running into memory pressure? >> >> On Mon, Dec 14, 2009 at 4:27 PM, Brian Burruss <[email protected]> wrote: >>> thx, i'm actually the "B. Todd Burruss" in that thread .. we changed our >>> email system and well now, i'm just Brian .. long story. >>> >>> anyway, in this case it isn't compaction pendings as i can kill the clients >>> and immediately restart and the latency is back to a reasonable number. >>> i'm still investigating. >>> >>> thx! >>> ________________________________________ >>> From: Eric Evans [[email protected]] >>> Sent: Monday, December 14, 2009 8:23 AM >>> To: [email protected] >>> Subject: RE: read latency creaping up >>> >>> On Sun, 2009-12-13 at 13:18 -0800, Brian Burruss wrote: >>>> if this isn't a known issue, lemme do some more investigating. my >>>> test client becomes "more random" with reads as time progresses, so >>>> possibly this is what causes the latency issue. however, all that >>>> being said, the performance really becomes bad after a while. >>> >>> Have a look at the following thread: >>> >>> http://thread.gmane.org/gmane.comp.db.cassandra.user/1402 >>> >>> >>> -- >>> Eric Evans >>> [email protected] >>> >>> >> -- Ian Holsman [email protected]
