Re: Preventing SessionExpired events

2009-04-11 Thread Andrew Purtell
encountering. Thanks again, - Andy > From: Mahadev Konar > Subject: Re: Preventing SessionExpired events > Date: Wednesday, April 8, 2009, 1:40 PM > Nitay, > Thanks for sending us the info. > > We have experienced such gc problem in our HDFS (hadoop > file system) setups.

RE: Preventing SessionExpired events

2009-04-08 Thread Benjamin Reed
op.apache.org Subject: Re: Preventing SessionExpired events Nitay is correct about the native threads. Using the pure Java API, the garbage collector will occasionally pause other Java threads to do a full mark and sweep. Even switching to the concurrent collector only delays the problem. The issues is

Re: Preventing SessionExpired events

2009-04-08 Thread Patrick Hunt
I see what you are saying, that's interesting. As Mahadev mentioned on another thread, we'd be interested to look at the JNI you've done. Perhaps we will need to include this as an option (and document where it's necessary, etc...) for other users who might run into this. Thanks much for bring

Re: Preventing SessionExpired events

2009-04-08 Thread Joey Echeverria
Nitay is correct about the native threads. Using the pure Java API, the garbage collector will occasionally pause other Java threads to do a full mark and sweep. Even switching to the concurrent collector only delays the problem. The issues is mixing a high throughput application (HBase) with a low

Re: Preventing SessionExpired events

2009-04-08 Thread Mahadev Konar
Nitay, Thanks for sending us the info. We have experienced such gc problem in our HDFS (hadoop file system) setups. The gc had been quite a problem for us with the Namenode (hadoop hdfs) process. We have seen the namenode just stalling for minutes doing garbage collection. We currently run the nam

Re: Preventing SessionExpired events

2009-04-08 Thread Nitay
The default session timeout in HBase is currently 10 seconds. Bumping it up to 30 and 60 reduced SessionExpired exceptions, according to Andrew. I believe Andrew did run it under jconsole. He was also tuning GC parameters. He mentioned running using incremental garbage collector (-XX:+UseConcMarkSw

Re: Preventing SessionExpired events

2009-04-08 Thread Patrick Hunt
This is good to know. It will allow us to try an replicate the situation, which we haven't been able to do. I'm hoping we can come up with something that we can proactively do to address this... Patrick Nitay wrote: Also, I should mention that some of the errors Andrew was seeing are relate

Re: Preventing SessionExpired events

2009-04-08 Thread Patrick Hunt
What are you running for a session timeout on your clients? Can you run with something like jvisualvm or jconsole, and watch the gc activity when the session timeouts occur? Might give you some insight. Have you tried one of the alternative GC's available in the VM? http://developer.amd.com/doc

Re: Preventing SessionExpired events

2009-04-08 Thread Nitay
Also, I should mention that some of the errors Andrew was seeing are related to ZOOKEEPER-344: I see this kind of stuff: 2009-04-07 17:58:13,344 - WARN [NIOServerCxn.Factory:2181: nioserverc...@417] - Exception causing close of session 0x2208296c38e due to java.io.IOException: Read error a