HBASE-4508 is in CDH3u3 (per http://archive.cloudera.com/cdh/3/hbase-0.90.4+49.137.releasenotes.html) so this probably seems to be something else.
On Fri, Apr 27, 2012 at 10:05 PM, Shrijeet Paliwal <[email protected]> wrote: > From top of my head see if you are hitting > https://issues.apache.org/jira/browse/HBASE-4508 (and hence > https://issues.apache.org/jira/browse/HBASE-3777) , 4508 is back port of > 3777 and has a new config which may help you. Of course it depends on if > the new hbase version you are running has 4508 or not. > > On Fri, Apr 27, 2012 at 9:04 AM, Dhaval Shah > <[email protected]>wrote: > >> We have an app written in Ruby which uses HBase as the backing store.. It >> uses >> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro >> until >> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 >> (which is >> the latest stable CDH release at this point) and in a matter of hours all >> Thrift >> servers went down.. >> >> Upon further investigation I realized that it was hitting the limit on the >> number >> of allowed file descriptors (which is pretty high at 32k).. This problem >> occurs >> if >> I use thrift in any configuration (hsha, framed transport, threadpool) >> except >> the >> nonblocking mode.. Digging further I realized a couple of things: >> 1. Even with light load (1-2 processes hitting the thrift server in quick >> succession), thrift is spinning up new threads and each of the threads is >> maintaining a socket connection to zookeeper.. In a matter on minutes >> (with this >> load test), thrift has > 32k open connections with > 8k threads having >> connection >> to zookeeper which do not seem to die even after a day.. >> 2. The logs show approx 3-4 open connections (presumably for each thread): >> java 53588 hbase 4135r FIFO 0,6 177426 pipe >> java 53588 hbase 4136w FIFO 0,6 177426 pipe >> java 53588 hbase 4137r 0000 0,11 0 177427 >> eventpoll >> java 53588 hbase 4138u IPv4 177428 TCP >> njhaddev05:49729->njhaddev01:2181 (ESTABLISH >> ED) >> >> CDH3B4 with the exact same configurations and the exact same setup works >> fine >> but >> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option >> because >> of the low throughput and single threaded nature.. >> >> Can someone help please? >> >> -- Harsh J
