I don't have the code in front of me, but I bet we're not reusing the Configuration object when getting the HTable handle. I'll take a look at when the regression went in this evening.
-Joey Sent from my iPhone On Apr 27, 2012, at 14:02, Dhaval Y Shah <[email protected]> wrote: > > We have an app written in Ruby which uses HBase as the backing store.. It > uses > Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro > until > now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which > is > the latest stable CDH release at this point) and in a matter of hours all > Thrift > servers went down.. > > Upon further investigation I realized that it was hitting the limit on the > number > of allowed file descriptors (which is pretty high at 32k).. This problem > occurs if > I use thrift in any configuration (hsha, framed transport, threadpool) > except the > nonblocking mode.. Digging further I realized a couple of things: > 1. Even with light load (1-2 processes hitting the thrift server in quick > succession), thrift is spinning up new threads and each of the threads is > maintaining a socket connection to zookeeper.. In a matter on minutes (with > this > load test), thrift has > 32k open connections with > 8k threads having > connection > to zookeeper which do not seem to die even after a day.. > 2. The logs show approx 3-4 open connections (presumably for each thread): > java 53588 hbase 4135r FIFO 0,6 177426 pipe > java 53588 hbase 4136w FIFO 0,6 177426 pipe > java 53588 hbase 4137r 0000 0,11 0 177427 > eventpoll > java 53588 hbase 4138u IPv4 177428 TCP > njhaddev05:49729->njhaddev01:2181 (ESTABLISH > ED) > > CDH3B4 with the exact same configurations and the exact same setup works > fine but > CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option > because > of the low throughput and single threaded nature.. > > Any help is greatly appreciated.. > > -- > View this message in context: > http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html > Sent from the HBase User mailing list archive at Nabble.com. >
