Hi Lars- We are running ZK 3.3.4, Cloudera cdh3u3, HBase 0.94.16.
Thanks, Ted > On Aug 13, 2014, at 5:36 PM, "lars hofhansl" <[email protected]> wrote: > > Hey Ted, > > so this is a problem with the ZK client, it seems to not clean itself up > correctly upon receiving an exception at the wrong moment. > Which version of ZK are you using? > > > -- Lars > > > > ----- Original Message ----- > From: Ted Tuttle <[email protected]> > To: "[email protected]" <[email protected]> > Cc: Development <[email protected]> > Sent: Wednesday, August 13, 2014 4:38 PM > Subject: HBase client hangs after client-side OOM > > Hello- > > We are running HBase v0.94.16 on an 8 node cluster. > > We have a recurring problem w/ HBase clients hanging. In latest occurrence, > I observed the following sequence of events: > > 0) client plays w/ HBase for a long time w/o issue > 1) client runs out of memory during HBase operation: > > http://pastebin.com/b5x44Lx7 > > 3) Exception is thrown, memory is released > 2) In some shutdown logic the client tries to access HBase again and hangs: > > http://pastebin.com/xU4MSq9k > > Clearly I need to fix OOM. However, the fact that client hangs is not nice. > Any ideas why? > > BTW- I started by looking at zookeeper log. Not much there but here you go: > > http://pastebin.com/wZvE0Fbv > > Thanks, > Ted >
