Thanks for digging in Erik. Nice one. Would you mind making an issue of your findings? File it against 0.20.2 so we can roll out the fix in next 0.20.x release. St.Ack
On Wed, Oct 21, 2009 at 2:25 PM, Erik Rozendaal <[email protected]>wrote: > I did some more investigation into this issue since after the original > issue stop occuring I noticed that MemStoreScanners where still being leaked > when scanning a store with an empty MemStore. > > The cause looks to be the KeyValueHeap constructor. It drops scanners when > the scanner's peek() method returns null (line 58 of KeyValueHeap). > > Unfortunately some scanners (like StoreScanner and MemStoreScanner) > register themselves to some global list when constructed and only deregister > on close(). > > So a quick fix may be to add an > > } else { > scanner.close() > } > > to the KeyValueHeap constructor when the scanner is not added. I'm not sure > if this is the cleanest fix though... > > Regards, > Erik > > > On 21 okt 2009, at 14:23, Guilherme Germoglio wrote: > > Hello Erik, >> I think your attachments were blocked. Could you please upload them >> somewhere else? >> >> On Wed, Oct 21, 2009 at 8:44 AM, Erik Rozendaal <[email protected] >> >wrote: >> >> Hi all, >>> >>> After some performance testing on my HBase 0.20.1 development environment >>> (running in pseudo- and full-distributed mode on a single laptop) I >>> noticed >>> that scanners do not get closed properly on the region server. After >>> creating a heap dump with Netbeans I can see the StoreScanner instances >>> are >>> still in the Store.changedReaderObservers collection. >>> >>> Each StoreScanner instance has the "closed" flag set to false, so it >>> looks >>> like the StoreScanner.close() method was never called. >>> >>> I double-checked my client code and counted the number of times I create >>> and close a scanner, and these counts match. I also found this is >>> repeatable >>> from the hbase shell. Open the shell, scan some table, take a heap dump >>> and >>> you'll find an unclosed StoreScanner objects in the >>> Store.changedReaderObservers collection. >>> >>> I've attached screenshots of the number of StoreScanner instances (after >>> 30.001 scans) and the Store.changedReaderObservers collection of one of >>> the >>> Stores (notice that the closed field's value is 0 => false). >>> >>> Ultimately the region server runs out of memory and crashes. >>> >>> Has anyone experience similar problems? >>> >>> Regards, >>> Erik >>> >>> >>> >>> >> >> -- >> Guilherme >> >> msn: [email protected] >> homepage: http://sites.google.com/site/germoglio/ >> > >
