I did some more investigation into this issue since after the original
issue stop occuring I noticed that MemStoreScanners where still being
leaked when scanning a store with an empty MemStore.
The cause looks to be the KeyValueHeap constructor. It drops scanners
when the scanner's peek() method returns null (line 58 of KeyValueHeap).
Unfortunately some scanners (like StoreScanner and MemStoreScanner)
register themselves to some global list when constructed and only
deregister on close().
So a quick fix may be to add an
} else {
scanner.close()
}
to the KeyValueHeap constructor when the scanner is not added. I'm not
sure if this is the cleanest fix though...
Regards,
Erik
On 21 okt 2009, at 14:23, Guilherme Germoglio wrote:
Hello Erik,
I think your attachments were blocked. Could you please upload them
somewhere else?
On Wed, Oct 21, 2009 at 8:44 AM, Erik Rozendaal
<[email protected]>wrote:
Hi all,
After some performance testing on my HBase 0.20.1 development
environment
(running in pseudo- and full-distributed mode on a single laptop) I
noticed
that scanners do not get closed properly on the region server. After
creating a heap dump with Netbeans I can see the StoreScanner
instances are
still in the Store.changedReaderObservers collection.
Each StoreScanner instance has the "closed" flag set to false, so
it looks
like the StoreScanner.close() method was never called.
I double-checked my client code and counted the number of times I
create
and close a scanner, and these counts match. I also found this is
repeatable
from the hbase shell. Open the shell, scan some table, take a heap
dump and
you'll find an unclosed StoreScanner objects in the
Store.changedReaderObservers collection.
I've attached screenshots of the number of StoreScanner instances
(after
30.001 scans) and the Store.changedReaderObservers collection of
one of the
Stores (notice that the closed field's value is 0 => false).
Ultimately the region server runs out of memory and crashes.
Has anyone experience similar problems?
Regards,
Erik
--
Guilherme
msn: [email protected]
homepage: http://sites.google.com/site/germoglio/
