Erik,
I just put up a patch with the fix you described and a unit test that
replicates the behavior.
Please test to confirm it works. If so, drop a note in the issue and I
will commit.
Thanks for finding the bug.
JG
Erik Rozendaal wrote:
Issue created: HBASE-1927
On 21 okt 2009, at 23:55, stack wrote:
Thanks for digging in Erik. Nice one. Would you mind making an issue of
your findings? File it against 0.20.2 so we can roll out the fix in next
0.20.x release.
St.Ack
On Wed, Oct 21, 2009 at 2:25 PM, Erik Rozendaal
<[email protected]>wrote:
I did some more investigation into this issue since after the original
issue stop occuring I noticed that MemStoreScanners where still being
leaked
when scanning a store with an empty MemStore.
The cause looks to be the KeyValueHeap constructor. It drops scanners
when
the scanner's peek() method returns null (line 58 of KeyValueHeap).
Unfortunately some scanners (like StoreScanner and MemStoreScanner)
register themselves to some global list when constructed and only
deregister
on close().
So a quick fix may be to add an
} else {
scanner.close()
}
to the KeyValueHeap constructor when the scanner is not added. I'm
not sure
if this is the cleanest fix though...
Regards,
Erik
On 21 okt 2009, at 14:23, Guilherme Germoglio wrote:
Hello Erik,
I think your attachments were blocked. Could you please upload them
somewhere else?
On Wed, Oct 21, 2009 at 8:44 AM, Erik Rozendaal <[email protected]
wrote:
Hi all,
After some performance testing on my HBase 0.20.1 development
environment
(running in pseudo- and full-distributed mode on a single laptop) I
noticed
that scanners do not get closed properly on the region server. After
creating a heap dump with Netbeans I can see the StoreScanner
instances
are
still in the Store.changedReaderObservers collection.
Each StoreScanner instance has the "closed" flag set to false, so it
looks
like the StoreScanner.close() method was never called.
I double-checked my client code and counted the number of times I
create
and close a scanner, and these counts match. I also found this is
repeatable
from the hbase shell. Open the shell, scan some table, take a heap
dump
and
you'll find an unclosed StoreScanner objects in the
Store.changedReaderObservers collection.
I've attached screenshots of the number of StoreScanner instances
(after
30.001 scans) and the Store.changedReaderObservers collection of
one of
the
Stores (notice that the closed field's value is 0 => false).
Ultimately the region server runs out of memory and crashes.
Has anyone experience similar problems?
Regards,
Erik
--
Guilherme
msn: [email protected]
homepage: http://sites.google.com/site/germoglio/
