Yes. I have create my own internal scanner using the internalscanner given to me in preCompact On Feb 12, 2013, at 8:44 AM, Anoop Sam John wrote:
> Asaf, > You have created a wrapper around the original InternalScanner > instance created by the compaction flow? > >> Where do the KV generated during the compaction process queue up before >> being written to the disk? Is this buffer configurable? > When I wrote the Region Observer my assumption was the the compaction process > works in Streaming fashion, thus even if I decide to generate a KV per KV I > see, it still shouldn't be a problem memory wise. > > There is no queuing. Your assumption is correct only. It is written to the > writer as and when. (Just like how memstore flush doing the HFile write) As > Lars said a look at your code can tell if some thing is going wrong. Do you > have blooms being used? > > -Anoop- > ________________________________________ > From: Mesika, Asaf [[email protected]] > Sent: Tuesday, February 12, 2013 11:16 AM > To: [email protected] > Subject: Custom preCompact RegionObserver crashes entire cluster on OOME: > Heap Space > > Hi, > > I wrote a RegionObserver which does preCompact. > I activated in pre-production, and then entire cluster dropped dead: One > RegionServer after another crashed on OutOfMemoryException: Heap Space. > > My preCompact method generates a KeyValue per each set of Column Qualifiers > it sees. > When I remove the coprocessor and restart the cluster, cluster remains stable. > I have 8 RS, each has 4 GB Heap. There about 9 regions (from a specific table > I'm working on) per Region Server. > Running HBase 0.94.3 > > The crash occur when the major compaction fires up, apparently cluster wide. > > > My question is this: Where do the KV generated during the compaction process > queue up before being written to the disk? Is this buffer configurable? > When I wrote the Region Observer my assumption was the the compaction process > works in Streaming fashion, thus even if I decide to generate a KV per KV I > see, it still shouldn't be a problem memory wise. > > Of course I'm trying to improve my code so it will generate much less new KV > (by simply altering the existing KVs received from the InternalScanner). > > Thank you, > > Asaf
