Thanks Viral and Jean,
Sounds like timing the compactions is the only options now. Is it ok
we put a large number to *|hbase.hstore.blockingStoreFiles|* so that
even if compaction happens, automatic or manual , blocking rarely happens.
Is compaction time like 20-40 mins normal for a 2GB store .
Regards,
Samar
On 04/03/13 6:20 PM, Jean-Marc Spaggiari wrote:
So can you simply do something like disabling automatic compactions,
and run small manual one after each light job? That way it might not
impact you heavy job with a big 40 minutes compactions since data will
almost alway be compacted correctly? You can even maybe keep the
automatic compaction to on if you do that since when you heavy job
will run, your data compaction will be almost totally done.
JM
2013/3/4 samar.opensource <[email protected]>:
Hi Viral,
The jobs dont run often. may be few times a day(10 ). But the problem is
there are other application which are running other jobs which may not be as
heavy but running compaction any time might block those jobs.
so there is
heavy jobs running less frequently
and light jobs running more frequently
Regards,
Samar
On 04/03/13 12:26 PM, Viral Bajaria wrote:
How often do you run those jobs ? Do they run periodically or are they
running all the time ?
If you have a predictable periodic behavior, you could disable automatic
compaction and trigger it manually using a cron job (not the recommended
approach, AFAIK). Or you could set the compaction to trigger at a set time
of the day when you know your jobs are not running.
-Viral
On Sun, Mar 3, 2013 at 10:44 PM, samar.opensource <
[email protected]> wrote:
Hi,
We are running some high load jobs which are mostly writes. During
these jobs, compaction is triggered which takes sometime as longs as
40mins
to complete. This causes blocking (as others wait for compaction in the
queue). Please suggest how much compaction time is reasonable for
compacting 2Gb store files . And best way to avoid long blocking
compactions.
Using Cloudera hbase vesion 3u3.
Regards,
Samar
