Hi, There:
While saving lots of data to on hbase, I noticed that the regionserver
CPU went to more than 100%. examination shows that the hbase CompactSplit is
spending full time working on compacting/splitting hbase store files. The
machine I have is an 8 core machine. because there is only one comact/split
thread in hbase, only one core is fully used.
I continue to submit map/reduce job to insert records to hbase. most of
the time, the job runs very fast, around 1-5 minutes. But occasionally, it
can take 2 hours. That is very bad to me. I highly suspect that the
occasional slow insertion is related to the
insufficient speed compactsplit thread.
I am thinking that I should parallize the compactsplit thread, the code
has this : the for loop "for (Store store: stores.values()) " can be
parallized via java 5's threadpool , thus multiple cores are used instead
only one core is used. I wonder if this will help to increase the
throughput.
Somebody mentioned that I can increase the regionsize to that I don't do
so many compaction. Under heavy writing situation.
does anybody have experience showing it helps ?
Jimmy.
byte [] compactStores(final boolean majorCompaction)
throws IOException {
if (this.closing.get() || this.closed.get()) {
LOG.debug("Skipping compaction on " + this + " because
closing/closed");
return null;
}
splitsAndClosesLock.readLock().lock();
try {
byte [] splitRow = null;
if (this.closed.get()) {
return splitRow;
}
try {
synchronized (writestate) {
if (!writestate.compacting && writestate.writesEnabled) {
writestate.compacting = true;
} else {
LOG.info("NOT compacting region " + this +
": compacting=" + writestate.compacting + ", writesEnabled="
+
writestate.writesEnabled);
return splitRow;
}
}
LOG.info("Starting" + (majorCompaction? " major " : " ") +
"compaction on region " + this);
long startTime = System.currentTimeMillis();
doRegionCompactionPrep();
long maxSize = -1;
for (Store store: stores.values()) {
final Store.StoreSize ss = store.compact(majorCompaction);
if (ss != null && ss.getSize() > maxSize) {
maxSize = ss.getSize();
splitRow = ss.getSplitRow();
}
}
doRegionCompactionCleanup();
String timeTaken =
StringUtils.formatTimeDiff(System.currentTimeMillis(),
startTime);
LOG.info("compaction completed on region " + this + " in " +
timeTaken);
} finally {
synchronized (writestate) {
writestate.compacting = false;
writestate.notifyAll();
}
}
return splitRow;
} finally {
splitsAndClosesLock.readLock().unlock();
}
}
