[
https://issues.apache.org/jira/browse/HBASE-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360084#comment-15360084
]
Anastasia Braginsky commented on HBASE-16156:
---------------------------------------------
Hi Guys,
I have just seen this and HBASE-16162. Both JIRAs have the same root. Indeed
there is a bug in synchronization of dispatching the *single* compaction
thread. This includes that active should be pushed to the compaction pipeline
only once. We have seen this synch problem and solved it like this:
private boolean shouldFlushInMemory() {
if(getActive().getSize() > inmemoryFlushSize) { // size above flush
threshold
if (allowCompaction.get()) // the compaction is allowed in the test
case
// the inMemoryFlushInProgress is CASed to be true here in order to
mutual exclude
// the insert of the active into the compaction pipeline
return (inMemoryFlushInProgress.compareAndSet(false,true)); // the
winner is the only thread who is going to deal with compaction
}
return false;
}
protected void checkActiveSize() {
if (shouldFlushInMemory()) {
/* The thread is dispatched to flush-in-memory. This cannot be done
* on the same thread, because for flush-in-memory we require updatesLock
* in exclusive mode while this method (checkActiveSize) is invoked
holding updatesLock
* in the shared mode. */
InMemoryFlushRunnable runnable = new InMemoryFlushRunnable();
LOG.info("Dispatching the MemStore in-memory flush for store " +
store.getColumnFamilyName());
getPool().execute(runnable);
}
}
// internally used method, externally visible only for tests
// when invoked directly from tests it must be verified that the caller
doesn't hold updatesLock,
// otherwise there is a deadlock
@VisibleForTesting
void flushInMemory() throws IOException {
// Phase I: Update the pipeline
getRegionServices().blockUpdates();
try {
MutableSegment active = getActive();
LOG.info("IN-MEMORY FLUSH: Pushing active segment into compaction
pipeline, " +
"and initiating compaction.");
pushActiveToPipeline(active);
} finally {
getRegionServices().unblockUpdates();
}
// Phase II: Compact the pipeline
try {
if (allowCompaction.get()) {
// setting the inMemoryFlushInProgress flag again for the case this
method is invoked
// directly (only in tests) in the common path setting from true to
true is idempotent
inMemoryFlushInProgress.set(true);
// Speculative compaction execution, may be interrupted if flush is
forced while
// compaction is in progress
compactor.start();
}
} catch (IOException e) {
LOG.warn("Unable to run memstore compaction. region "
+ getRegionServices().getRegionInfo().getRegionNameAsString()
+ "store: "+ getFamilyName(), e);
} finally {
stopCompaction();
}
}
After the fix we didn't see synch issues. This is included in HBASE-14921 patch
that is going to be released tomorrow. But truly we should have done it as
separate fix...
> In write heavy scenario creating in memory flushes leads to contention
> ----------------------------------------------------------------------
>
> Key: HBASE-16156
> URL: https://issues.apache.org/jira/browse/HBASE-16156
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 2.0.0
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
> In write heavy cases, the inmemory flushes blocks the write because it takes
> a update lock. Dumps show that it is causing heavy contention and leads to
> memstore filling up and so intern blocks the flush from happening sooner.
> This JIRA is to discuss optimal settings and then make suitable changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
