[
https://issues.apache.org/jira/browse/HBASE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476756#comment-13476756
]
ramkrishna.s.vasudevan commented on HBASE-6980:
-----------------------------------------------
@Kannan
I can try to explain what i see from the code. I am sure you would have
checked but just in case you have not. I may be wrong so pls correct me.
>From the flush code we see that once we do the startFlush() we acquire the
>cacheFlushLock.
The same is cleared in completeCacheFlush(). It is here where we write the
latest seqid corresponding to that flush. Suppose the seq ids 2,3,4 are
getting flushed we write a seq id 5 for the current flush entry.
As the javadoc too says, now suppose a log rolling happens without cacheflush
lock
{code}
synchronized (updateLock) {
// Clean up current writer.
Path oldFile = cleanupCurrentWriter(currentFilenum);
this.writer = nextWriter;
this.hdfs_out = nextHdfsOut;
{code}
The log writer may be changed and the completeCacheflush may write to a new
file (if am not wrong).
Also currently whenever we do flush the oldest seq id for the region is removed
from the lastSeqWritten and the same is again populated back with a sort of
encoded name for the region
{code}
Long oldseq =
lastSeqWritten.put(getSnapshotName(encodedRegionName), seq);
{code}
This was done for a dataloss issue by FB. So may be if we don acquire the
cacheFlushLock on rollWriter, the rollWriter() may see some regions(with
encoded name) which has min seq id and will try to flush them too in
{code}
private byte [] getOldestRegion(final Long oldestOutstandingSeqNum) {
byte [] oldestRegion = null;
for (Map.Entry<byte [], Long> e: this.lastSeqWritten.entrySet()) {
if (e.getValue().longValue() == oldestOutstandingSeqNum.longValue()) {
// Key is encoded region name.
oldestRegion = e.getKey();
break;
}
}
return oldestRegion;
}
{code}
May be other experts can give a better answer if am not right here. Thanks.
> Parallel Flushing Of Memstores
> ------------------------------
>
> Key: HBASE-6980
> URL: https://issues.apache.org/jira/browse/HBASE-6980
> Project: HBase
> Issue Type: New Feature
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
>
> For write dominated workloads, single threaded memstore flushing is an
> unnecessary bottleneck. With a single flusher thread, we are basically not
> setup to take advantage of the aggregate throughput that multi-disk nodes
> provide.
> * For puts with WAL enabled, the bottleneck is more likely the "single" WAL
> per region server. So this particular fix may not buy as much unless we
> unlock that bottleneck with multiple commit logs per region server. (Topic
> for a separate JIRA-- HBASE-6981).
> * But for puts with WAL disabled (e.g., when using HBASE-5783 style fast bulk
> imports), we should be able to support much better ingest rates with parallel
> flushing of memstores.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
