[
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964536#comment-14964536
]
ramkrishna.s.vasudevan commented on HBASE-13082:
------------------------------------------------
We can rename the states to ACTIVE and DISCARDED or may be IN_ACTIVE something
of this sort.
The NON_COMPACTED indicates that it is an ACTIVE file which can be used in
reads. COMPACTED indicates that it is no longer an active file and its
contents are already copied to a new file.
bq.Is that a new Chore per Store per Region? Every two minutes seems like a
long time to hold on to files?
Ya it is a per store per region. But I think we need to pass the region
details for selecting only those files pertaining to that region. Or do you
think it is better to start this per region?
bq.Now in case of the versioned storefile approach change, there is a chance
that there are reference files which are not yet archived after compaction
because of the change in the store file management."
My thinking was that once split is done - we will have reference files still
they are removed by compaction. Now since we don't immediately remove the
compacted files and depend on the cleaner to do it, there is a chance that when
the next split occurs (may be due to user issuing split) there may be
reference files but they are actually in the COMPACTED state (here i mean they
are IN_ACTIVE or DISCARDED state) and we can safely proceed with splits?
bq.Otherwise, flush registers it has happened at the region level. Scanners
check for flush events every-so-often (we already have checkpoints in the scan
to ensure we don't go over size or time constraints... could check at these
times) and when they find one, they swap in the flushed file. When all Scanners
have done this, then we let go the snapshot.
This seems fine but again swapping in the flushed file into the current scan
will need to us to hold the locks?
bq.So how much is the perf gain we get by fully avoiding this lock? Worth of
doing all these complex stuff. I think it is still worth as the gain may be
high.
I was able to get 10-12% latency gain just by using the PE tool with default
load. In that case all the contents were flushed and only pure reads/scans
were issued.
bq.So I think resetting heap is necessary when there is a flush, and the
problem is how to do it without a expensive overhead? Maybe something like a
Biased Locking?
Biased locking - yes. You mean a new type of lock implementaion? By default
JVM works with biased locking enabled or did you mean something else? In
latest processors Re-entrant locking is considered to be much efficient.
http://mechanical-sympathy.blogspot.in/2011/11/java-lock-implementations.html.
> Coarsen StoreScanner locks to RegionScanner
> -------------------------------------------
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Assignee: ramkrishna.s.vasudevan
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt,
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1_WIP.patch,
> HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch, gc.png,
> gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and
> required in the documentation (not picking on Phoenix, this one is my fault
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load.
> RegionScanner operations would keep getting the locks and the
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
