[
https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262537#comment-13262537
]
ramkrishna.s.vasudevan edited comment on HBASE-5161 at 4/26/12 11:53 AM:
-------------------------------------------------------------------------
I did not want to open a new issue. Thought i can reopen this issue. Correct
me if am wrong.
I have a high load write operation going on. Region split keeps happening.
When to one of my region the load becomes too heavy the split starts and
creates lot of reference files which is greater than my maxfilestocompact.
So suppose my 'hbase.hstore.compaction.max' is 20 and the reference files that
are created is 32*2 (64 files).
In compaction selection
{code}
if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) {
int pastMax =
compactSelection.getFilesToCompact().size() - this.maxFilesToCompact;
compactSelection.clearSubList(0, pastMax);
}
{code}
The filesToCompact is ordered based on seq id. In this case the set of files
from 0 to pastMax (i.e) the reference files which has lesser seq id are not
considered for compaction. By the time more store files are created and once
again the earlier created ones are avoided. Those being reference files, the
split never happens. The region grew upto 400G.
Note that - We even tried to stop the writes for 2 to 3 hours but still the
compaction did not pick up the reference file.
was (Author: ram_krish):
I did not want to open a new issue. Thought i can reopen this issue.
Correct me if am wrong.
I have a high load write operation going on. Region split keeps happening.
When to one of my region the load becomes too heavy the split starts and
creates lot of reference files which is greater than my maxfilestocompact.
So suppose my 'hbase.hstore.compaction.max' is 20 and the reference files that
are created is 32*2 (64 files).
In compaction selection
{code}
if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) {
int pastMax =
compactSelection.getFilesToCompact().size() - this.maxFilesToCompact;
compactSelection.clearSubList(0, pastMax);
}
{code}
The filesToCompact is ordered based on seq id. In this case the set of files
from 0 to pastMax (i.e) the reference files which has lesser seq id are not
considered for compaction. By the time more store files are created and once
again the earlier created ones are avoided. Those being reference files, the
split never happens. The region grew upto 400G.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the
> insert load was coming in. Emergency flushes because of the low memory
> barrier don't check the number of store files so it never blocks, to a point
> where I had in one case 45 store files and the compactions were almost never
> done on the reference files (had 15 of them, went down by one in 20 minutes).
> Since you can't split regions with reference files, that region couldn't
> split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback
> mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
