[
https://issues.apache.org/jira/browse/HBASE-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436382#comment-13436382
]
Jesse Yates commented on HBASE-6055:
------------------------------------
Looking at implementing concurrent compactions, there is an issue around
allowing compactions and how to get a consistent view of the directory for each
store. If a compaction is taking place and we 'ls' the directory for a store,
the following may occur (or some semantically correct subset of the following):
* get the first set of HFiles in the directory
* compaction removes all the files
* compaction moves in its own files
* we get the next batch of files from the namenode for the original 'ls'
This leads to a munged (not necessarily incorrect) view of the hfiles that will
require another compaction on restore to get a reasonable performance. There
are a couple considerations here.
(1) the above situation occurs only when we have _more files in a store that
the ls limit on the namenode_, which is 1000 by default - the unit of
atomicity. As long as a single store doesn't have more than 1000 files, then we
can just ignore compactions entirely and snap away. However, once we breach
1000 files, this becomes a different, potentially far more complex to reason
about, problem.
(2) We can block for the currently running compactions to finish and then get a
quick 'ls' between compactions starting. This is a bit more intrustive and will
potentialy hold up the compaction queue for a little bit. Also as we have more
files and a more active system it becomes increasingly likely to get a
compaction and cause your snapshot to fail as it waits on the compaction to
finish (since we time-bound snapshots to minimize impact on the system).
Personally, it seems unlikely that we are going to get more than 1000 files in
a single store. However, if its unlikely, that means its probably going to
happen :) Option (2) is far more intrusive and code intensive, potentially
causing some lag in the system, but is sure to be safe once we get it right.
Thoughts?
> Snapshots in HBase 0.96
> -----------------------
>
> Key: HBASE-6055
> URL: https://issues.apache.org/jira/browse/HBASE-6055
> Project: HBase
> Issue Type: New Feature
> Components: client, master, regionserver, zookeeper
> Reporter: Jesse Yates
> Assignee: Jesse Yates
> Fix For: 0.96.0
>
> Attachments: Snapshots in HBase.docx
>
>
> Continuation of HBASE-50 for the current trunk. Since the implementation has
> drastically changed, opening as a new ticket.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
