[
https://issues.apache.org/jira/browse/HBASE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240625#comment-13240625
]
Jesse Yates commented on HBASE-50:
----------------------------------
bq. maybe fully consistent across the whole table isn't necessary
This was just something that has come up in multiple conversations - is full
table, consistent snapshots necessary? The default answer seems to be "yes,
because that is what we get with mysql/postgres/oracle", with the implication
that we _have to have it_ for production HBase. However, its inherently a
different storage model and it may be that we just need to change the
pointy-haired managers' ideas of what's necessary. If instead we can say its is
consistent view, per regionserver, at some point from 2:51:00 to 2:51:30 pm on
friday, then we can save a lot of pain and can be done with _very little
downtime_, to the point where you can mask it in a 'slow request'. If that
isn't acceptable, then the only way I see to do it is to drop write
availability while the snapshot is taking place (fastest region has to to wait
on the slowest region to finish) to get the fully consistent view.
No reason with the current design that we couldn't allow both, it would just be
another flag to pass in while making the snapshot and really only touches about
5 lines of the guts of the implementation.
Currently, we only guarantee row-level consistency. Getting into cross-RS
consistency means we bump up against CAP and have to give up even more
availability (for a fully consistent view, all RS needs to block writes, which
could take as long as the timeout (30sec - 1min in the worst case) or slowest
region - whichever is faster, which in many cases is unacceptable).
However, if you can take point-in-time within a window as acceptable ("sloppy
snapshot") - maybe the window is thirty seconds - when each region blocks
writes for the time each region takes to make the snapshot (max of maybe a few
seconds as no data is being copied, but rather just a few references created
and counts incremented) you keep availability high without really sacrificing
any of the consistency guarantees we currently make.
Clearly, this breaks the multi-put situation where two puts go to different
servers but that is potentially acceptable since we don't make guarantees there
either, just that on return, all of the puts have completed. Same problem as
with doing any of the current backup solutions (replication not included). If
you are worried about cross-RS transactions, you have to use something like
Omid (or similar) to track transactions, and that system can then also decide
when a snapshot is allowable to
ensure all parts of the multi-put complete.
bq. If we're not reusing much of Chongxin's code, we should put discussion into
a new JIRA
A lot of the infrastructure we have has changed (eg. zookeeper utilities,
locking on the RS), but the new features - reference counting, possibly
importing/exporting snapshots, etc - will definitely be reused exactly or only
slightly modified. So at 50/50 on what is kept and what is tossed, at least
right now.
We have also gone through like 3 different stops and starts on this ticket. I
worry moving to a new ticket will cause even worse fragmentation, at least
until the code being used doesn't even resemble Chongxin's :)
> Snapshot of table
> -----------------
>
> Key: HBASE-50
> URL: https://issues.apache.org/jira/browse/HBASE-50
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: Billy Pearson
> Assignee: Li Chongxin
> Priority: Minor
> Labels: gsoc
> Fix For: 0.96.0
>
> Attachments: HBase Snapshot Design Report V2.pdf, HBase Snapshot
> Design Report V3.pdf, HBase Snapshot Implementation Plan.pdf, Snapshot Class
> Diagram.png
>
>
> Havening an option to take a snapshot of a table would be vary useful in
> production.
> What I would like to see this option do is do a merge of all the data into
> one or more files stored in the same folder on the dfs. This way we could
> save data in case of a software bug in hadoop or user code.
> The other advantage would be to be able to export a table to multi locations.
> Say I had a read_only table that must be online. I could take a snapshot of
> it when needed and export it to a separate data center and have it loaded
> there and then i would have it online at multi data centers for load
> balancing and failover.
> I understand that hadoop takes the need out of havening backup to protect
> from failed servers, but this does not protect use from software bugs that
> might delete or alter data in ways we did not plan. We should have a way we
> can roll back a dataset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
