Not a crazy idea at all :) It becomes very tractable if you are willing to allow keys to be munged to the nearest region boundary. The snapshot only considers the HFiles in each region and creates links to those files for the snapshot. So just capturing a subset of regions (as dictated by the 'hint' key ranges) would be reasonable.
We might need a way to differentiate them from normal snapshots, but maybe not - if you supply key ranges, then its on you to know what you are doing with that snapshot. Would you ever want to restore only part of a table? Im not sure that even makes sense.... maybe restoring a chunk at a time? If the latter, then we will likely need to change the restore mechanics to make sure it works (but it may just work out the box, IIRC). we could do the process in batches Would you be willing to manage that your self or would you see this as something HBase would manage for you? ------------------- Jesse Yates @jesse_yates jyates.github.com On Thu, Feb 12, 2015 at 9:18 AM, rahul gidwani <rahul.gidw...@gmail.com> wrote: > Before proposing this idea, I would like to state I have recently had a > through psychiatric evaluation and I'm not crazy. > > We here at flurry land have some very large tables on the order of 1PB, 3PB > with dfs replication. We wanted to ship this table to another cluster > using snapshots. Problem is that the data will take weeks to ship and > during that time major compaction will happen and we will end up with > potentially double the data on our cluster. (We really don't want to turn > off major compaction because we will really suffer with reads). > > Additionally there is one really large CF that dominates this table. So to > mitigate this problem we were thinking that a user could pass in the key > ranges for a snapshot and we could do the process in batches. This might > also be useful for sampling data, or keys which are based on something like > timestamps, where you could archive certain portions of data known to be > stale. > > If people are interested we could get into more details about > implementation. > > Cheers > rahul >
