Re: snapshotting key ranges
thanks for the response guys, I've created a ticket for this issue and we can start to take it from there. https://issues.apache.org/jira/browse/HBASE-13031 maybe we can hash out how to handle the splitting / merging logic and I can start working on some patches. On Thu, Feb 12, 2015 at 9:48 AM, Ted Yu yuzhih...@gmail.com wrote: bq. allow keys to be munged to the nearest region boundary. Interesting idea. If there is region merge, this may get a bit complicated. On Thu, Feb 12, 2015 at 9:26 AM, Jesse Yates jesse.k.ya...@gmail.com wrote: Not a crazy idea at all :) It becomes very tractable if you are willing to allow keys to be munged to the nearest region boundary. The snapshot only considers the HFiles in each region and creates links to those files for the snapshot. So just capturing a subset of regions (as dictated by the 'hint' key ranges) would be reasonable. We might need a way to differentiate them from normal snapshots, but maybe not - if you supply key ranges, then its on you to know what you are doing with that snapshot. Would you ever want to restore only part of a table? Im not sure that even makes sense maybe restoring a chunk at a time? If the latter, then we will likely need to change the restore mechanics to make sure it works (but it may just work out the box, IIRC). we could do the process in batches Would you be willing to manage that your self or would you see this as something HBase would manage for you? --- Jesse Yates @jesse_yates jyates.github.com On Thu, Feb 12, 2015 at 9:18 AM, rahul gidwani rahul.gidw...@gmail.com wrote: Before proposing this idea, I would like to state I have recently had a through psychiatric evaluation and I'm not crazy. We here at flurry land have some very large tables on the order of 1PB, 3PB with dfs replication. We wanted to ship this table to another cluster using snapshots. Problem is that the data will take weeks to ship and during that time major compaction will happen and we will end up with potentially double the data on our cluster. (We really don't want to turn off major compaction because we will really suffer with reads). Additionally there is one really large CF that dominates this table. So to mitigate this problem we were thinking that a user could pass in the key ranges for a snapshot and we could do the process in batches. This might also be useful for sampling data, or keys which are based on something like timestamps, where you could archive certain portions of data known to be stale. If people are interested we could get into more details about implementation. Cheers rahul
Re: snapshotting key ranges
Not a crazy idea at all :) It becomes very tractable if you are willing to allow keys to be munged to the nearest region boundary. The snapshot only considers the HFiles in each region and creates links to those files for the snapshot. So just capturing a subset of regions (as dictated by the 'hint' key ranges) would be reasonable. We might need a way to differentiate them from normal snapshots, but maybe not - if you supply key ranges, then its on you to know what you are doing with that snapshot. Would you ever want to restore only part of a table? Im not sure that even makes sense maybe restoring a chunk at a time? If the latter, then we will likely need to change the restore mechanics to make sure it works (but it may just work out the box, IIRC). we could do the process in batches Would you be willing to manage that your self or would you see this as something HBase would manage for you? --- Jesse Yates @jesse_yates jyates.github.com On Thu, Feb 12, 2015 at 9:18 AM, rahul gidwani rahul.gidw...@gmail.com wrote: Before proposing this idea, I would like to state I have recently had a through psychiatric evaluation and I'm not crazy. We here at flurry land have some very large tables on the order of 1PB, 3PB with dfs replication. We wanted to ship this table to another cluster using snapshots. Problem is that the data will take weeks to ship and during that time major compaction will happen and we will end up with potentially double the data on our cluster. (We really don't want to turn off major compaction because we will really suffer with reads). Additionally there is one really large CF that dominates this table. So to mitigate this problem we were thinking that a user could pass in the key ranges for a snapshot and we could do the process in batches. This might also be useful for sampling data, or keys which are based on something like timestamps, where you could archive certain portions of data known to be stale. If people are interested we could get into more details about implementation. Cheers rahul
Re: snapshotting key ranges
I do not think there is a need for new API. Take a look at TableSnapshotInputFormat which you can customize to to work with key ranges. It allows M/R over snapshots. You make a snapshot of a full table, then you run first batch of keys in M/R job then you delete snapshot and create new one ... repeat until last key range. You will need to control major compaction during this migration. How to output the data - is your choice: TableOutputFormat or HFileOutputFormat2 -Vladimir Rodionov On Thu, Feb 12, 2015 at 9:18 AM, rahul gidwani rahul.gidw...@gmail.com wrote: Before proposing this idea, I would like to state I have recently had a through psychiatric evaluation and I'm not crazy. We here at flurry land have some very large tables on the order of 1PB, 3PB with dfs replication. We wanted to ship this table to another cluster using snapshots. Problem is that the data will take weeks to ship and during that time major compaction will happen and we will end up with potentially double the data on our cluster. (We really don't want to turn off major compaction because we will really suffer with reads). Additionally there is one really large CF that dominates this table. So to mitigate this problem we were thinking that a user could pass in the key ranges for a snapshot and we could do the process in batches. This might also be useful for sampling data, or keys which are based on something like timestamps, where you could archive certain portions of data known to be stale. If people are interested we could get into more details about implementation. Cheers rahul
Re: snapshotting key ranges
bq. allow keys to be munged to the nearest region boundary. Interesting idea. If there is region merge, this may get a bit complicated. On Thu, Feb 12, 2015 at 9:26 AM, Jesse Yates jesse.k.ya...@gmail.com wrote: Not a crazy idea at all :) It becomes very tractable if you are willing to allow keys to be munged to the nearest region boundary. The snapshot only considers the HFiles in each region and creates links to those files for the snapshot. So just capturing a subset of regions (as dictated by the 'hint' key ranges) would be reasonable. We might need a way to differentiate them from normal snapshots, but maybe not - if you supply key ranges, then its on you to know what you are doing with that snapshot. Would you ever want to restore only part of a table? Im not sure that even makes sense maybe restoring a chunk at a time? If the latter, then we will likely need to change the restore mechanics to make sure it works (but it may just work out the box, IIRC). we could do the process in batches Would you be willing to manage that your self or would you see this as something HBase would manage for you? --- Jesse Yates @jesse_yates jyates.github.com On Thu, Feb 12, 2015 at 9:18 AM, rahul gidwani rahul.gidw...@gmail.com wrote: Before proposing this idea, I would like to state I have recently had a through psychiatric evaluation and I'm not crazy. We here at flurry land have some very large tables on the order of 1PB, 3PB with dfs replication. We wanted to ship this table to another cluster using snapshots. Problem is that the data will take weeks to ship and during that time major compaction will happen and we will end up with potentially double the data on our cluster. (We really don't want to turn off major compaction because we will really suffer with reads). Additionally there is one really large CF that dominates this table. So to mitigate this problem we were thinking that a user could pass in the key ranges for a snapshot and we could do the process in batches. This might also be useful for sampling data, or keys which are based on something like timestamps, where you could archive certain portions of data known to be stale. If people are interested we could get into more details about implementation. Cheers rahul
