Hi Eric, Socorro is Java and Python, I was just mentioning it as a possible source of inspiration :) You can learn Ruby and implement it (I hear it is easy... *cough*) or write that same in a small Java app and use it from the command line or so.
And yes, you can range scan using a prefix. We were discussing this recently and there is this notion of design for reads, or design for writes. DFR is usually sequential keys and DFW is random keys. It is tough to find common grounds as both designs are on the far end of the same spectrum. Finding a middle ground is the bucketed (or salted) approach, which gives you distribution but still being able to scan... but not without some client side support. One typical class of data is timeseries based keys. As for scanning them, you need N client side scanners. Imagine this example: row 1 ... 1000 -> Prefix "h1_" row 1001 ... 2000 -> Prefix "h2_" row 2001 ... 3000 -> Prefix "h3_" row 3001 ... 4000 -> Prefix "h4_" row 4001 ... 5000 -> Prefix "h5_" row 5001 ... 6000 -> Prefix "h6_" row 6001 ... 7000 -> Prefix "h7_" So you have divided the entire range into 7 buckets. The prefixes (also sometimes called salt) are used to distribute them row keys to region servers. To scan the entire range as one large key space you need to create 7 scanners: 1. scanner: start row: "h1_", end row "h2_" 2. scanner: start row: "h2_", end row "h3_" 3. scanner: start row: "h3_", end row "h4_" 4. scanner: start row: "h4_", end row "h5_" 5. scanner: start row: "h5_", end row "h6_" 6. scanner: start row: "h6_", end row "h7_" 7. scanner: start row: "h7_", end row "" Now each of them gives you the first row that matches the start and end row keys they are configure for. So you then take that first KV they offer and add it to a list, sorted by ky.getRow() while removing the hash prefix. For example, scanner 1 may have row "h1_1" to offer, then split and drop the prefix "h1_" to get "1". The list then would hold something like: 1. row "1" -> kv from scanner 1 2. row "1010" -> kv from scanner 2 3. row "2001" -> kv from scanner 3 4. row "3033" -> kv from scanner 4 5. row "4001" -> kv from scanner 5 6. row "5002" -> kv from scanner 6 7. row "6000" -> kv from scanner 7 (assuming that the keys are not contiguous but have gaps) You then pop element #1 and do a "scanner1.next()" to get its next KV offering. Then insert that into the list and you get 1. row "3" -> kv from scanner 1 2. row "1010" -> kv from scanner 2 3. row "2001" -> kv from scanner 3 4. row "3033" -> kv from scanner 4 5. row "4001" -> kv from scanner 5 6. row "5002" -> kv from scanner 6 7. row "6000" -> kv from scanner 7 Notice how you always only have a list with N elements on the client side, each representing the next value the scanners offer. Since the list is sorted you always access item #1 and therefore the next in the entire key space. Once scanner 1 runs out you can close and remove it, the list will then give you values from scanner 2 as the first elements in it. And so on. Makes more sense? Lars On Wed, Mar 16, 2011 at 12:09 PM, Eric Charles <[email protected]> wrote: > Hi Lars, > Are you talking about http://code.google.com/p/socorro/ ? > I can find python scripts, but no jruby one... > > Aside the hash function I could reuse, are you saying that range queries are > possible even with hashed keys (randomly distributed)? > (If possible with the script, it will also be possible from the hbase java > client). > Even with your explanation, I can't figure out how compound keys > (hasedkey+key) can be range-queried. > > Tks, > - Eric > > On 16/03/2011 11:38, Lars George wrote: >> >> Hi Eric, >> >> Mozilla Socorro uses an approach where they bucket ranges using >> leading hashes to distribute them across servers. When you want to do >> scans you need to create N scans, where N is the number of hashes and >> then do a next() on each scanner, putting all KVs into one sorted list >> (use the KeyComparator for example) while stripping the prefix hash >> first. You can then access the rows in sorted order where the first >> element in the list is the one with the first key to read. Once you >> took of the first element (being the lowest KV key) you next the >> underlying scanner and reinsert it into the list, reordering it. You >> keep taking from the top and therefore always see the entire range, >> even if the same scanner would return the next logical rows to read. >> >> The shell is written in JRuby, so any function you can use there would >> make sense to use in the prefix, then you could compute it on the fly. >> This will not help with merging the bucketed key ranges, you need to >> do this with the above approach in code. Though since this is JRuby >> you could write that code in Ruby and add it to you local shell giving >> you what you need. >> >> Lars >> >> On Wed, Mar 16, 2011 at 9:01 AM, Eric Charles >> <[email protected]> wrote: >>> >>> Oops, forget my first question about range query (if keys are hashed, >>> they >>> can not be queried based on a range...) >>> Still curious to have info on hash function in shell shell (2.) and >>> advice >>> on md5/jenkins/sha1 (3.) >>> Tks, >>> Eric >>> >>> On 16/03/2011 09:52, Eric Charles wrote: >>>> >>>> Hi, >>>> >>>> To help avoid hotspots, I'm planning to use hashed keys in some tables. >>>> >>>> 1. I wonder if this strategy is adviced for range queries (from/to key) >>>> use case, because the rows will be randomly distributed in different >>>> regions. Will it cause some performance loose? >>>> 2. Is it possible to query from hbase shell with something like "get >>>> 't1', >>>> @hash('r1')", to let the shell compute the hash for you from the >>>> readable >>>> key. >>>> 3. There are MD5 and Jenkins classes in hbase.util package. What would >>>> you >>>> advice? what about SHA1? >>>> >>>> Tks, >>>> - Eric >>>> >>>> PS: I searched the archive but didn't find the answers. >>>> >>> > >
