Hi there-
The last part of 6.3.2.3 is important: "Expect tradeoffs
when designing rowkeys". Some of this stuff you just have to
prototype.
In terms of performance...
http://hbase.apache.org/book.html#keyvalue
.. if you have huge keys, you'll feel it there.
For the MR part, see...
http://hbase.apache.org/book.html#mapreduce.example.readwrite
On 11/21/11 1:46 PM, "Mark" <[email protected]> wrote:
>I was actually referring to the HBase book that states:
>
>
> 6.3.2.3. Rowkey Length
>
>Keep them as short as is reasonable such that they can still be useful
>for required data access
>
>
>Didn't know if adding 16 bytes to each row key would have any
>performance implications.
>
>As a side question having, what would be the best way of converting
>existing sequential keys to a hashed version? M/R script? Is there a
>simple dump?
>
>
>On 11/21/11 8:18 AM, Nicolas Spiegelberg wrote:
>> Mark: you are correct about the old_key suffix. I'm assuming that
>>you're
>> worried about this because of keyspace size, correct? The default
>> algorithm for pre-splitting assumes a 32-bit (4 byte) hash prefix, which
>> should be perfectly scalable for all use cases in the near future of
>> computing. Really, you could get away with an 8-bit hash prefix if your
>> cluster is small& you plan to auto-split after a certain size. This is
>> available if you use UniformSplit but will require a little power user
>> investigation. I don't think anybody deviates from the default, mainly
>> just because current use cases aren't as finicky about the extra
>>overhead.
>>
>> For the medium term, note that HBASE-4218 will also introduce key
>> compression& further reduce overhead. This won't be available until 94
>> or so, but you probably won't be worried about an extra 4 bytes until
>> then. We currently use the HexStringSplit algorithm in production,
>>which
>> is 8-bytes but is human-readable. With preliminary investigation, we
>> predict an 80%+ compression in our key size (currently ~80 bytes) with
>> HBASE-4218.
>>
>> On 11/21/11 9:55 AM, "Mark"<[email protected]> wrote:
>>
>>> Damn, I was hoping my understanding was flawed.
>>>
>>> In your example I am guessing the addition of old_key suffix is to
>>> prevent against any possible collision. Is that correct?
>>>
>>> On 11/20/11 9:39 PM, Nicolas Spiegelberg wrote:
>>>> Sequential writes are also an argument for pre-splitting and using
>>>>hash
>>>> prefixing. In other words, presplit your table into N regions instead
>>>> of
>>>> the default of 1& transform your keys into:
>>>>
>>>> new_key = md5(old_key) + old_key
>>>>
>>>> Using this method your sequential writes under the old_key are now
>>>> spread
>>>> evenly across all regions. There are some limitations to hash
>>>> prefixing,
>>>> such as non-sequential scans across row boundaries. However, it's a
>>>> tradeoff between even distribution& advanced query options.
>>>>
>>>> On 11/20/11 7:54 PM, "Amandeep Khurana"<[email protected]> wrote:
>>>>
>>>>> Mark,
>>>>>
>>>>> Yes, your understanding is correct. If your keys are sequential
>>>>> (timestamps
>>>>> etc), you will always be writing to the end of the table and "older"
>>>>> regions will not get any writes. This is one of the arguments against
>>>>> using
>>>>> sequential keys.
>>>>>
>>>>> -ak
>>>>>
>>>>> On Sun, Nov 20, 2011 at 11:33 AM, Mark<[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Say we have a use case that has sequential row keys and we have rows
>>>>>> 0-100. Let's assume that 100 rows = the split size. Now when there
>>>>>>is
>>>>>> a
>>>>>> split it will split at the halfway mark so there will be two regions
>>>>>> as
>>>>>> follows:
>>>>>>
>>>>>> Region1 [START-49]
>>>>>> Region2 [50-END]
>>>>>>
>>>>>> So now at this point all inserts will be writing to Region2 only
>>>>>> correct?
>>>>>> Now at some point Region2 will need to split and it will look like
>>>>>>the
>>>>>> following before the split:
>>>>>>
>>>>>> Region1 [START-49]
>>>>>> Region2 [50-150]
>>>>>>
>>>>>> After the split it will look like:
>>>>>>
>>>>>> Region1 [START-49]
>>>>>> Region2 [50-100]
>>>>>> Region3 [150-END]
>>>>>>
>>>>>> And this pattern will continue correct? My question is when there
>>>>>>is a
>>>>>> use
>>>>>> case that has sequential keys how would any of the older regions
>>>>>>every
>>>>>> receive anymore writes? It seems like they would always be stuck at
>>>>>> MaxRegionSize/2. Can someone please confirm or clarify this issue?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
