Thanks Michael I'm 100% sure its not the UUID distribution that's causing the problem. I'm going to try us the API to create the table and see if that changes things.
The reason I want to pre-split the table is that HBase doesn't handle the initial load to a single regionserver and I can't start the system off slowly and allow a few splits to happen before fully loading it. Its 100% or nothing. I'm also stuck with only 8Gb of RAM per server and only 5 servers so I need to try and get as much as I can from the get go. Simon On 12 June 2012 13:37, Michael Segel <[email protected]> wrote: > Ok, > Now that I'm awake, and am drinking my first cup of joe... > > If you just generate UUIDs you are not going to have an even distribution. > Nor are they going to be truly random due to how the machines are > generating their random numbers. > But this is not important in solving your problem.... > > There is a set of UUIDs which are hashed and then truncated back down to a > 128 bit string. > You can generate the UUID, take a hash (SHA-1 or MD5) and then truncate it > to 128 bits. > This would generate a more random distribution across your splits. > > I'm also a bit curious about why you're pre-splitting in the first place. > I mean I understand why people do it, but its a short term fix and I > wonder how much pain you feel. > > Of course YMMV based on your use case. > > Hash your key and you'll be ok. > > > > On Jun 12, 2012, at 4:41 AM, Simon Kelly wrote: > > > Yes, I'm aware that UUID's are designed to be unique and not evenly > > distributed but I wouldn't expect a big gap in their distribution either. > > > > The other thing that is really confusing me is that the regions splits > > aren't lexicographical sorted. Perhaps there is a problem with the way > I'm > > specifying the splits in the split file. I haven't been able to find any > > docs on what format the splits keys should be in so I've used what's > > produced by Bytes.toStringBinary. Is that correct? > > > > Simon > > > > On 12 June 2012 10:23, Michael Segel <[email protected]> wrote: > > > >> UUIDs are unique but not necessarily random and even in random > samplings, > >> you may not see an even distribution except over time. > >> > >> > >> Sent from my iPhone > >> > >> On Jun 12, 2012, at 3:18 AM, "Simon Kelly" <[email protected]> > wrote: > >> > >>> Hi > >>> > >>> I'm getting some unexpected results with a pre-split table where some > of > >>> the regions are not getting any data. > >>> > >>> The table keys are UUID (generated using Java's UUID.randomUUID() ) > which > >>> I'm storing as a byte[16]: > >>> > >>> key[0-7] = uuid most significant bits > >>> key[8-15] = uuid least significant bits > >>> > >>> The table is created via the shell as follows: > >>> > >>> create 'table', {NAME => 'cf'}, {SPLITS_FILE => 'splits.txt'} > >>> > >>> The splits.txt is generated using the code here: > >>> http://pastebin.com/DAExXMDz which generates 32 regions split between > >> x00 > >>> and xFF. I have also tried with 16 byte regions keys (x00x00... to > >>> xFFxFF...). > >>> > >>> As far as I understand this should distribute the rows evenly across > the > >>> regions but I'm getting a bunch of regions with no rows. I'm also > >> confused > >>> as the the ordering of the regions since it seems the start and end > keys > >>> aren't really matching up correctly. You can see the regions and the > >>> requests they are getting here: http://pastebin.com/B4771g5X > >>> > >>> Thanks in advance for the help. > >>> Simon > >> > >
