Hi Murali,

What you see is normal the last keys do indeed overlap. The last key of a region is exclusive and marks the first key of the subsequent region.

Lars

On Nov 7, 2009, at 9:05, "Murali Krishna. P" <[email protected]> wrote:

Hi,
I got it resolved. https://issues.apache.org/jira/browse/HADOOP-5750 was causing this, even though I supplied a custom total ordering partitioner, it didnt use that.


Now the regions looks properly sorted, but facing a new issue. The last key of the each region is not retrievable. The table.jsp page shows the start and end key wrongly.
for eg, take first 2 regions
region1: start : end: 333305184e0f7c3e
region2: start: 333305184e0f7c3e end: 666629fe4378c096

The end key of first region = start key of second ??

If I get the first and last key using HFile.Reader, it shows as follows:

HFileUtil /hbase/test12/98766318/image/9052388247118781160
FirstKey:00000d7d4f36c112imagevalue�������
LastKey:333305184e0f7c3eimagevalue�������

HFileUtil /hbase/test12/170637321/image/7602871928600243730
FirstKey:33338d45cc2491b8imagevalue�������
LastKey:666629fe4378c096imagevalue�������

So, according to this first key of 2nd region is 33338d45cc2491b8 not 333305184e0f7c3e which is correct!

Now when I do a get on 333305184e0f7c3e with debug on, it is loading the second region which is wrong!

Some thing went wrong with the index?

Thanks,
Murali Krishna




________________________________
From: stack <[email protected]>
To: [email protected]
Sent: Sat, 7 November, 2009 6:26:03 AM
Subject: Re: Issue with bulk loader tool

On Fri, Nov 6, 2009 at 12:58 AM, Murali Krishna. P
<[email protected]>wrote:

Hi,
If I increase hbase.hregion.max.filesize so that all the records holds in
one region (and one reducer ), all the records as retrievable. If one
reducer creates multiple hfile or multiple reducer creates one hfile each,
the problem occurs.



Multiple hfiles in a region?  Or are you saying if a reducer creates
multiple regions? There is supposed to be one file per region only when
done.

Thanks for digging in,
St.Ack




Does that give any clue?

Thanks,
Murali Krishna




________________________________
From: Murali Krishna. P <[email protected]>
To: [email protected]
Sent: Thu, 5 November, 2009 6:34:20 PM
Subject: Re: Issue with bulk loader tool

Hi Stack,
Sorry, could not look into this last week...

I got problem with the Htable interface as well. Some records i am not
retrieve from Htable as well.
I lost the old table, but reproduced the problem with a different table.

I cannot send the region since it is very huge. will try to give as much
info as possible here :)

There are total 5 regions as below in that table:
Name

Encoded Name
Start Key
End Key
test1,,1257414794600
106817540
fffe9c7f87c8332a
test1,fffe9c7f87c8332a,1257414794616
1346846599 fffe9c7f87c8332a fffebe279c0ac4d2
test1,fffebe279c0ac4d2,1257414794628
1835851728 fffebe279c0ac4d2 fffec418284d6fbc
test1,fffec418284d6fbc,1257414794637
1078205908 fffec418284d6fbc fffef7a12ea22498
test1,fffef7a12ea22498,1257414794647
1515378663 fffef7a12ea22498

I am looking for a key, say 000011d1bc8cd6fe . This should be in the first
region ?

using hfile tool,
org.apache.hadoop.hbase.io.hfile.HFile -k -f
/hbase/test1/106817540/image/3828859735461759684 -v -m -p |  grep
000011d1bc8cd6fe
The first region doesn't have it. Not sure what happened to that record.

For a working key, it gives the record properly as below
K:
\x00\x100003bdd08ca88ee2\x05imagevalue\x7F\xFF\xFF\xFF\xFF\xFF\xFF \xFF\x04
V: \xFF...

Please let me know if you need more information

Thanks,
Murali Krishna




________________________________
From: stack <[email protected]>
To: [email protected]
Sent: Mon, 2 November, 2009 11:05:43 PM
Subject: Re: Issue with bulk loader tool

Murali:

Any developments worth mentioning?

St.Ack


On Fri, Oct 30, 2009 at 10:14 AM, stack <[email protected]> wrote:

That is interesting. It'd almost point to a shell issue. Enable DEBUG
so
client can see it. Then rerun shell. Is it at least loading the right region? (The regions start and end keys span the asked for key?). I
took a
look at your attached .META. scan. All looks good there. The region specifications look right. If you want to bundle up the region that is failing -- the one that the failing key comes out of, I can take a look
here.  You could also try playing with the HFile tool: ./bin/hbase
org.apache.hadoop.hbase.io.hfile.HFile. Run the former and it'll output usage. You should be able to get it to dump content of the region (You
need
to supply flags like -v to see actual keys to the HFile tool else it just
runs its check silently).    Check for your key.  Check things like
timestamp on it. Maybe its 100 years in advance of now or something?

Yours,
St.Ack


On Fri, Oct 30, 2009 at 9:01 AM, Murali Krishna. P <
[email protected]
wrote:

Attached ".META"

Interesting, I was able to get the row from HTable via java code. But
from
the shell, still getting following

hbase(main):004:0> get 'TestTable2', 'ffffef95bcbf2638'
0 row(s) in 1.2250 seconds

Thanks,
Murali Krishna

Thanks,
Murali Krishna


------------------------------
*From:* stack <[email protected]>
*To:* [email protected]
*Sent:* Fri, 30 October, 2009 8:39:46 PM
*Subject:* Re: Issue with bulk loader tool

Can you send a listing of ".META."?

hbase> scan ".META."

Also, can you bring a region down from hdfs, tar and gzip it, and then
put
it someplace I can pull so I can take a look?

Thanks,
St.Ack


On Fri, Oct 30, 2009 at 3:31 AM, Murali Krishna. P
<[email protected]>wrote:

Hi guys,
I created a table according to hbase-48. A mapreduce job which
creates
HFiles and then used loadtable.rb script to create the table.
Everything
worked fine and i was able to scan the table. But when i do a get for
a
key
displayed in the scan output, it is not retrieving the row. shell says
0
row.

I tried using one reducer to ensure total ordering, but still same
issue.


My mapper is like:
context.write(new
ImmutableBytesWritable(((Text)key).toString().getBytes()), new
KeyValue(((Text)key).toString().getBytes(), "family1".getBytes(),
                  "column1".getBytes(), getValueBytes()));


Please help me investigate this.

Thanks,
Murali Krishna





Reply via email to