[
https://issues.apache.org/jira/browse/HBASE-428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569184#action_12569184
]
stack commented on HBASE-428:
-----------------------------
Thanks for posting the .META. select Marc.
I've noticed a few things. Here's a region whose start and end key is same:
{code}
2008-02-10 16:18:15,134 DEBUG org.apache.hadoop.hbase.HMaster: Received
MSG_REPORT_OPEN :
pagefetch,http://fun.twilightwap.com/rate.asp?joke_id=183&rating=0 wap2
20080102055026,1202660291003 from 66.135.42.137:60020
2008-02-10 16:18:15,134 DEBUG org.apache.hadoop.hbase.HMaster: Main processing
loop: PendingOpenOperation from 66.135.42.137:60020
2008-02-10 16:18:15,134 INFO org.apache.hadoop.hbase.HMaster:
66.135.42.137:60020 serving
pagefetch,http://fun.twilightwap.com/rate.asp?joke_id=183&rating=0 wap2
20080102055026,1202660291003
2008-02-10 16:18:15,134 INFO org.apache.hadoop.hbase.HMaster: regionname:
pagefetch,http://flirtbox.mobi/new.php?type=html&forum_id=95&topic_index=0 wap2
20071222232620,1202660291003, startKey:
<http://flirtbox.mobi/new.php?type=html&forum_id=95&topic_index=0 wap2
20071222232620>, endKey:
<http://fun.twilightwap.com/rate.asp?joke_id=183&rating=0 wap2 20080102055026>,
encodedName: 1636112728, tableDesc: {name: pagefetch, families:
{changedata:={name: changedata, max versions: 1, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}, data:={name: data, max
versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom
filter: none}, headers:={name: headers, max versions: 1, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, info:={name: info,
max versions: 1, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, redirects:={name: redirects, max versions: 1, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none}}} open on
66.135.42.137:60020
{code}
Here is the region that was split that produced the above:
{code}
2008-02-10 16:17:54,112 INFO org.apache.hadoop.hbase.HMaster: regionname:
pagefetch,http://flirtbox.mobi/new.php?type=html&forum_id=95&topic_index=0 wap2
20071222232620,1202660269165, startKey:
<http://flirtbox.mobi/new.php?type=html&forum_id=95&topic_index=0 wap2
20071222232620>, endKey: <http://go2uwash.com/ wap2 20071222205139>,
encodedName: 7645492, tableDesc: {name: pagefetch, families:
{changedata:={name: changedata, max versions: 1, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}, data:={name: data, max
versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom
filter: none}, headers:={name: headers, max versions: 1, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, info:={name: info,
max versions: 1, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, redirects:={name: redirects, max versions: 1, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none}}} open on
66.135.42.137:60020
{code}
Looks like it has go2uwash as end key. Why doesn't fun.twilightwap.com region
have go2wash as its end key? The row we are trying to insert is
'http://go2purdue.com/Indiana_State_University_Terre_Haute.cfm?pt=2&sp=2&vid=1199235588_3X02X1468516268&rpt=2&kt=5&kp=8
wap2 20080102090745' which would go into this region if go2wash was the end
key.
For good measure, here is the regionserver split report:
{code}
2008-02-10 16:18:12,053 INFO org.apache.hadoop.hbase.HRegionServer: region
split, META updated, and report to master all successful. Old
region=pagefetch,http://flirtbox.mobi/new.php?type=html&forum_id=95&topic_index=0
wap2 20071222232620,1202660269165, new regions:
pagefetch,http://flirtbox.mobi/new.php?type=html&forum_id=95&topic_index=0 wap2
20071222232620,1202660291003,
pagefetch,http://fun.twilightwap.com/rate.asp?joke_id=183&rating=0 wap2
20080102055026,1202660291003. Split took 1sec
{code}
> Under continuous upload of rows, WrongRegionExceptions are thrown that reach
> the client even after retries
> ----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-428
> URL: https://issues.apache.org/jira/browse/HBASE-428
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.1.0, 0.2.0
> Environment: Linux 2.6.9-67.0.1.ELsmp #1 SMP Wed Dec 19 16:01:12 EST
> 2007 i686 athlon i386 GNU/Linux
> Reporter: Marc Harris
> Attachments: filesbysize.csv, lsr, selectfrommeta.txt
>
>
> I have installed 0.16.0 rc 1 which I believe contains a fix for similar issue
> HBASE-138, but I still see the same problem.
> - I am using a single node.
> - The client application runs in a single thread, loading data into a single
> table.
> - I get good throughput of about 200 rows/sec to start with, with occasional
> significant drops due to NotServingRegionException's that are recoverable on
> client retry (internal to hbase).
> - After 54 minutes, and about 500,000 rows I start to see
> WrongRegionException's in the client application, i.e. real failures. (Note
> that this compares to 0.15.3 which would being to throw
> NotServingRegionExceptions after a few tens of thousands of rows).
> My data consists of a single table with 5 column families. The data written
> is as follows:>>
> key: a URL
> family 1: a small string, often emty, 2 longs, 1 int
> family 2: a byte averaging averaging between 1k and 10k, a small string
> family 3: several columns with different names per row, values of small
> strings
> family 4: most rows have zero columns, some rows have 1 or more columns with
> a UL value
> The URLs are typically "long-ish" URL as seen when crawling a site, not short
> home page URLs
>
> I am assuming the data is stored in files of the form
> <hbaseroot>//<tablename>/<9digitnum>/data/mapfiles/<19digitnum>/data. I have
> attached a csv file showing the distribution of size of these files. Average
> size is 19Mb, but the sizes are not evenly distributed at all
> Here are two sample exceptions thrown, copied from the region server log:
> 2008-02-08 02:08:22,495 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 4 on 60020, call
> batchUpdate(pagefetch,http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924,1202401088077, 9223372036854775807, [EMAIL PROTECTED])
> from 66.135.42.137:38484: error:
> org.apache.hadoop.hbase.WrongRegionException: Requested row out of range for
> HRegion
> pagefetch,http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924,1202401088077,
> startKey='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> getEndKey()='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> row='http://go2purdue.com/Redeemer_University.cfm?pt=2&sp=2&vid=1199243289_3X02X1468757255&rpt=2&kt=4&kp=1
> wap2 20080102081237'
> org.apache.hadoop.hbase.WrongRegionException: Requested row out of range for
> HRegion
> pagefetch,http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924,1202401088077,
> startKey='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> getEndKey()='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> row='http://go2purdue.com/Redeemer_University.cfm?pt=2&sp=2&vid=1199243289_3X02X1468757255&rpt=2&kt=4&kp=1
> wap2 20080102081237'
> at org.apache.hadoop.hbase.HRegion.checkRow(HRegion.java:1486)
> at org.apache.hadoop.hbase.HRegion.obtainRowLock(HRegion.java:1531)
> at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1226)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1433)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 2008-02-08 02:08:22,696 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 6 on 60020, call
> batchUpdate(pagefetch,http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924,1202401088077, 9223372036854775807, [EMAIL PROTECTED])
> from 66.135.42.137:38484: error:
> org.apache.hadoop.hbase.WrongRegionException: Requested row out of range for
> HRegion
> pagefetch,http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924,1202401088077,
> startKey='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> getEndKey()='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> row='http://go2umass.com/Travel.cfm?pt=2&sp=2&vid=1199230721_3X04X1485302803&rpt=2&kt=5&kp=8
> wap2 20080102081239'
> org.apache.hadoop.hbase.WrongRegionException: Requested row out of range for
> HRegion
> pagefetch,http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924,1202401088077,
> startKey='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> getEndKey()='http://galsn1.mobilook.mobiwap.com/bm/listproducts;jsessionid=D2ED1EB898163CDB27135DC2CF6958B3.197B?rsi=78011
> wap2 20080102052924',
> row='http://go2umass.com/Travel.cfm?pt=2&sp=2&vid=1199230721_3X04X1485302803&rpt=2&kt=5&kp=8
> wap2 20080102081239'
> at org.apache.hadoop.hbase.HRegion.checkRow(HRegion.java:1486)
> at org.apache.hadoop.hbase.HRegion.obtainRowLock(HRegion.java:1531)
> at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1226)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1433)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
