As Jonathan said it is normal , I still need to do more tests to be sure
that I hasn't lost the inserted data.
I paid attention that I got this exceptions during compaction
2010-11-08 18:28:53,663 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction
started. Attempting to free 52292232 bytes
2010-11-08 18:28:53,666 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction
completed. Freed 52309408 bytes. Priority Sizes: Single=57.505157MB
2010-11-08 18:29:22,111 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region
URLs_sanity,20101107_n/gs/34468.html,1289233112752
2010-11-08 18:29:22,141 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compaction size of globalcounters: 65.5m; Skipped 1 file(s), size: 41262334
2010-11-08 18:29:22,141 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Started compaction of 3 file(s) into
/hbase/URLs_sanity/compaction.dir/613365657, seqid=2855632301
2010-11-08 18:29:22,896 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
URLs_sanity,20101107_gallery.mobile9.com/f/1139457/,1289233750914
2010-11-08 18:29:22,897 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
URLs_sanity,20101107_gallery/1139457/,1289233750914
2010-11-08 18:29:22,897 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region URLs_sanity,20101107_bl/1139457/,1289233750914,
encoded=1345660410
My questions are:
1) what happens with insertion process when the job got NSRE exception?
2) Could compaction cause to additional load to the system? and if yes
what is the recommendation regarding
compaction parameters? In my case reducers inserts to hbase almost 3 hours.
3) In addition such behavior I got only in one machine , but I have 5.
Oleg.
On Tue, Nov 9, 2010 at 6:26 AM, Hari Sreekumar <[email protected]>wrote:
> Hi Jonathan,
>
> I have faced the exact same issue as Oleg and it seems to
> happen randomly. Sometimes I get this exception and the task fails,
> sometimes I don't and it succeeds. Do you mean that this is normal
> behavior?
>
> hari
>
> On Tue, Nov 9, 2010 at 12:25 AM, Jonathan Gray <[email protected]> wrote:
>
> > NSRE is normal, this happens when regions move around and your client
> needs
> > to update the location.
> >
> > That seems like an awful lot of mappers/reducers on a 5 server / dual
> core
> > setup... You have only 2 cores per server but you have a DataNode,
> > RegionServer, and 4 map tasks and 3 reduce tasks? That may be over
> > extending things a bit.
> >
> > JG
> >
> > > -----Original Message-----
> > > From: Oleg Ruchovets [mailto:[email protected]]
> > > Sent: Monday, November 08, 2010 7:52 AM
> > > To: [email protected]
> > > Subject: Hbase insertion process cause to region server down.
> > >
> > > Hi ,
> > >
> > > I run map/reduce job which from reducers writes to hbase. I have to
> > > write
> > > 8-9 million raws.
> > >
> > > My environment:
> > > 5 servers dual core , 8G ram.
> > > HDFS -- 9TB
> > >
> > > hadoop has 1000 heap_size ,
> > > hbase has 2000 heap_size
> > > zookeeper has 1000 heap_size
> > > 20 map / 16 reducers while running the job
> > >
> > > Allmost all the time get such exception
> > >
> > > org.apache.hadoop.hbase.NotServingRegionException:
> > > org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
> > > 20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337
> > > closed
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
> > > :1607)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
> > > va:1731)
> > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > > rImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > > at
> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
> > > 5)
> > >
> > >
> > >
> > > Thanks in advance.
> >
>
