Have you posted this on cdh mailing list (I guess you're using cdh5) ? Cheers
On Fri, Jun 27, 2014 at 7:19 AM, John Cooper <[email protected]> wrote: > Ok, so it is just a confirmation the region moved rather than an error. > Sorry I meant after this error the import performance dropped to 25% > (import logs show exactly same time as error, now taking 220s compared to > 50s per 5000 entries). So tried stopping import and restarted cluster to > see if performance would improve but still 25% of original. Each data > server is in its own availability zone so wondering if a network speed > issue causing the delay, however nothing obvious in server metric graphs. > > Thanks, John. > > -----Original Message----- > From: Ted Yu [mailto:[email protected]] > Sent: 27 June 2014 14:53 > To: [email protected] > Cc: [email protected] > Subject: Re: RegionMovedException: Region moved to Error > > This exception meant that the region moved from server 3 to cdh5-data2. > > After restart, the import was stuck. > Did you check region server logs to see where it got stuck ? > > Pastebin server log snippet and server stack trace would help. > > Cheers > > On Jun 27, 2014, at 6:32 AM, John Cooper <[email protected]> > wrote: > > > Hi, I am importing millions of records in to HBase using a custom > application. We have 3 data servers. The processing time increased 4x after > getting this error from region server 3 (master controller) > > > > 10:36:46.090 AM ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer > > > > Closing scanner for > Prod_AttributeDefinitions,,1398766425236.96fbfbb002634be30c24506056ea80d8. > > org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved > to: hostname=cdh5-data2 port=60020 startCode=1398766273743. As of > locationSeqNum=1. > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2584) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2570) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:2377) > > at > org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122) > > at java.lang.Thread.run(Thread.java:744) > > > > Restarted cluster and import but still stuck at 25%. Error hasn't > reoccurred. Any ideas on cause or what it means? > > > > Thanks, John. > > > > > >
