Ok, so it is just a confirmation the region moved rather than an error. Sorry I meant after this error the import performance dropped to 25% (import logs show exactly same time as error, now taking 220s compared to 50s per 5000 entries). So tried stopping import and restarted cluster to see if performance would improve but still 25% of original. Each data server is in its own availability zone so wondering if a network speed issue causing the delay, however nothing obvious in server metric graphs.
Thanks, John. -----Original Message----- From: Ted Yu [mailto:[email protected]] Sent: 27 June 2014 14:53 To: [email protected] Cc: [email protected] Subject: Re: RegionMovedException: Region moved to Error This exception meant that the region moved from server 3 to cdh5-data2. After restart, the import was stuck. Did you check region server logs to see where it got stuck ? Pastebin server log snippet and server stack trace would help. Cheers On Jun 27, 2014, at 6:32 AM, John Cooper <[email protected]> wrote: > Hi, I am importing millions of records in to HBase using a custom > application. We have 3 data servers. The processing time increased 4x after > getting this error from region server 3 (master controller) > > 10:36:46.090 AM ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer > > Closing scanner for > Prod_AttributeDefinitions,,1398766425236.96fbfbb002634be30c24506056ea80d8. > org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: > hostname=cdh5-data2 port=60020 startCode=1398766273743. As of > locationSeqNum=1. > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2584) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2570) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:2377) > at > org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122) > at java.lang.Thread.run(Thread.java:744) > > Restarted cluster and import but still stuck at 25%. Error hasn't reoccurred. > Any ideas on cause or what it means? > > Thanks, John. > >
