I didn't think this was cdh related. We have had another region move to a
different data server ( 1) and performance is back to normal. Could the
locationSeqNum=1 mean the first block on data server 2?
moved to: hostname=cdh5-data2 port=60020 startCode=1398766273743. As of
locationSeqNum=1
Wondering if this is just an Amazon EBS needing to allocate storage on new
disk? Perhaps moving back to 1 has pre-allocated storage to use hence speed
increase.
moved to: hostname=cdh5-data1 port=60020 startCode=1403861243069. As of
locationSeqNum=154640160
Thanks, John.
-----Original Message-----
From: Ted Yu [mailto:[email protected]]
Sent: 27 June 2014 15:46
To: [email protected]
Subject: Re: RegionMovedException: Region moved to Error
Have you posted this on cdh mailing list (I guess you're using cdh5) ?
Cheers
On Fri, Jun 27, 2014 at 7:19 AM, John Cooper <[email protected]>
wrote:
> Ok, so it is just a confirmation the region moved rather than an error.
> Sorry I meant after this error the import performance dropped to 25%
> (import logs show exactly same time as error, now taking 220s compared
> to 50s per 5000 entries). So tried stopping import and restarted
> cluster to see if performance would improve but still 25% of original.
> Each data server is in its own availability zone so wondering if a
> network speed issue causing the delay, however nothing obvious in server
> metric graphs.
>
> Thanks, John.
>
> -----Original Message-----
> From: Ted Yu [mailto:[email protected]]
> Sent: 27 June 2014 14:53
> To: [email protected]
> Cc: [email protected]
> Subject: Re: RegionMovedException: Region moved to Error
>
> This exception meant that the region moved from server 3 to cdh5-data2.
>
> After restart, the import was stuck.
> Did you check region server logs to see where it got stuck ?
>
> Pastebin server log snippet and server stack trace would help.
>
> Cheers
>
> On Jun 27, 2014, at 6:32 AM, John Cooper <[email protected]>
> wrote:
>
> > Hi, I am importing millions of records in to HBase using a custom
> application. We have 3 data servers. The processing time increased 4x
> after getting this error from region server 3 (master controller)
> >
> > 10:36:46.090 AM ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer
> >
> > Closing scanner for
> Prod_AttributeDefinitions,,1398766425236.96fbfbb002634be30c24506056ea80d8.
> > org.apache.hadoop.hbase.exceptions.RegionMovedException: Region
> > moved
> to: hostname=cdh5-data2 port=60020 startCode=1398766273743. As of
> locationSeqNum=1.
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedN
> ame(HRegionServer.java:2584)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionSe
> rver.java:2570)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lea
> seExpired(HRegionServer.java:2377)
> > at
> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> > at java.lang.Thread.run(Thread.java:744)
> >
> > Restarted cluster and import but still stuck at 25%. Error hasn't
> reoccurred. Any ideas on cause or what it means?
> >
> > Thanks, John.
> >
> >
>
>
