hbase threw NotServingRegionException

2012-07-23 Thread Ey-Chih chow
Hi, We got a Map/Reduce job that threw NotServingRegionException when the reducer was about to insert data into a Hbase table. The error message is as follows. I also copied the corresponding region server log at the end of the message. Also, we browsed through the hbase administrative page

Re: hbase threw NotServingRegionException

2012-07-23 Thread Ey-Chih chow
Sorry I pasted the wrong portion of the region server log. The right portion should be as follows: == 2012-07-22T00:48:57.147-0700: [GC [ParNew: 18863K-2112K(19136K), 0.0029870 secs] 57106K-42831K(97048K) icms_dc=0 , 0.0030480 secs] [Times: user=0.01

Re: hbase threw NotServingRegionException

2012-07-23 Thread Mohammad Tariq
Hello sir, A possible reason could be, your client is contacting the given regionserver, and that regionserver kept on rejecting the requests. Are you sure your table and all regions online? Use hbck once and see if you find anything interesting. Regards, Mohammad Tariq On Tue, Jul

Re: hbase threw NotServingRegionException

2012-07-23 Thread Ey-Chih chow
Thanks. But if we do a scan on the table via the hbase shell, data in the table did show up. Ey-Chih On Jul 23, 2012, at 1:10 PM, Mohammad Tariq wrote: Hello sir, A possible reason could be, your client is contacting the given regionserver, and that regionserver kept on rejecting

Re: hbase threw NotServingRegionException

2012-07-23 Thread Elliott Clark
hbck should help expose more problems than a single scan would. With that said the logs are the best bet in understanding what is going on with the cluster at the time of the issue. You posted logs that seem to only really contain GC info. Do you have more information about the cluster state

Re: hbase threw NotServingRegionException

2012-07-23 Thread Ey-Chih chow
Besides GC info, the master log is as follows. Ey-Chih === 12/07/22 00:47:18 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is empty in