Thank you J-D. The out file is like this. It has an "NullPointerException" error.
2010-08-24 02:30:14.187::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2010-08-24 02:30:14.187::INFO: jetty-6.1.14 2010-08-24 02:30:14.122::INFO: Started [email protected]:60030 Exception in thread "regionserver/192.168.158.187:60020.leaseChecker" java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.ReadWriteConsistencyControl.getThreadReadPoint(ReadWriteConsistencyControl.java:40) at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:532) at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.seek(MemStore.java:558) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:320) at org.apache.hadoop.hbase.regionserver.StoreScanner.checkReseek(StoreScanner.java:306) at org.apache.hadoop.hbase.regionserver.StoreScanner.peek(StoreScanner.java:143) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:151) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion.java:1971) at org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:1962) at org.apache.hadoop.hbase.Leases.run(Leases.java:98) > Date: Tue, 24 Aug 2010 11:16:34 -0700 > Subject: Re: Region servers down... > From: [email protected] > To: [email protected] > > The last log to look at would be the .out file. > > J-D > > 2010/8/23 xiujin yang <[email protected]>: > > > > Thank you J-D, > > > > I posted today's whole RS log: > > http://pastebin.com/djGnNJxk > > > > GC log: > > http://pastebin.com/AQH5kUCE > > > > I don't see the messages started with "We slept". > > > > > > > > > >> Date: Mon, 23 Aug 2010 23:00:32 -0700 > >> Subject: Re: Region servers down... > >> From: [email protected] > >> To: [email protected] > >> > >> I don't really see the cause of the shutdown in there, it seems it was > >> already under way. Do you see messages starting with "We slept" and > >> then telling how long it slept? It should be not very far from that in > >> the log. > >> > >> J-D > >> > >> 2010/8/23 xiujin yang <[email protected]>: > >> > > >> > Hi, > >> > > >> > RS of HBase was frequently down when running. And job will failed after > >> > the region server down. > >> > > >> > [regionserver/192.168.158.187:60020] 2010-08-24 04:15:14,929 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegion: Closed > >> > whitetable,com.cnet.download:http/Seal-Online/3640-7540_4-10816413.html,1282619615378 > >> > [regionserver/192.168.158.187:60020] 2010-08-24 04:15:14,929 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: telling master that > >> > region server is shutting down at: 192.168.158.187:60020 > >> > [regionserver/192.168.158.187:60020] 2010-08-24 04:15:14,929 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server at: > >> > 192.168.158.187:60020 > >> > [regionserver/192.168.158.187:60020.worker] 2010-08-24 04:15:15,803 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting > >> > [regionserver/192.168.158.187:60020] 2010-08-24 04:15:15,829 INFO > >> > org.apache.zookeeper.ZooKeeper: Session: 0x12a9f2d85010005 closed > >> > [regionserver/192.168.158.187:60020] 2010-08-24 04:15:15,928 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: > >> > regionserver/192.168.158.187:60020 exiting > >> > [Thread-17] 2010-08-24 04:15:16,115 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown > >> > thread. > >> > [Thread-17] 2010-08-24 04:15:16,115 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread > >> > complete > >> > [HCM.shutdownHook] 2010-08-24 04:15:16,115 INFO > >> > org.apache.zookeeper.ZooKeeper: Session: 0x12a9f2d85010041 closed > >> > > >> > > >> > Could anyone help me? > >> > > >> > Here is snippet from the region server log: > >> > http://pastebin.com/YCUDLqc3 > >> > > >> > Version: > >> > HBase: 0.20.5 > >> > Hadoop: 0.20.2 > >> > Zookeeper: 3.3.0 > >> > > >> > > >> > > >
