This doesn't look like a clock skew issue. @Dmitry, while you should be running CMS, this is still a garbage collector and is still vulnerable to GC pauses. There are additional configuration parameters to tune even more.
How much heap are you running with on your RSs? If you are hitting your servers with lots of load you should run with 4GB or more. Also, having ZK on the same servers as RS/DN is going to create problems if you're already hitting your IO limits. JG > -----Original Message----- > From: Arun Ramakrishnan [mailto:[email protected]] > Sent: Wednesday, July 14, 2010 3:33 PM > To: [email protected] > Subject: RE: region servers crashing > > Had a problem that caused issues that looked like this. > > > 2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper: We > slept > > 86246ms, ten times longer than scheduled: 1000 > > Our problem was with clock skew. We just had to make sure ntp was > running on all machines and also the timezones detected on all the > machines were the same. > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Jean- > Daniel Cryans > Sent: Wednesday, July 14, 2010 3:11 PM > To: [email protected] > Subject: Re: region servers crashing > > Dmitry, > > Your log shows this: > > > 2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper: We > slept > > 86246ms, ten times longer than scheduled: 1000 > > This is a pause that lasted more than a minute, the process was in > that state (GC, swapping, mix of all of them) for some reason and it > was long enough to expire the ZooKeeper session (since from its point > of view the region server stopped responding). > > The NPE is just a side-effect, it is caused by the huge pause. > > It's well worth upgrading, but it won't solve your pausing issues. I > can only recommend closer monitoring, setting swappiness to 0 and > giving more memory to HBase (if available). > > J-D > > On Wed, Jul 14, 2010 at 3:03 PM, Dmitry Chechik <[email protected]> > wrote: > > Hi all, > > We've been having issues for a few days with HBase region servers > crashing > > when under load from mapreduce jobs. > > There are a few different errors in the region server logs - I've > attached a > > sample log of 4 different region servers crashing within an hour of > each > > other. > > Some details: > > - This happens when a full table scan from a mapreduce is in > progress. > > - We are running HBase 0.20.3, with a 16-slave cluster, on EC2. > > - Some of the region server errors are NPEs which look a lot > > like https://issues.apache.org/jira/browse/HBASE-2077. I'm not sure > if that > > is the exact problem or if this issue is fixed in 0.20.5. Is it worth > > upgrading to 0.20.5 to fix this? > > - Some of the region server errors are scanner lease expired errors: > > 2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper: We > slept > > 86246ms, ten times longer than scheduled: 1000 > > 2010-07-12 15:10:03,299 WARN org.apache.zookeeper.ClientCnxn: > Exception > > closing session 0x229c72b89360001 to > sun.nio.ch.selectionkeyi...@7f712b3a > > java.io.IOException: TIMED OUT > > at > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906) > > 2010-07-12 15:10:03,299 INFO > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > > 1779060682963568676 lease expired > > 2010-07-12 15:10:03,406 ERROR > > org.apache.hadoop.hbase.regionserver.HRegionServer: > > org.apache.hadoop.hbase.UnknownScannerException: Name: > 1779060682963568676 > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.j > ava:1877) > > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown > Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso > rImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > We tried increasing hbase.regionserver.lease.period to 2 minutes but > that > > didn't seem to make a difference here. > > - Our configuration and table size haven't changed significantly in > those > > days. > > - We're running a 3-node Zookeeper cluster collocated on the same > machines > > as the HBase/Hadoop cluster. > > - Based on Ganglia output, it doesn't look like the regionservers (or > any of > > the machines) are swapping. > > - At the time of the crash, it doesn't appear that the network was > > overloaded (i.e. we've seen higher network traffic without crashes). > So it > > doesn't seem that this is a problem communicating with Zookeeper. > > - We have "-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode" enabled, > so it > > doesn't seem like we should be pausing due to GC too much. > > Any thoughts? > > Thanks, > > - Dmitry
