Looking in RS log I see this:
#
2010-08-26 13:34:11,056 WARN
org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 36 on
60020 took 148265ms appending an edit to hlog; editcount=2990
#
2010-08-26 13:34:11,056 WARN
org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 86 on
60020 took 148265ms appending an edit to hlog; editcount=2991
#
That it took near 3 minutes appending the log is probably because we
were under a stop-the-world GC pause.
> Regarding CMSInitiatingOccupancyFraction, we used to have it set to 88% and
> we thought that by setting it lower we'd kick off more frequent (but
> smaller) GC collections, and reduce the chance of any one of them pausing.
>
Isn't 88% almost the default? Come down more I'd say. I see earlier
you ran with 50% CMSInitiatingOccupancyFraction? That didn't work?
Starting earlier, you'll pay in CPU but should help. Set
UseCMSInitiatingOccupancyOnly too so the GC will consider your
CMSInitiatingOccupancyFraction setting only before it starts the CMS
full GC [1].
How many cores do you have? If <= 4, you should add -XX:+CMSIncrementalMode.
Update your JVM. u21 has some fixes to help with heap fragmentation (
-XX:+DoEscapeAnalysis probably won't work on u21, IIRC).
I just noticed you are running on EC2. Use bigger nodes?
You are also running old hbase which had a particular way of scanning
in a manner that was RAM expensive. Update to 0.20.6 hbase.
Doing this will have no effect on the zk lease, the thing that is
responsible for servers going down:
> We tried increasing hbase.regionserver.lease.period to 2 minutes but that
> didn't seem to make a difference here.
Up the ticktime, this setting:
<property>
<name>hbase.zookeeper.property.tickTime</name>
<value>3000</value>
<description>Property from ZooKeeper's config zoo.cfg.
The number of milliseconds of each tick. See
zookeeper.session.timeout description.
</description>
</property>
Also up the below (see description for relation between below and the
above ticktime).
<property>
<name>zookeeper.session.timeout</name>
<value>60000</value>
<description>ZooKeeper session timeout.
HBase passes this to the zk quorum as suggested maximum time for a
session. See
http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions
"The client sends a requested timeout, the server responds with the
timeout that it can give the client. The current implementation
requires that the timeout be a minimum of 2 times the tickTime
(as set in the server configuration) and a maximum of 20 times
the tickTime." Set the zk ticktime with hbase.zookeeper.property.tickTime.
In milliseconds.
</description>
</property>
St.Ack
1. http://markmail.org/thread/e43gybkrcecg5rxo
> So, if this isn't a GC issue, is there anything else it could be, based on
> the logs?
>
> Thanks,
>
> - Dmitry
>
> On Thu, Aug 26, 2010 at 12:37 PM, Ryan Rawson <[email protected]> wrote:
>
>> Ok I didnt see your logs earlier - normally attachments are filtered
>> out and we use pastebin for logs.
>>
>> I am not seeing any large pauses in your gc logs. Not sure if the log
>> is complete enough or what...
>>
>> -ryan
>>
>> On Thu, Aug 26, 2010 at 12:32 PM, Ryan Rawson <[email protected]> wrote:
>> > Without gc logs you cannot diagnose what you suspect are gc issues...
>> > make sure you are logging and then check them out. If you are running
>> > a recent JVM you can use -XX:+PrintGCDateStamps and get better log
>> > entries.
>> >
>> > Also you cannot swap at all, even 1 page of swapping in a java process
>> > can be killer. Combined with the hypervisor stealing your CPU you can
>> > have a lot of elapsed wall time with not very many cpu slices being
>> > executed. Consider vmstat and top to diagnose that one issue.
>> >
>> > On the GC issue, the one setting you are using which is initiating
>> > occupancy fraction is set kind of low. This means you will kick in the
>> > GC once you hit 50% of your memory usage. You might consider testing
>> > with that set to a medium level, say 75% or so.
>> >
>> > -ryan
>> >
>> > On Thu, Aug 26, 2010 at 12:17 PM, Dmitry Chechik <[email protected]>
>> wrote:
>> >> Hi all,
>> >> We're still seeing these crashes pretty frequently. Attached is the
>> error
>> >> from the regionserver logs as well as a GC dump of the last hour of the
>> >> regionserver:
>> >> 2010-08-26 13:34:10,855 WARN org.apache.hadoop.hbase.util.Sleeper: We
>> slept
>> >> 157041ms, ten times longer than scheduled: 10000
>> >> 2010-08-26 13:34:10,925 WARN org.apache.hadoop.hbase.util.Sleeper: We
>> slept
>> >> 148602ms, ten times longer than scheduled: 1000
>> >> 2010-08-26 13:34:10,925 WARN
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
>> >> master for 148602 milliseconds - retrying
>> >> Since our workload is mostly scans in mapreduce, we've turned off block
>> >> caching as per https://issues.apache.org/jira/browse/HBASE-2252 in case
>> that
>> >> had anything to do with it.
>> >> We've also decreased NewSize and MaxNewSize and decreased
>> >> CMSInitiatingOccupancyFraction, so our GC settings now are:
>> >> -Xmx2000m
>> >> -XX:+UseConcMarkSweepGC
>> >> -XX:CMSInitiatingOccupancyFraction=50
>> >>
>> >>
>> >>
>> >> -XX:NewSize=32m
>> >>
>> >>
>> >>
>> >> -XX:MaxNewSize=32m
>> >>
>> >>
>> >>
>> >> -XX:+DoEscapeAnalysis
>> >> -XX:+AggressiveOpts
>> >> -verbose:gc
>> >> -XX:+PrintGCDetails
>> >> -XX:+PrintGCTimeStamp
>> >> We're running with 2G of RAM.
>> >> Is the solution here only to move to machines with more RAM, or are
>> there
>> >> other GC settings we should look at?
>> >> Thanks,
>> >> - Dmitry
>> >> On Wed, Jul 14, 2010 at 4:39 PM, Dmitry Chechik <[email protected]>
>> >> wrote:
>> >>>
>> >>> We're running with 1GB of heap space.
>> >>> Thanks all - we'll look into GC tuning some more.
>> >>>
>> >>> On Wed, Jul 14, 2010 at 3:47 PM, Jonathan Gray <[email protected]>
>> wrote:
>> >>>>
>> >>>> This doesn't look like a clock skew issue.
>> >>>>
>> >>>> @Dmitry, while you should be running CMS, this is still a garbage
>> >>>> collector and is still vulnerable to GC pauses. There are additional
>> >>>> configuration parameters to tune even more.
>> >>>>
>> >>>> How much heap are you running with on your RSs? If you are hitting
>> your
>> >>>> servers with lots of load you should run with 4GB or more.
>> >>>>
>> >>>> Also, having ZK on the same servers as RS/DN is going to create
>> problems
>> >>>> if you're already hitting your IO limits.
>> >>>>
>> >>>> JG
>> >>>>
>> >>>> > -----Original Message-----
>> >>>> > From: Arun Ramakrishnan [mailto:[email protected]]
>> >>>> > Sent: Wednesday, July 14, 2010 3:33 PM
>> >>>> > To: [email protected]
>> >>>> > Subject: RE: region servers crashing
>> >>>> >
>> >>>> > Had a problem that caused issues that looked like this.
>> >>>> >
>> >>>> > > 2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper:
>> We
>> >>>> > slept
>> >>>> > > 86246ms, ten times longer than scheduled: 1000
>> >>>> >
>> >>>> > Our problem was with clock skew. We just had to make sure ntp was
>> >>>> > running on all machines and also the timezones detected on all the
>> >>>> > machines were the same.
>> >>>> >
>> >>>> > -----Original Message-----
>> >>>> > From: [email protected] [mailto:[email protected]] On Behalf Of
>> Jean-
>> >>>> > Daniel Cryans
>> >>>> > Sent: Wednesday, July 14, 2010 3:11 PM
>> >>>> > To: [email protected]
>> >>>> > Subject: Re: region servers crashing
>> >>>> >
>> >>>> > Dmitry,
>> >>>> >
>> >>>> > Your log shows this:
>> >>>> >
>> >>>> > > 2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper:
>> We
>> >>>> > slept
>> >>>> > > 86246ms, ten times longer than scheduled: 1000
>> >>>> >
>> >>>> > This is a pause that lasted more than a minute, the process was in
>> >>>> > that state (GC, swapping, mix of all of them) for some reason and it
>> >>>> > was long enough to expire the ZooKeeper session (since from its
>> point
>> >>>> > of view the region server stopped responding).
>> >>>> >
>> >>>> > The NPE is just a side-effect, it is caused by the huge pause.
>> >>>> >
>> >>>> > It's well worth upgrading, but it won't solve your pausing issues. I
>> >>>> > can only recommend closer monitoring, setting swappiness to 0 and
>> >>>> > giving more memory to HBase (if available).
>> >>>> >
>> >>>> > J-D
>> >>>> >
>> >>>> > On Wed, Jul 14, 2010 at 3:03 PM, Dmitry Chechik <
>> [email protected]>
>> >>>> > wrote:
>> >>>> > > Hi all,
>> >>>> > > We've been having issues for a few days with HBase region servers
>> >>>> > crashing
>> >>>> > > when under load from mapreduce jobs.
>> >>>> > > There are a few different errors in the region server logs - I've
>> >>>> > attached a
>> >>>> > > sample log of 4 different region servers crashing within an hour
>> of
>> >>>> > each
>> >>>> > > other.
>> >>>> > > Some details:
>> >>>> > > - This happens when a full table scan from a mapreduce is in
>> >>>> > progress.
>> >>>> > > - We are running HBase 0.20.3, with a 16-slave cluster, on EC2.
>> >>>> > > - Some of the region server errors are NPEs which look a lot
>> >>>> > > like https://issues.apache.org/jira/browse/HBASE-2077. I'm not
>> sure
>> >>>> > if that
>> >>>> > > is the exact problem or if this issue is fixed in 0.20.5. Is it
>> worth
>> >>>> > > upgrading to 0.20.5 to fix this?
>> >>>> > > - Some of the region server errors are scanner lease expired
>> errors:
>> >>>> > > 2010-07-12 15:10:03,299 WARN org.apache.hadoop.hbase.util.Sleeper:
>> We
>> >>>> > slept
>> >>>> > > 86246ms, ten times longer than scheduled: 1000
>> >>>> > > 2010-07-12 15:10:03,299 WARN org.apache.zookeeper.ClientCnxn:
>> >>>> > Exception
>> >>>> > > closing session 0x229c72b89360001 to
>> >>>> > sun.nio.ch.selectionkeyi...@7f712b3a
>> >>>> > > java.io.IOException: TIMED OUT
>> >>>> > > at
>> >>>> > >
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
>> >>>> > > 2010-07-12 15:10:03,299 INFO
>> >>>> > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> >>>> > > 1779060682963568676 lease expired
>> >>>> > > 2010-07-12 15:10:03,406 ERROR
>> >>>> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
>> >>>> > > org.apache.hadoop.hbase.UnknownScannerException: Name:
>> >>>> > 1779060682963568676
>> >>>> > > at
>> >>>> > >
>> >>>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.j
>> >>>> > ava:1877)
>> >>>> > > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown
>> >>>> > Source)
>> >>>> > > at
>> >>>> > >
>> >>>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> >>>> > rImpl.java:25)
>> >>>> > > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > > at
>> >>>> > >
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>> >>>> > > We tried increasing hbase.regionserver.lease.period to 2 minutes
>> but
>> >>>> > that
>> >>>> > > didn't seem to make a difference here.
>> >>>> > > - Our configuration and table size haven't changed significantly
>> in
>> >>>> > those
>> >>>> > > days.
>> >>>> > > - We're running a 3-node Zookeeper cluster collocated on the same
>> >>>> > machines
>> >>>> > > as the HBase/Hadoop cluster.
>> >>>> > > - Based on Ganglia output, it doesn't look like the regionservers
>> (or
>> >>>> > any of
>> >>>> > > the machines) are swapping.
>> >>>> > > - At the time of the crash, it doesn't appear that the network was
>> >>>> > > overloaded (i.e. we've seen higher network traffic without
>> crashes).
>> >>>> > So it
>> >>>> > > doesn't seem that this is a problem communicating with Zookeeper.
>> >>>> > > - We have
>> "-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode" enabled,
>> >>>> > so it
>> >>>> > > doesn't seem like we should be pausing due to GC too much.
>> >>>> > > Any thoughts?
>> >>>> > > Thanks,
>> >>>> > > - Dmitry
>> >>>
>> >>
>> >>
>> >>
>> >
>>
>