Just changing JAVA_HOME worked for us.
On Sun, Apr 27, 2014 at 9:44 PM, Henry Hung <[email protected]> wrote: > @Bryan, > > Do I have to recompile the Hadoop-2.2.0 and hbase-0.96 using java 1.7 or > just change the JAVA_HOME and restart it? > > I would probably use Oracle Java 1.7 u55 to test it. > > Best regards, > Henry > > -----Original Message----- > From: Bryan Beaudreault [mailto:[email protected]] > Sent: Monday, April 28, 2014 8:45 AM > To: [email protected] > Subject: Re: suggestion for how eliminate memory problem in heavy-write > hbase region server > > Keep in mind the math for heap. Memstore is 40%, and that is divided > across all regions for a RS. With 2GB, that's under 30mb per region > assuming well distributed writes. > > Are your regionservers starved for CPU? Either way, I'd try the java7 G1 > GC on one regionserver and report back. We run with 25GB heaps and never > have long pauses, so 16GB should be fine with enough CPU. > > > On Sun, Apr 27, 2014 at 8:27 PM, Henry Hung <[email protected]> wrote: > > > @Vladimir: > > > > My total region count is 330 right now, but I expect the number will > > surpass 1000 at the end of this year. > > Although current system is heavy-write, but I expect it to be also a > > read intense system, because I want to do a lot of data analysis in the > future. > > > > About memory allocation, I assign 16 GB heap memory just because I > > have so many RAM space left over to allocate (each node has 32GB RAM > installed). > > I never really thought that 2GB heap will be enough, but will give it > > a shot. > > > > Best regards, > > Henry > > > > -----Original Message----- > > From: Vladimir Rodionov [mailto:[email protected]] > > Sent: Saturday, April 26, 2014 1:48 AM > > To: [email protected] > > Subject: RE: suggestion for how eliminate memory problem in > > heavy-write hbase region server > > > > I am just wondering why do you need large heaps on write - heavy cluster? > > How many regions per RS do you have? Usually large heaps need for read > > intensive applications to keep blocks in a cache, but now we have > > option to keep these blocks off heap (at least since 0.96) With > > MemSLAB for MemStore enabled you need 2MB (by default) per region, > > this is what you need to take into account first. Even 1000 regions > won't eat more than 2GB of heap. > > > > Best regards, > > Vladimir Rodionov > > Principal Platform Engineer > > Carrier IQ, www.carrieriq.com > > e-mail: [email protected] > > > > ________________________________________ > > From: Ted Yu [[email protected]] > > Sent: Friday, April 25, 2014 9:47 AM > > To: [email protected] > > Subject: Re: suggestion for how eliminate memory problem in > > heavy-write hbase region server > > > > Henry: > > Please also take a look at the following thread: > > > > > > http://search-hadoop.com/m/51M4jeDMyy1/GC+recommendations+for+large+Re > > gion+Server+heaps&subj=RE+GC+recommendations+for+large+Region+Server+h > > eaps > > > > > > On Thu, Apr 24, 2014 at 11:17 PM, Mikhail Antonov > > <[email protected] > > >wrote: > > > > > Henry, > > > > > > http://blog.ragozin.info/2011/10/java-cg-hotspots-cms-and-heap.html > > > - that may give some insights. > > > > > > -Mikhail > > > > > > > > > 2014-04-24 23:07 GMT-07:00 Henry Hung <[email protected]>: > > > > > > > Dear All, > > > > > > > > My current hbase environment is heavy write cluster with constant > > > > 2000+ insert rows / second spread to 10 region servers. > > > > Each day I also need to do data deletion, and that will add a lot > > > > of IO > > > to > > > > the cluster. > > > > > > > > The problem is sometimes after a week, one of the region server > > > > will > > > crash > > > > because > > > > 2014-04-10T10:17:47.200+0800: 1281486.956: [GC 1281486.956: > > > > [ParNew (promotion failed): 235959K->235959K(235968K), 0.0836790 > > > secs]1281487.040: > > > > [CMS2014-04-10T10:21:14.957+0800: 1281694.712: [CMS-concurrent-sweep: > > > > 267.111/279.155 secs] [Times: user=334.79 sys=14.38, real=279.11 > > > > secs] (concurrent mode failure): 13961950K->6802914K(16515072K), > > > > 209.9436660 secs] 14186496K->6802914K(16751040K), [CMS Perm : > > > 42864K->42859K(71816K)], > > > > 210.0274680 secs] [Times: user=210.18 sys=0.01, real=209.99 secs] > > > > > > > > I look into the gc log and usually find some information about CMS > > > > concurrent sweep that took a very long time to complete, such as: > > > > 2014-04-10T10:15:56.929+0800: 1281376.684: [CMS-concurrent-sweep: > > > > 48.834/58.027 secs] [Times: user=101.52 sys=11.82, real=58.02 > > > > secs] > > > > > > > > I do a lot of google-ing and already read the Todd Lipcon avoiding > > > > full GC, or other blogs that sometimes tells me how to set jvm > > > > flags such as > > > > this: > > > > -XX:+UseParNewGC > > > > -XX:CMSInitiatingOccupancyFraction=70 > > > > -Xmn256m > > > > -Xmx16384m > > > > -XX:+DisableExplicitGC > > > > -XX:+UseCompressedOops > > > > -XX:PermSize=160m > > > > -XX:MaxPermSize=160m > > > > -XX:GCTimeRatio=19 > > > > -XX:SoftRefLRUPolicyMSPerMB=0 > > > > -XX:SurvivorRatio=2 > > > > -XX:MaxTenuringThreshold=1 > > > > -XX:+UseFastAccessorMethods > > > > -XX:+UseParNewGC > > > > -XX:+UseConcMarkSweepGC > > > > -XX:+CMSParallelRemarkEnabled > > > > -XX:+UseCMSCompactAtFullCollection > > > > -XX:CMSFullGCsBeforeCompaction=0 > > > > -XX:+CMSClassUnloadingEnabled > > > > -XX:CMSMaxAbortablePrecleanTime=300 > > > > -XX:+CMSScavengeBeforeRemark > > > > > > > > But alas, the problem still exist. > > > > > > > > I also know that java 1.7 has a new G1GC that probably can be used > > > > to fix this problem, but I don't know if hbase 0.96 is ready to use > it? > > > > > > > > I would really appreciate it if someone out there can share one or > > > > two things about jvm configuration to achieve a more stable region > > server. > > > > > > > > Best regards, > > > > Henry > > > > > > > > ________________________________ > > > > The privileged confidential information contained in this email is > > > > intended for use only by the addressees as indicated by the > > > > original > > > sender > > > > of this email. If you are not the addressee indicated in this > > > > email or > > > are > > > > not responsible for delivery of the email to such a person, please > > > > kindly reply to the sender indicating this fact and delete all > > > > copies of it from your computer and network server immediately. > > > > Your cooperation is highly appreciated. It is advised that any > > > > unauthorized use of confidential information of Winbond is > > > > strictly prohibited; and any information in > > > this > > > > email irrelevant to the official business of Winbond shall be > > > > deemed as neither given nor endorsed by Winbond. > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Michael Antonov > > > > > > > Confidentiality Notice: The information contained in this message, > > including any attachments hereto, may be confidential and is intended > > to be read only by the individual or entity to whom this message is > > addressed. If the reader of this message is not the intended recipient > > or an agent or designee of the intended recipient, please note that > > any review, use, disclosure or distribution of this message or its > > attachments, in any form, is strictly prohibited. If you have > > received this message in error, please immediately notify the sender > > and/or [email protected] and delete or destroy any copy of > this message and its attachments. > > > > The privileged confidential information contained in this email is > > intended for use only by the addressees as indicated by the original > > sender of this email. If you are not the addressee indicated in this > > email or are not responsible for delivery of the email to such a > > person, please kindly reply to the sender indicating this fact and > > delete all copies of it from your computer and network server > > immediately. Your cooperation is highly appreciated. It is advised > > that any unauthorized use of confidential information of Winbond is > > strictly prohibited; and any information in this email irrelevant to > > the official business of Winbond shall be deemed as neither given nor > endorsed by Winbond. > > > > The privileged confidential information contained in this email is > intended for use only by the addressees as indicated by the original sender > of this email. If you are not the addressee indicated in this email or are > not responsible for delivery of the email to such a person, please kindly > reply to the sender indicating this fact and delete all copies of it from > your computer and network server immediately. Your cooperation is highly > appreciated. It is advised that any unauthorized use of confidential > information of Winbond is strictly prohibited; and any information in this > email irrelevant to the official business of Winbond shall be deemed as > neither given nor endorsed by Winbond. >
