St.Ack,
Please see my answers below: -----Original Message----- From: stack [mailto:[email protected]] Sent: Wednesday, January 28, 2009 9:43 PM To: [email protected] Cc: [email protected] Subject: Re: Hbase 0.19 failed to start: exceeds the limit of concurrent xcievers 3000 Genady wrote: > Thanks for your answer Jean-Adrien, > > > > I've verified a setting the timeout parameter to the default value and > xceivers to original 3000(too small for our env regions number), after a > while HBase indeed succeeded to start( with tons of exceeds xceiver limit > exceptions), nevertheless performance of the MR task remain too slow, as > Jean-Daniel suggested( in previous post) probably as result of too much > regions per region server, so we going to increase file size and rebuild > data. > Leaving the default means that less resources are concurrently occupied in the datanode -- sockets and threads of under utilized files have been let go (you'll see the timeout exception in your log when the let-go happens). Resources are maximally used at startup when all the region opens are happening. You might even consider setting down the default timeout from 8 minutes to something like 2 or 4 if you run into max xceivers again. Tell us more about your slowness before you go about changing region sizes. How is it slow? Is it lookups against the .META. table? Try some yourself in the shell to see how well these are doing. See if you can narrow why its slow. Are you swapping (as J-D asked earlier). How long does the MR job run? Is it slow over its whole life? Are your tasks short? If so, you might make them run longer so you better exploit the cache of region locations built by a client. How many mappers do you have running concurrently? If many, try cutting them in half. Gennady: As soon as HBase is up even copyFromLocal to Hadoop DFS is working ten times much slower, my task have 10-20M records, which normally takes about 10 minutes, now it takes about 1 hour, no swapping was noticed on all servers, MR tasks are slow all the time. Most strange is that nothing could be seen in logs(debug is on), only higher than usual cpu rates(~90%) of region and datanode servers. Besides cutting down thread stack size is there anything else to try? Thanks, Gennady > Regarding your question about JVM errors, according to the following post it > seems that in case of the following OOM error("java.lang.OutOfMemoryError: > unable to create new native thread"), increasing a heap size will not > prevent OOM problem: > > http://www.egilh.com/blog/archive/2006/06/09/2811.aspx > > Yes, its a complaint about resources outside of the JVM heap. Upping heap size won't help. You could try playing with the -Xss -- thread stack size -- downing it from whatever the java6 default is to see if that helps. St.Ack > > > Anyway after setting Hadoop heap size to 1 or !.5GB the error didn't come > back. > > > > Gennady > > > > > > >> probably as result of increasing xceivers thread number, >> > > > > > > -----Original Message----- > From: Jean-Adrien [mailto:[email protected]] > Sent: Wednesday, January 28, 2009 6:03 PM > To: [email protected] > Subject: Re: Hbase 0.19 failed to start: exceeds the limit of concurrent > xcievers 3000 > > > > > > Hello Genady, > > > > You might be interested in one of our previous post about this topic: > > http://www.nabble.com/Datanode-Xceivers-td21372227.html > > > > if you are using hadoop / HBase 0.19 you should leave the timeout > > dfs.datanode.socket.write.timeout to its original default value 480000 (8 > > min) > > Stack tested this, and the effect is that the Xcievers threads of hadoop > > eventually ends with errors, but the errors does not affect HBase stability > > since HADOOP-3831 have been fixed for 0.19 > > And it should decrease the number of threads, and therefore the memory > > needed for the jvm process. > > > > Personally, I haven't updated to 0.19 yet, therefore I haven't tested this > > for now, but I can't wait... > > > > One think I don't understand in your problem is that the memory allocated > > per thread in the jvm is not the heap, but the stack. Anyway the global > > process virtual memory allocated should decrease (which allow you to > > increase the heap.) > > For your information I run 3 region servers with a 512Mb heap and about 150 > > regions each. I see my first OOM these days. > > About Xcievers I see peaks of 1300 Xcievers during HBase startup with 2 > > datanodes, and a replication factor of 2; but if I enable the timeout I > > guess about 800 should be enough. > > > > > > Genady wrote: > > > > >> Hi, >> > > > > >> >> > > > > >> It seems that HBase 0.19 on Hadoop 0.19 fail to start because of exceeding >> > > >> limit of concurrent xceivers( in hadoop datanode logs), which is currently >> > > >> 3000, setting more than 3000 xceivers is causing JVM out of memory >> > > >> exception, is there is something wrong with configuration parameters of >> > > >> cluster( three nodes, 430 regions,Hadoop heap size is default - 1GB)? >> > > >> Additional parameters in hbase configuration are: >> > > > > >> dfs.datanode.handler.count = 6, >> > > > > >> dfs.datanode.socket.write.timeout=0 >> > > > > >> >> > > > > >> java.io.IOException: xceiverCount 3001 exceeds the limit of concurrent >> > > >> xcievers 3000 >> > > > > >> at >> > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:87) > > > > >> at java.lang.Thread.run(Thread.java:619) >> > > > > >> >> > > > > >> Any help is very appreciated, >> > > > > >> Genady >> > > > > >> >> > > > > > > > > > >
