Hello Genady, You might be interested in one of our previous post about this topic: http://www.nabble.com/Datanode-Xceivers-td21372227.html
if you are using hadoop / HBase 0.19 you should leave the timeout dfs.datanode.socket.write.timeout to its original default value 480000 (8 min) Stack tested this, and the effect is that the Xcievers threads of hadoop eventually ends with errors, but the errors does not affect HBase stability since HADOOP-3831 have been fixed for 0.19 And it should decrease the number of threads, and therefore the memory needed for the jvm process. Personally, I haven't updated to 0.19 yet, therefore I haven't tested this for now, but I can't wait... One think I don't understand in your problem is that the memory allocated per thread in the jvm is not the heap, but the stack. Anyway the global process virtual memory allocated should decrease (which allow you to increase the heap.) For your information I run 3 region servers with a 512Mb heap and about 150 regions each. I see my first OOM these days. About Xcievers I see peaks of 1300 Xcievers during HBase startup with 2 datanodes, and a replication factor of 2; but if I enable the timeout I guess about 800 should be enough. Genady wrote: > > Hi, > > > > It seems that HBase 0.19 on Hadoop 0.19 fail to start because of exceeding > limit of concurrent xceivers( in hadoop datanode logs), which is currently > 3000, setting more than 3000 xceivers is causing JVM out of memory > exception, is there is something wrong with configuration parameters of > cluster( three nodes, 430 regions,Hadoop heap size is default - 1GB)? > Additional parameters in hbase configuration are: > > dfs.datanode.handler.count = 6, > > dfs.datanode.socket.write.timeout=0 > > > > java.io.IOException: xceiverCount 3001 exceeds the limit of concurrent > xcievers 3000 > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:87) > > at java.lang.Thread.run(Thread.java:619) > > > > Any help is very appreciated, > > Genady > > > > > -- View this message in context: http://www.nabble.com/Hbase-0.19-failed-to-start%3A-exceeds-the-limit-of-concurrent-xcievers-3000-tp21706173p21708680.html Sent from the HBase User mailing list archive at Nabble.com.
