Inline. J-D
On Mon, Sep 26, 2011 at 11:47 PM, Robert J Berger <[email protected]> wrote: > Hi, > I saw a posting from a year or so ago where Andy said he was running > datanodes with xcievers set to 10k > > Is there any problem doing that? We were running with a 9 > regionserver/datanode cluster and started getting > java.io.IOException: xceiverCount 4097 exceeds the limit of concurrent > xcievers 4096 Ouch. > > We had more than 4k regions on most of the nodes. Ah that explains. > > We added 4 more nodes but still getting some of the xceiverCount errors every > once in a while (I think from when we are doing backups via the HBase export > job). Depending on the number of nodes you have, it might not have been enough. > > We're not doing any of the things that could be done to reduce the number of > files and regions. If you feel brave, you could try https://issues.apache.org/jira/browse/HBASE-1621 > We generally don't have a heave read/write load across the cluster, but we've > been doing more hadoop and backup export jobs than we use to... > > We're working on moving to a more modern cluster so I don't want to invest > much effort in the current one, but have to keep it running till we cut over. I hope it's not because of this xciever problem. > > Is there any problem to bump up the dfs.datanode.max.xcievers to like 8k or > even 10k? It really just means that the DNs will keep at most 8k or 10k threads open at max. As long as you have enough native threads it will hold on. > > Can we / should we increase the hbase.hregion.max.filesize to something more > than 256M on a running system? You'd have to change it on the existing tables for the change to take effect. This means disabling the table, doing an alter, re-enabling. But it won't merge. > > I presume to change ether we would need to restart the regionservers. Can we > do it as a rolling restart where we change one regionserver at a time and > bring one online before updating the next? Or does it have to be updated as a > cluster? Previous answer related... also yes you can roll restart: http://hbase.apache.org/book/node.management.html#rolling For that change to take effect on the new tables, I think only the master would need to be bounced. > > > Thanks! > Rob > > __________________ > Robert J Berger - CTO > Runa Inc. > +1 408-838-8896 > http://blog.ibd.com > > > >
