I assume you're using HBase 0.90.x where HBASE-4015 isn't available. >> 5. And so on, till some of Slaves fail with "java.net.SocketException: Too many open files". Do you have some monitoring setup so that you can know the number of open file handles ?
Cheers On Sun, Oct 30, 2011 at 7:21 AM, Matthew Tovbin <[email protected]> wrote: > Hi guys, > > I've bulkloaded a solid amount of data (650GB, ~14000 files) into Hbase > (1master + 3regions) and now enabling the table results the > following behavior on the cluster: > > 1. Master says that opening started - > "org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, server=slave..." > 2. Slaves report about opening files in progress - > "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...." > 3. Then after ~10 mins the following error occurs on hmaster - > "org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition > timed out / Region has been OPENING for too long, reassigning region=..." > 4. More slaves report about opening files in progress - > "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...." > 5. And so on, till some of Slaves fail with "java.net.SocketException: > Too many open files". > > > What I've done already to solve the issue (which DID NOT help though): > > 1. Set 'ulimit -n 65536' for hbase user > 2. Set hbase.hbasemaster.maxregionopen=3600000 (1 hour) in hbase-site.xml > > > What else can I try?! > > > Best regards, > Matthew Tovbin =) >
