Hi guys,
I've bulkloaded a solid amount of data (650GB, ~14000 files) into Hbase
(1master + 3regions) and now enabling the table results the
following behavior on the cluster:
1. Master says that opening started -
"org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=slave..."
2. Slaves report about opening files in progress -
"org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
3. Then after ~10 mins the following error occurs on hmaster -
"org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out / Region has been OPENING for too long, reassigning region=..."
4. More slaves report about opening files in progress -
"org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
5. And so on, till some of Slaves fail with "java.net.SocketException:
Too many open files".
What I've done already to solve the issue (which DID NOT help though):
1. Set 'ulimit -n 65536' for hbase user
2. Set hbase.hbasemaster.maxregionopen=3600000 (1 hour) in hbase-site.xml
What else can I try?!
Best regards,
Matthew Tovbin =)
