Ted, thanks for such a rapid response.
You're right, we use hbase 0.90.3 from cdh3u1.
So, I suppose I need to make bulk loading in smaller bulks then. Any other
suggestions?
Best regards,
Matthew Tovbin =)
>
>
>I assume you're using HBase 0.90.x where HBASE-4015 isn't available.
>
>>> 5. And so on, till some of Slaves fail with "java.net.SocketException:
>Too many open files".
>Do you have some monitoring setup so that you can know the number of open
>file handles ?
>
>Cheers
>
>On Sun, Oct 30, 2011 at 7:21 AM, Matthew Tovbin <[EMAIL PROTECTED]> wrote:
>
>> Hi guys,
>>
>> I've bulkloaded a solid amount of data (650GB, ~14000 files) into Hbase
>> (1master + 3regions) and now enabling the table results the
>> following behavior on the cluster:
>>
>> 1. Master says that opening started -
>> "org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> transition=RS_ZK_REGION_OPENING, server=slave..."
>> 2. Slaves report about opening files in progress -
>> "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
>> 3. Then after ~10 mins the following error occurs on hmaster -
>> "org.apache.hadoop.hbase.master.AssignmentManager: Regions in
transition
>> timed out / Region has been OPENING for too long, reassigning
region=..."
>> 4. More slaves report about opening files in progress -
>> "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
>> 5. And so on, till some of Slaves fail with "java.net.SocketException:
>> Too many open files".
>>
>>
>> What I've done already to solve the issue (which DID NOT help though):
>>
>> 1. Set 'ulimit -n 65536' for hbase user
>> 2. Set hbase.hbasemaster.maxregionopen=3600000 (1 hour) in
hbase-site.xml
>>
>>
>> What else can I try?!
>>
>>
>> Best regards,
>> Matthew Tovbin =)
>>
