Mike, thanks for responding.
BTW, I have a small update. I succeeded opening the table by setting
"hbase.master.assignment.timeoutmonitor.timeout" to 1 hour.
Now the table is hosted on single region server which is bad (see status
below). Should I compact the table and then split it?
>>>> What did you set your max region size to be for this table?
I did not set it explicitly, so default settings of 0.90.3-cdh3u1 are used.
What setting should I use?
>>>> 14K files totalling 650GB means you have a lot of small files...
>>>> On average ~45MB (rough calc).
Correct, I'd like to minimize this number but I am not sure how.
Maybe splits generated by my bulkloader MR job are just wrong, cause now I
just have only one region with bunch of small files.
>>How many regions?
Here is a status:
hbase(main):012:0> status 'detailed'
version 0.90.3-cdh3u1
0 regionsInTransition
3 live servers
slave113:60020 1320067636128
requests=0, regions=1, usedHeap=7296, maxHeap=16346
mytable,,1319730467540.69e5825d3fea11030d9f370a9219328e.
stores=2, storefiles=14917, storefileSizeMB=677337,
memstoreSizeMB=0, storefileIndexSizeMB=5774
slave115:60020 1320067640784
requests=0, regions=2, usedHeap=37, maxHeap=16346
.META.,,1
stores=1, storefiles=2, storefileSizeMB=0, memstoreSizeMB=0,
storefileIndexSizeMB=0
-ROOT-,,0
stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
storefileIndexSizeMB=0
slave114:60020 1320067640288
requests=0, regions=1, usedHeap=30, maxHeap=16346
0 dead servers
>>>> Do you have mslabs set up?
Nope. Should I?
>>>> GC tuning?
Nope. Should I? I use: "-ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
Best regards,
Matthew Tovbin =)
On Mon, Oct 31, 2011 at 15:58, Michael Segel <[email protected]>wrote:
>
> Matthew,
>
> What did you set your max region size to be for this table?
> 14K files totalling 650GB means you have a lot of small files...
> On average ~45MB (rough calc).
>
> How many regions? Do you have mslabs set up?
> (GC tuning?)
>
> Sorry for jumping in on the end of this conversation.
>
> -Mike
>
> > From: [email protected]
> > Date: Mon, 31 Oct 2011 12:07:53 +0200
> > Subject: Re: Region has been OPENING for too long
> > To: [email protected]
> >
> > Ted, thanks for such a rapid response.
> >
> > You're right, we use hbase 0.90.3 from cdh3u1.
> >
> > So, I suppose I need to make bulk loading in smaller bulks then. Any
> other
> > suggestions?
> >
> >
> > Best regards,
> > Matthew Tovbin =)
> >
> > >
> > >
> > >I assume you're using HBase 0.90.x where HBASE-4015 isn't available.
> > >
> > >>> 5. And so on, till some of Slaves fail with
> "java.net.SocketException:
> > >Too many open files".
> > >Do you have some monitoring setup so that you can know the number of
> open
> > >file handles ?
> > >
> > >Cheers
> > >
> > >On Sun, Oct 30, 2011 at 7:21 AM, Matthew Tovbin <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> Hi guys,
> > >>
> > >> I've bulkloaded a solid amount of data (650GB, ~14000 files) into
> Hbase
> > >> (1master + 3regions) and now enabling the table results the
> > >> following behavior on the cluster:
> > >>
> > >> 1. Master says that opening started -
> > >> "org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > >> transition=RS_ZK_REGION_OPENING, server=slave..."
> > >> 2. Slaves report about opening files in progress -
> > >> "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
> > >> 3. Then after ~10 mins the following error occurs on hmaster -
> > >> "org.apache.hadoop.hbase.master.AssignmentManager: Regions in
> > transition
> > >> timed out / Region has been OPENING for too long, reassigning
> > region=..."
> > >> 4. More slaves report about opening files in progress -
> > >> "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
> > >> 5. And so on, till some of Slaves fail with
> "java.net.SocketException:
> > >> Too many open files".
> > >>
> > >>
> > >> What I've done already to solve the issue (which DID NOT help though):
> > >>
> > >> 1. Set 'ulimit -n 65536' for hbase user
> > >> 2. Set hbase.hbasemaster.maxregionopen=3600000 (1 hour) in
> > hbase-site.xml
> > >>
> > >>
> > >> What else can I try?!
> > >>
> > >>
> > >> Best regards,
> > >> Matthew Tovbin =)
> > >>
>
