The blog Julian mentioned referred to HBASE-7101 That JIRA was marked dup of HBASE-7551 which was resolved in 0.94.5
This is why upgrading HBase is desired. On Wed, Oct 23, 2013 at 10:10 AM, Ted Yu <[email protected]> wrote: > bq. our hbase version was 0.94.0 and hadoop version was 1.0.3 > > If possible, consider upgrading HBase to 0.94.12 and Hadoop to 1.2.1 > > Cheers > > > On Wed, Oct 23, 2013 at 7:18 AM, 张莉苹 <[email protected]> wrote: > >> Hello Ted, and Julian, >> >> It seemed I didn't receive your mail from my gmail inbox. It was strange. >> Anyhow I found your reply by google search. ：） >> >> I'll answer your questions here, and thanks very much for your reply. >> >> >> From *Ted*: >> >> What version of HBase and Hadoop are you using ? >> >> >> our hbase version was 0.94.0 and hadoop version was 1.0.3 >> >> Can you show us more of the master log ? >> >> >> I'm so sorry, the master log could not be accessed. The environment >> was only used by me for a short while. >> >> >> From * Michael < >> http://www.mail-archive.com/[email protected]&q=from:%22Michael+Segel%22 >> >:* >> >> >> Why 9 zookeepers? >> >> >> the previous zookeeper number was 5, we also thought the number of >> zookeeper was too small, so we increased it into 9, but it still failed. >> >> >> From *Julian*: >> >> >> Hello Michelle, >> How many regions totally are there in your 600 nodes cluster? Looks >> like many of them are pending for open and being assigned to region >> servers. >> Can you see many items under zookeeper dir /hbase/unassigned? >> >> >> we had 60K regions, the env could not be accessed. >> >> You would like to refer >> http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.html? >> >> >> kind of help, thanks! >> >> >> >> >> >> Cheers, >> ----- >> Big Data - Big Wisdom - Big Value >> -------------- >> Michelle Zhang (Li Ping Zhang) >> >> >> 2013/10/23 张莉苹 <[email protected]> >> >> > Dear HBase dev and users, >> > >> > Did you meet this >> > >> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables" >> > issue? >> > >> > We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase, >> > but it seemed hbase master was busy handling transition with zookeeper, >> > and hbase “list” could not get response. The hbase table was created but >> > it didn't do any insert. >> > >> > Do you have any idea of the root cause and how to fix it? :)Highly >> > appreciate for your answers! >> > >> > >> > >> > Here is the exception stack: >> > --------------------------------------------------- >> > java.lang.reflect.UndeclaredThrowableException >> > at $Proxy7.getHTableDescriptors(Unknown Source) >> > at >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237) >> > at >> > >> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317) >> > >> > >> > >> > >> > hbase master log: >> > >> > ----------------------------- >> > >> > 2013-10-18 06:19:41,279 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZKAssign: >> > master:60000-0x341be88202300ab* Deleting existing unassigned node* for >> > 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state >> > RS_ZK_REGION_OPENED >> > >> > 2013-10-18 06:19:41,279 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager:* Handling >> > transition=RS_ZK_REGION_OPENING*, s*erver=node0878*. >> > ic.analyticsworkbench.com,60020,1381883086785, >> > region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds >> late >> > >> > 2013-10-18 06:19:41,280 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: *Handling >> > transition=RS_ZK_REGION_OPENING, server=node0898*. >> > ic.analyticsworkbench.com,60020,1381883200494, >> > region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds >> late >> > >> > 2013-10-18 06:19:41,289 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZKAssign: >> > master:60000-0x341be88202300ab Successfully *deleted unassigned node >> *for >> > region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state >> > RS_ZK_REGION_OPENED >> > >> > 2013-10-18 06:19:41,289 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: Handling >> > transition=RS_ZK_REGION_OPENING, server= >> node0693.ic.analyticsworkbench.com,60020,1381881773670, >> > region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds >> late >> > >> > >> > >> > We also try to "list" in hbase shell,it also failed: >> > >> > The hbase “list” got error as: >> > >> > ------------------------------------------ >> > >> > >> > >> > hbase(main):001:0> list >> > >> > TABLE >> > >> > >> > >> > >> > ERROR: java.lang.reflect.UndeclaredThrowableException: Call to >> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket >> > timeout exception: java.net.SocketTimeoutException: 120000 millis >> timeout >> > while waiting for channel to be ready for read. ch : >> > java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726remote= >> > node0997.ic.analyticsworkbench.com/10.1.50.17:60000] >> > >> > >> > >> > >> > Cheers, >> > ----- >> > Big Data - Big Wisdom - Big Value >> > -------------- >> > Michelle Zhang (Li Ping Zhang) >> > >> > >
