What version of HBase and Hadoop are you using ? Can you show us more of the master log ?
Thanks On Oct 22, 2013, at 10:46 PM, 张莉苹 <[email protected]> wrote: > Dear HBase dev and users, > > Did you meet this > "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables" > issue? > > We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase, > but it seemed hbase master was busy handling transition with zookeeper, and > hbase “list” could not get response. The hbase table was created but it > didn't do any insert. > > Do you have any idea of the root cause and how to fix it? :)Highly > appreciate for your answers! > > > > Here is the exception stack: > --------------------------------------------------- > java.lang.reflect.UndeclaredThrowableException > at $Proxy7.getHTableDescriptors(Unknown Source) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2237) > at > org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317) > > > > > hbase master log: > > ----------------------------- > > 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:60000-0x341be88202300ab* Deleting existing unassigned node* for > 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state > RS_ZK_REGION_OPENED > > 2013-10-18 06:19:41,279 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager:* Handling > transition=RS_ZK_REGION_OPENING*, s*erver=node0878*. > ic.analyticsworkbench.com,60020,1381883086785, > region=15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds late > > 2013-10-18 06:19:41,280 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: *Handling > transition=RS_ZK_REGION_OPENING, > server=node0898*.ic.analyticsworkbench.com,60020,1381883200494, > region=1a4c929534e6828c85f22b062f949304, which is more than 15 seconds late > > 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > master:60000-0x341be88202300ab Successfully *deleted unassigned node *for > region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state > RS_ZK_REGION_OPENED > > 2013-10-18 06:19:41,289 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, > server=node0693.ic.analyticsworkbench.com,60020,1381881773670, > region=d47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds late > > > > We also try to "list" in hbase shell,it also failed: > > The hbase “list” got error as: > > ------------------------------------------ > > > > hbase(main):001:0> list > > TABLE > > > > > ERROR: java.lang.reflect.UndeclaredThrowableException: Call to > node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket > timeout exception: java.net.SocketTimeoutException: 120000 millis timeout > while waiting for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/10.1.50.15:45726 remote= > node0997.ic.analyticsworkbench.com/10.1.50.17:60000] > > > > > Cheers, > ----- > Big Data - Big Wisdom - Big Value > -------------- > Michelle Zhang (Li Ping Zhang)
