Hi Ted - could you provide details on the asynchbase mailing list.. how do i send them this post ?
regds, Karan Alang On Wed, Dec 14, 2016 at 5:21 PM, Ted Yu-3 [via Apache HBase] < [email protected]> wrote: > I took a look at opentsdb.log which shows asynchbase logs. > > Have you considered polling asynchbase mailing list ? > > On Wed, Dec 14, 2016 at 4:59 PM, karanalang <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=4085137&i=0>> wrote: > > > I've kerberized HDP 2.4 (HBase version - 1.1.2.2.4.0.0-169, openTSDB > > version > > - 2.2.1) > > On startiong OpenTSDB, i'm getting following error -> > > > > What i understand (from the posts on the internet) is that this is > because > > of connection issues between HBase & OpenTSDB > > > > In connecting to HBase .META, the connection is failing. > > Any ideas on how to fix this ? > > > > Here is the command used to start OpenTSDB > > /build/tsdb tsd --zkbasedir=/hbase-secure --port=9999 > > --zkquorum=sandbox.hortonworks.com:2181 --cachedir=/tmp/tsd > > --staticroot=build/staticroot --auto-metric > > > > Also attaching the hbase-site.xml, opentsdb.log, opentsdb.conf > > > > > > > > ------------------------------------------------------------ > > ---------------------------------------- > > > > 2016-12-14 18:55:58,002 INFO [AsyncHBase I/O Worker #4] HBaseClient: > > Channel [id: 0x9040cb54, /10.0.2.15:51692 => /10.0.2.15:16020] is > > disconnecting: [id: 0x9040cb54, /10.0.2.15:51692 => /10.0.2.15:16020] > > CLOSE > > 2016-12-14 18:55:58,002 INFO [AsyncHBase I/O Worker #4] HBaseClient: > Lost > > connection with the .META. region > > 2016-12-14 18:55:58,003 INFO [AsyncHBase I/O Worker #4] HBaseClient: > > Invalidated cache for .META. as null still seems to be splitting or > closing > > it. > > 2016-12-14 18:55:58,006 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > ZooKeeper: Session: 0x158fa6cce65022b closed > > 2016-12-14 18:55:58,007 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > ClientCnxn: EventThread shut down > > 2016-12-14 18:55:59,524 INFO [AsyncHBase Timer HBaseClient #1] > ZooKeeper: > > Initiating client connection, connectString=sandbox.hortonworks.com:2181 > > sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@438f3a6 > > 2016-12-14 18:55:59,531 INFO [AsyncHBase Timer HBaseClient > > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket > > connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 2016-12-14 18:55:59,533 INFO [AsyncHBase Timer HBaseClient > > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket > connection > > established to sandbox.hortonworks.com/10.0.2.15:2181, initiating > session > > 2016-12-14 18:55:59,535 INFO [AsyncHBase Timer HBaseClient #1] > > HBaseClient: > > Need to find the .META. region > > 2016-12-14 18:55:59,536 INFO [AsyncHBase Timer HBaseClient > > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session > > establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, > > > sessionid = 0x158fa6cce65022c, negotiated timeout = 5000 > > 2016-12-14 18:55:59,539 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 > > 2016-12-14 18:55:59,545 INFO [AsyncHBase I/O Worker #1] HBaseClient: > > Channel [id: 0x860a8aa5, /10.0.2.15:51700 :> /10.0.2.15:16020] is > > disconnecting: [id: 0x860a8aa5, /10.0.2.15:51700 :> /10.0.2.15:16020] > > DISCONNECTED > > 2016-12-14 18:55:59,548 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > HBaseClient: Need to find the .META. region > > 2016-12-14 18:55:59,553 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > ZooKeeper: Session: 0x158fa6cce65022c closed > > 2016-12-14 18:55:59,553 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > ZooKeeper: Initiating client connection, > > connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 > > watcher=org.hbase.async.HBaseClient$ZKClient@438f3a6 > > 2016-12-14 18:55:59,576 INFO [AsyncHBase Timer HBaseClient > > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket > > connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not > > attempt to authenticate using SASL (unknown error) > > 2016-12-14 18:55:59,579 INFO [AsyncHBase Timer HBaseClient > > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket > connection > > established to sandbox.hortonworks.com/10.0.2.15:2181, initiating > session > > 2016-12-14 18:55:59,585 INFO [AsyncHBase Timer HBaseClient > > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session > > establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, > > > sessionid = 0x158fa6cce65022d, negotiated timeout = 5000 > > 2016-12-14 18:55:59,586 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > ClientCnxn: EventThread shut down > > 2016-12-14 18:55:59,594 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 > > 2016-12-14 18:55:59,603 INFO [AsyncHBase I/O Worker #2] HBaseClient: > > Channel [id: 0x6c0b2673, /10.0.2.15:51704 :> /10.0.2.15:16020] is > > disconnecting: [id: 0x6c0b2673, /10.0.2.15:51704 :> /10.0.2.15:16020] > > DISCONNECTED > > 2016-12-14 18:55:59,606 WARN [AsyncHBase Timer HBaseClient > #1-EventThread] > > HBaseClient: Probe Exists(table="hbase:meta", key=[116, 115, 100, 98, > 44, > > 0, > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, > 110, > > 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, > 60, > > 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, > > 0, > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, > 121, > > 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, > > 95, > > 60], family=null, qualifiers=null, attempt=0, > > region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", > > stop_key="")) failed > > org.hbase.async.NonRecoverableException: Too many attempts: > > Exists(table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, > 0, > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, > > 115, > > 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, > 0, > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, > 66, > > 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], > > family=null, qualifiers=null, attempt=11, > > region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", > > stop_key="")) > > at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient. > > java:2056) > > [asynchbase-1.7.2.jar:na] > > at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient. > > java:1920) > > [asynchbase-1.7.2.jar:na] > > at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient. > > java:1944) > > [asynchbase-1.7.2.jar:na] > > at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient. > > java:1927) > > [asynchbase-1.7.2.jar:na] > > at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) > > [async-1.4.0.jar:na] > > at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) > > > [async-1.4.0.jar:na] > > at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) > > [async-1.4.0.jar:na] > > at > > org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient. > > > java:3645) > > [asynchbase-1.7.2.jar:na] > > at > > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java: > > > 558) > > [zookeeper-3.4.5.jar:3.4.5-1392090] > > at org.apache.zookeeper.ClientCnxn$EventThread.run( > > ClientCnxn.java:495) > > [zookeeper-3.4.5.jar:3.4.5-1392090] > > 2016-12-14 18:55:59,607 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > HBaseClient: Need to find the .META. region > > 2016-12-14 18:55:59,611 INFO [AsyncHBase Timer HBaseClient > #1-EventThread] > > ZooKeeper: Session: 0x158fa6cce65022d closed > > ------------------------------------------------------------ > > ----------------- > > > > hbase-site.txt > > <http://apache-hbase.679495.n3.nabble.com/file/n4085136/hbase-site.txt> > > opentsdb.zip > > <http://apache-hbase.679495.n3.nabble.com/file/n4085136/opentsdb.zip> > > hbase-site.txt > > <http://apache-hbase.679495.n3.nabble.com/file/n4085136/hbase-site.txt> > > > > > > > > -- > > View this message in context: http://apache-hbase.679495.n3. > > nabble.com/Regionserver-IP-in-Meta-tp4077711p4085136.html > > Sent from the HBase User mailing list archive at Nabble.com. > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-hbase.679495.n3.nabble.com/Regionserver-IP-in- > Meta-tp4077711p4085137.html > To unsubscribe from Regionserver IP in Meta, click here > <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4077711&code=a2FyYW4uYWxhbmdAZ21haWwuY29tfDQwNzc3MTF8NTY2MTQ3ODQx> > . > NAML > <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Regionserver-IP-in-Meta-tp4077711p4085138.html Sent from the HBase User mailing list archive at Nabble.com.
