You can get the source code from 0.98 branch of: https://github.com/apache/hbase
0.98.17 hasn't been released yet. RC would come out this month. If you cannot wait, you can build locally and deploy. Cheers > On Jan 11, 2016, at 11:30 PM, Rajeshkumar J <[email protected]> > wrote: > > Hi Ted, > > As mentioned in that HBASE-14926 it is fixed in HBase 0.98.17. I have > googled it for both its source code and binary but I didn't find one. can > you guide me where I can find Hbase 0.98.17 source code or binary package > > Thanks > >> On Tue, Jan 12, 2016 at 12:57 AM, Ted Yu <[email protected]> wrote: >> >> I took a look at the stack trace you posted. >> >> Some thrift worker was in the state shown in HBASE-14926 >> >> I wonder if the fix for HBASE-14926 would help in your case. >> >> Cheers >> >> On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J < >> [email protected]> >> wrote: >> >>> Hi Ted, >>> >>> Plz ignore my previous mail as the log is taken when hbase is working >>> fine. Please find the logs below when Hbase didn't return any records >>> >>> http://pastebin.com/APYjiGSP >>> >>> Thanks >>> >>>> On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <[email protected]> wrote: >>>> >>>> When you search for 'java jstack tutorial' using google, you would >> find a >>>> lot of tutorials. >>>> >>>> You can use 'ps aux' command to search for the following class: >>>> org.apache.hadoop.hbase.thrift.ThriftServer >>>> >>>> Apply jstack on the corresponding process id. >>>> >>>> Cheers >>>> >>>> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J < >>>> [email protected]> >>>> wrote: >>>> >>>>> Hi Ted, >>>>> >>>>> I don't know how how to take stacktrace using jstack. But i tried >>>> using >>>>> some commands but it fails. can you help me in this? >>>>> >>>>> Thanks >>>>> >>>>> On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <[email protected]> >> wrote: >>>>> >>>>>> What I meant was to capture stack trace of the thrift server (using >>>>> jstack >>>>>> command) when your query is running. >>>>>> >>>>>> This would give us some more clue as to the state thrift server was >>> in >>>>> when >>>>>> the problem happened. >>>>>> >>>>>> Cheers >>>>>> >>>>>> On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J < >>>>>> [email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Ted, >>>>>>> >>>>>>> This is what I found in the Hbase thrift server logs >>>>>>> >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:java.library.path=;C:\Hadoop\bin >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:java.io.tmpdir=C:\Windows\TEMP\ >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:java.compiler=<NA> >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:os.name=Windows Server 2012 >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:os.arch=amd64 >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:os.version=6.2 >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:user.name=xxxx >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:user.home=C:\ >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:user.dir=C:\HBase\bin >>>>>>> 2016-01-09 06:27:43,142 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>>> Initiating client connection, >>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181 >>>>>>> sessionTimeout=90000 watcher=hconnection-0x45e35460x0, >>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181, >> baseZNode=/hbase >>>>>>> 2016-01-09 06:27:43,339 INFO >>>>> [thrift-worker-0-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Opening socket connection to server >>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using >>> SASL >>>>>>> (unknown error) >>>>>>> 2016-01-09 06:27:43,342 INFO >>>>> [thrift-worker-0-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Socket connection established to >>>>>>> hostname/xx.xx.x.xx:2181, initiating session >>>>>>> 2016-01-09 06:27:43,365 INFO >>>>> [thrift-worker-0-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Session establishment complete on server >>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, >> negotiated >>>>>> timeout >>>>>>> = 40000 >>>>>>> 2016-01-09 07:14:10,607 INFO >>>>>>> [hostname,60000,1452337815006_splitLogManager__ChoreService_1] >>>>>>> master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager >>> Timeout >>>>>>> Monitor missed its start time >>>>>>> 2016-01-09 07:16:43,511 INFO >>>>>>> [B.defaultRpcServer.handler=8,queue=2,port=60000] >> master.HMaster: >>>>>>> Client=Administrator/null create 'chk2', {NAME => 'fa', >>>>>> DATA_BLOCK_ENCODING >>>>>>> => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', >> VERSIONS >>>> => >>>>>> '1', >>>>>>> COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', >>>>>>> KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => >>>>>> 'false', >>>>>>> BLOCKCACHE => 'true'} >>>>>>> 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1] >>>>>>> procedure2.ProcedureExecutor: Rolledback procedure >>>> CreateTableProcedure >>>>>>> (table=chk2) user=Administrator (auth:SIMPLE) id=2 >> state=ROLLEDBACK >>>>>>> exec-time=462msec >>>>> exception=org.apache.hadoop.hbase.TableExistsException: >>>>>>> chk2 >>>>>>> 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1] >>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>> zookeeper >>>>>>> sessionid=0x1522613893e0004 >>>>>>> 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1] >>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed >>>>>>> 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread] >>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>> 2016-01-09 18:40:29,469 INFO [thrift-worker-1] >>>>>>> zookeeper.RecoverableZooKeeper: Process >>>>> identifier=hconnection-0x591287f8 >>>>>>> connecting to ZooKeeper >>>>>> ensemble=hostname:2181,hostname:2181,hostname:2181 >>>>>>> 2016-01-09 18:40:29,482 INFO [thrift-worker-1] >>> zookeeper.ZooKeeper: >>>>>>> Initiating client connection, >>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181 >>>>>>> sessionTimeout=90000 watcher=hconnection-0x591287f80x0, >>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181, >> baseZNode=/hbase >>>>>>> 2016-01-09 18:40:29,553 INFO >>>>> [thrift-worker-1-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Opening socket connection to server >>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using >>> SASL >>>>>>> (unknown error) >>>>>>> 2016-01-09 18:40:29,559 INFO >>>>> [thrift-worker-1-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Socket connection established to >>>>>>> hostname/xx.xx.x.xx:2181, initiating session >>>>>>> 2016-01-09 18:40:29,569 INFO >>>>> [thrift-worker-1-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Session establishment complete on server >>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, >> negotiated >>>>>> timeout >>>>>>> = 40000 >>>>>>> 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1] >>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>> zookeeper >>>>>>> sessionid=0x35226137e300008 >>>>>>> 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1] >>>>>>> zookeeper.ZooKeeper: Session: 0x35226137e300008 closed >>>>>>> 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread] >>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>> 2016-01-09 20:05:42,469 INFO [thrift-worker-13] >>>>>>> zookeeper.RecoverableZooKeeper: Process >>>>> identifier=hconnection-0x6305dbeb >>>>>>> connecting to ZooKeeper >>>>>> ensemble=hostname:2181,hostname:2181,hostname:2181 >>>>>>> 2016-01-09 20:05:42,476 INFO [thrift-worker-13] >>> zookeeper.ZooKeeper: >>>>>>> Initiating client connection, >>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181 >>>>>>> sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0, >>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181, >> baseZNode=/hbase >>>>>>> 2016-01-09 20:05:42,507 INFO >>>>>> [thrift-worker-13-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Opening socket connection to server >>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using >>> SASL >>>>>>> (unknown error) >>>>>>> 2016-01-09 20:05:42,510 INFO >>>>>> [thrift-worker-13-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Socket connection established to >>>>>>> hostname/xx.xx.x.xx:2181, initiating session >>>>>>> 2016-01-09 20:05:42,708 INFO >>>>>> [thrift-worker-13-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Session establishment complete on server >>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, >> negotiated >>>>>> timeout >>>>>>> = 40000 >>>>>>> 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1] >>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>> zookeeper >>>>>>> sessionid=0x1522613893e0005 >>>>>>> 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1] >>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Rajeshkumar J >>>>>>> >>>>>>> On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <[email protected]> >>>> wrote: >>>>>>> >>>>>>>> Can you take a stack trace of the thrift server and pastebin >> the >>>>> trace >>>>>> ? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J < >>>>>>>> [email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> >>>>>>>>> I am connecting via Hbase thrift server to access records in >>>> Hbase >>>>>> and >>>>>>> I >>>>>>>> am >>>>>>>>> doing this in C# and i am using range scan to get records >> from >>>> the >>>>>>> table. >>>>>>>>> This seems to be working for few hours but after few hours it >>>>> didn't >>>>>>>>> returned any records and then i searched hbase logs i found >>> this >>>>>>>>> >>>>>>>>> 2016-01-04 00:44:54,880 INFO [ConnectionCleaner] >>>>>>>>> client.HConnectionManager$HConnectionImplementation: Closing >>>>>> zookeeper >>>>>>>>> sessionid=0x3520b1721650006 >>>>>>>>> 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] >>>>>> zookeeper.ZooKeeper: >>>>>>>>> Session: 0x3520b1721650006 closed >>>>>>>>> 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread] >>>>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>>>> >>>>>>>>> And I am using hbase 0.98.13 >>>>>>>>> >>>>>>>>> I have asked this question before and some user suggested me >> to >>>>>> upgrade >>>>>>>> to >>>>>>>>> new version so i have upgraded HBase to 1.1.2. But now it >> also >>> it >>>>>>> didn't >>>>>>>>> return any records but different log generated as shown below >>>>>>>>> >>>>>>>>> 2016-01-09 20:05:42,708 INFO >>>>>> [thrift-worker-13-SendThread(xxxx:2181)] >>>>>>>>> zookeeper.ClientCnxn: Session establishment complete on >> server >>>>>>>>> xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, >> negotiated >>>>>>> timeout = >>>>>>>>> 40000 >>>>>>>>> 2016-01-09 20:15:45,541 INFO >> [ConnectionCache_ChoreService_1] >>>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>>>> zookeeper >>>>>>>>> sessionid=0x1522613893e0005 >>>>>>>>> 2016-01-09 20:15:45,567 INFO >> [ConnectionCache_ChoreService_1] >>>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed >>>>>>>>> 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread] >>>>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>>>> >>>>>>>>> Can any one help me in this? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Rajeshkumar J >>
