Can you check the Transaction Manager logs and see if there are any error? Also can you do a jps and see confirm the Transaction Manager is running ?
On Wed, Aug 31, 2016 at 2:12 AM, F21 <f21.gro...@gmail.com> wrote: > Just another update. Even though the logs says that the transaction > manager is not running, it is actually running. > > I confirmed this by checking the output of ps and connecting to the > transaction manager: > > bash-4.3# ps > PID USER TIME COMMAND > 1 root 0:01 bash /run-hbase-phoenix.sh > 137 hadoop 0:19 /usr/lib/jvm/java-1.8-openjdk/bin/java > -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC > -XX:PermSize=128m -XX:Ma > 189 hadoop 0:08 /usr/lib/jvm/java-1.8-openjdk/bin/java > -XX:+UseConcMarkSweepGC -cp /opt/hbase/bin/../lib/*:/opt/h > base/bin/../conf/:/opt/hbase/phoenix-c > 542 root 0:00 /bin/bash > 9035 root 0:00 sleep 1 > 9036 root 0:00 ps > > bash-4.3# wget localhost:15165 > Connecting to localhost:15165 (127.0.0.1:15165) > wget: error getting response: Connection reset by peer > > > On 31/08/2016 3:25 PM, F21 wrote: > >> This only seems to be a problem when I have HBase running in fully >> distributed mode (1 master, 1 regionserver and 1 zookeeper node in >> different docker images). >> >> If I have HBase running in standalone mode with HBase and Phoenix and the >> Query server in 1 docker image, it works correctly. >> >> On 31/08/2016 11:21 AM, F21 wrote: >> >>> I have HBase 1.2.2 and Phoenix 4.8.0 running on my HBase master running >>> on alpine linux with OpenJDK JRE 8. >>> >>> This is my hbase-site.xml: >>> >>> <?xml version="1.0"?> >>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> >>> <configuration> >>> <property> >>> <name>hbase.rootdir</name> >>> <value>hdfs://mycluster/hbase</value> >>> </property> >>> <property> >>> <name>zookeeper.znode.parent</name> >>> <value>/hbase</value> >>> </property> >>> <property> >>> <name>hbase.cluster.distributed</name> >>> <value>true</value> >>> </property> >>> <property> >>> <name>hbase.zookeeper.quorum</name> >>> <value>m9edd51-zookeeper.m9edd51</value> >>> </property> >>> <property> >>> <name>data.tx.snapshot.dir</name> >>> <value>/tmp/tephra/snapshots</value> >>> </property> >>> <property> >>> <name>data.tx.timeout</name> >>> <value>60</value> >>> </property> >>> <property> >>> <name>phoenix.transactions.enabled</name> >>> <value>true</value> >>> </property> >>> </configuration> >>> >>> I am able to start the master correctly. I am also able to create >>> non-transactional table. >>> >>> However, if I create a transactional table, I get this error: ERROR >>> [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error >>> processing startShort >>> >>> This is what I see in the logs: >>> >>> 2016-08-31 01:08:33,560 WARN >>> [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] >>> zookeeper.ClientCnxn: Session 0x156de22abec0004 for server null, unexpected >>> error, closing socket connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientC >>> nxnSocketNIO.java:361) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn. >>> java:1081) >>> 2016-08-31 01:08:33,616 INFO [DefaultMetricsCollector STOPPING] >>> metrics.DefaultMetricsCollector: Stopped metrics reporter >>> 2016-08-31 01:08:33,623 INFO [ThriftRPCServer] >>> tephra.TransactionManager: Took 170.7 ms to stop >>> 2016-08-31 01:08:33,623 INFO [ThriftRPCServer] rpc.ThriftRPCServer: RPC >>> server for TTransactionServer stopped. >>> 2016-08-31 01:08:34,776 INFO >>> [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] >>> zookeeper.ClientCnxn: Opening socket connection to server >>> m9edd51-zookeeper.m9edd51/172.18.0.2:2181. Will not attempt to >>> authenticate using SASL (unknown error) >>> 2016-08-31 01:08:34,777 INFO >>> [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] >>> zookeeper.ClientCnxn: Socket connection established to >>> m9edd51-zookeeper.m9edd51/172.18.0.2:2181, initiating session >>> 2016-08-31 01:08:34,778 INFO >>> [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] >>> zookeeper.ClientCnxn: Session establishment complete on server >>> m9edd51-zookeeper.m9edd51/172.18.0.2:2181, sessionid = >>> 0x156de22abec0004, negotiated timeout = 40000 >>> 2016-08-31 01:08:34,783 INFO [leader-election-tx.service-leader] >>> zookeeper.LeaderElection: Connected to ZK, running election: >>> m9edd51-zookeeper.m9edd51 for /tx.service/leader >>> 2016-08-31 01:08:34,815 INFO [ThriftRPCServer] rpc.ThriftRPCServer: >>> Starting RPC server for TTransactionServer >>> 2016-08-31 01:08:34,815 INFO [ThriftRPCServer] rpc.ThriftRPCServer: >>> Running RPC server for TTransactionServer >>> 2016-08-31 01:08:34,816 INFO [ThriftRPCServer] >>> server.TThreadedSelectorServerWithFix: Starting >>> TThreadedSelectorServerWithFix >>> 2016-08-31 01:08:34,822 INFO [leader-election-tx.service-leader] >>> distributed.TransactionService: Transaction Thrift Service started >>> successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165 >>> 2016-08-31 01:10:42,830 ERROR [TTransactionServer-rpc-0] >>> thrift.ProcessFunction: Internal error processing startShort >>> java.lang.IllegalStateException: Transaction Manager is not running. >>> at com.google.common.base.Preconditions.checkState(Precondition >>> s.java:149) >>> at org.apache.tephra.TransactionManager.ensureAvailable(Transac >>> tionManager.java:709) >>> at org.apache.tephra.TransactionManager.startTx(TransactionMana >>> ger.java:768) >>> at org.apache.tephra.TransactionManager.startShort(TransactionM >>> anager.java:728) >>> at org.apache.tephra.TransactionManager.startShort(TransactionM >>> anager.java:716) >>> at org.apache.tephra.distributed.TransactionServiceThriftHandle >>> r.startShort(TransactionServiceThriftHandler.java:71) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>> ssorImpl.java:62) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServ >>> er.java:261) >>> at com.sun.proxy.$Proxy17.startShort(Unknown Source) >>> at org.apache.tephra.distributed.thrift.TTransactionServer$Proc >>> essor$startShort.getResult(TTransactionServer.java:974) >>> at org.apache.tephra.distributed.thrift.TTransactionServer$Proc >>> essor$startShort.getResult(TTransactionServer.java:959) >>> at org.apache.thrift.ProcessFunction.process(ProcessFunction. >>> java:39) >>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor. >>> java:39) >>> at org.apache.thrift.server.AbstractNonblockingServer$FrameBuff >>> er.invoke(AbstractNonblockingServer.java:478) >>> at org.apache.thrift.server.Invocation.run(Invocation.java:18) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> 2016-08-31 01:10:42,936 ERROR [TTransactionServer-rpc-0] >>> thrift.ProcessFunction: Internal error processing startShort >>> java.lang.IllegalStateException: Transaction Manager is not running. >>> at com.google.common.base.Preconditions.checkState(Precondition >>> s.java:149) >>> at org.apache.tephra.TransactionManager.ensureAvailable(Transac >>> tionManager.java:709) >>> at org.apache.tephra.TransactionManager.startTx(TransactionMana >>> ger.java:768) >>> at org.apache.tephra.TransactionManager.startShort(TransactionM >>> anager.java:728) >>> at org.apache.tephra.TransactionManager.startShort(TransactionM >>> anager.java:716) >>> at org.apache.tephra.distributed.TransactionServiceThriftHandle >>> r.startShort(TransactionServiceThriftHandler.java:71) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>> ssorImpl.java:62) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServ >>> er.java:261) >>> at com.sun.proxy.$Proxy17.startShort(Unknown Source) >>> at org.apache.tephra.distributed.thrift.TTransactionServer$Proc >>> essor$startShort.getResult(TTransactionServer.java:974) >>> at org.apache.tephra.distributed.thrift.TTransactionServer$Proc >>> essor$startShort.getResult(TTransactionServer.java:959) >>> at org.apache.thrift.ProcessFunction.process(ProcessFunction. >>> java:39) >>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor. >>> java:39) >>> at org.apache.thrift.server.AbstractNonblockingServer$FrameBuff >>> er.invoke(AbstractNonblockingServer.java:478) >>> at org.apache.thrift.server.Invocation.run(Invocation.java:18) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> This seems very similar to PHOENIX-2504 (https://issues.apache.org/jir >>> a/browse/PHOENIX-2504), but that appears to be fixed in 4.8.0? >>> >>> Any ideas why this might be happening? >>> >>> Cheers, >>> >>> Francis >>> >>> >>> >> >
