Hi Francis,
Is thatthe complete log for the transaction manager and if not would
you mind attaching that to a new JIRA?
Thanks,
James
On Wednesday, August 31, 2016, F21 <[email protected]
<mailto:[email protected]>> wrote:
Hey Thomas,
Where are the Transaction Manager logs located? I have a
/tmp/tephra-/tephra-service--m9edd51-hmaster1.m9edd51.log, which
was where I got the logs from yesterday. There wasn't any
information on whether the Transaction Manager crashed or not:
Here's the last half of the logs:
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name <http://os.name>=Linux
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=4.4.0-36-generic
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:user.name <http://user.name>=hadoop
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=/opt/hbase
2016-08-31 22:31:43,783 INFO [main] zookeeper.ZooKeeper: Client
environment:user.dir=/opt/hbase
2016-08-31 22:31:43,784 INFO [main] zookeeper.ZooKeeper:
Initiating client connection,
connectString=m9edd51-zookeeper.m9edd51 sessionTimeout=90000
watcher=org.apache.tephra.zookeeper.TephraZKClientService$5@45c7e403
2016-08-31 22:31:43,824 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>. Will not attempt to authenticate using
SASL (unknown error)
2016-08-31 22:31:43,829 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Socket connection established to
m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>, initiating session
2016-08-31 22:31:43,835 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>, sessionid = 0x156e2ba5ba50004,
negotiated timeout = 40000
2016-08-31 22:31:43,881 INFO [main]
inmemory.InMemoryTransactionService: Configuring
TransactionService, address: 0.0.0.0, port: 15165, threads: 20, io
threads: 2, max read buffer (bytes): 16777216
2016-08-31 22:31:43,882 INFO [main]
tephra.TransactionServiceMain: Starting TransactionServiceMain
2016-08-31 22:31:43,890 INFO [main] zookeeper.LeaderElection:
Start leader election on
m9edd51-zookeeper.m9edd51/tx.service/leader with guid
9fd6dcfa-72eb-4705-a79f-aad1ec914945
2016-08-31 22:31:43,953 INFO [leader-election-tx.service-leader]
metrics.DefaultMetricsCollector: Configured metrics report to emit
every 60 seconds
2016-08-31 22:31:44,177 INFO [ThriftRPCServer]
tephra.TransactionManager: Starting transaction manager.
2016-08-31 22:31:44,179 INFO [DefaultMetricsCollector STARTING]
metrics.DefaultMetricsCollector: Started metrics reporter
2016-08-31 22:31:44,309 WARN [HDFSTransactionStateStorage
STARTING] util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable
2016-08-31 22:31:45,041 INFO [HDFSTransactionStateStorage
STARTING] persist.HDFSTransactionStateStorage: Using snapshot dir
/tmp/tephra/snapshots
2016-08-31 22:31:45,118 INFO [ThriftRPCServer]
persist.HDFSTransactionStateStorage: Creating snapshot dir at
/tmp/tephra/snapshots
2016-08-31 22:31:45,212 INFO [ThriftRPCServer]
persist.HDFSTransactionStateStorage: No snapshot files found in
/tmp/tephra/snapshots
2016-08-31 22:31:45,214 INFO [ThriftRPCServer]
tephra.TransactionManager: Starting periodic timed-out transaction
cleanup every 10 seconds with default timeout of 60 seconds.
2016-08-31 22:31:45,215 INFO [ThriftRPCServer]
tephra.TransactionManager: Starting periodic snapshot thread,
frequency = 300 seconds, location = /tmp/tephra/snapshots
2016-08-31 22:31:45,215 INFO [ThriftRPCServer]
tephra.TransactionManager: Starting periodic Metrics Emitter
thread, frequency = 10000
2016-08-31 22:31:45,242 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: Starting RPC server for TTransactionServer
2016-08-31 22:31:45,255 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: Running RPC server for TTransactionServer
2016-08-31 22:31:45,255 INFO [ThriftRPCServer]
server.TThreadedSelectorServerWithFix: Starting
TThreadedSelectorServerWithFix
2016-08-31 22:31:45,359 INFO [leader-election-tx.service-leader]
distributed.TransactionService: Transaction Thrift Service started
successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165
<http://172.18.0.12:15165>
2016-08-31 22:32:43,325 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Unable to read additional data from server
sessionid 0x156e2ba5ba50004, likely server has closed socket,
closing socket connection and attempting reconnect
2016-08-31 22:32:43,427 INFO [leader-election-tx.service-leader]
zookeeper.LeaderElection: Disconnected from ZK:
m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-08-31 22:32:43,427 INFO [leader-election-tx.service-leader]
zookeeper.LeaderElection: Stepping down as leader due to
disconnect: m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-08-31 22:32:43,428 INFO [leader-election-tx.service-leader]
rpc.ThriftRPCServer: Request to stop RPC server for TTransactionServer
2016-08-31 22:32:43,430 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: Done running RPC server for TTransactionServer
2016-08-31 22:32:43,431 INFO [ThriftRPCServer]
tephra.TransactionManager: Shutting down gracefully...
2016-08-31 22:32:43,432 INFO [tx-clean-timeout]
tephra.TransactionManager: Exiting thread tx-clean-timeout
2016-08-31 22:32:43,432 INFO [tx-metrics]
tephra.TransactionManager: Exiting thread tx-metrics
2016-08-31 22:32:43,432 INFO [tx-snapshot]
tephra.TransactionManager: Writing final snapshot prior to shutdown
2016-08-31 22:32:43,510 INFO [tx-snapshot]
tephra.TransactionManager: Exiting thread tx-snapshot
2016-08-31 22:32:43,613 INFO [DefaultMetricsCollector STOPPING]
metrics.DefaultMetricsCollector: Stopped metrics reporter
2016-08-31 22:32:43,614 INFO [ThriftRPCServer]
tephra.TransactionManager: Took 181.4 ms to stop
2016-08-31 22:32:43,614 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: RPC server for TTransactionServer stopped.
2016-08-31 22:32:44,130 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181. Will not
attempt to authenticate using SASL (unknown error)
2016-08-31 22:32:44,130 WARN
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Session 0x156e2ba5ba50004 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-31 22:32:45,452 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>. Will not attempt to authenticate using
SASL (unknown error)
2016-08-31 22:32:45,453 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Socket connection established to
m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>, initiating session
2016-08-31 22:32:45,453 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>, sessionid = 0x156e2ba5ba50004,
negotiated timeout = 40000
2016-08-31 22:32:45,458 INFO [leader-election-tx.service-leader]
zookeeper.LeaderElection: Connected to ZK, running election:
m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-08-31 22:32:45,529 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: Starting RPC server for TTransactionServer
2016-08-31 22:32:45,530 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: Running RPC server for TTransactionServer
2016-08-31 22:32:45,530 INFO [ThriftRPCServer]
server.TThreadedSelectorServerWithFix: Starting
TThreadedSelectorServerWithFix
2016-08-31 22:32:45,540 INFO [leader-election-tx.service-leader]
distributed.TransactionService: Transaction Thrift Service started
successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165
<http://172.18.0.12:15165>
2016-08-31 22:35:32,201 ERROR [TTransactionServer-rpc-0]
thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at
org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at
org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-31 22:35:32,311 ERROR [TTransactionServer-rpc-0]
thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at
org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at
org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-31 22:35:32,717 ERROR [TTransactionServer-rpc-0]
thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at
org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at
org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-31 22:35:34,321 ERROR [TTransactionServer-rpc-0]
thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at
org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at
org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
If I run jps, I get the following:
bash-4.3# jps
771 Jps
138 HMaster
190 TransactionServiceMain
Cheers,
Francis
On 1/09/2016 4:01 AM, Thomas D'Silva wrote:
Can you check the Transaction Manager logs and see if there are
any error? Also can you do a jps and see confirm the Transaction
Manager is running ?
On Wed, Aug 31, 2016 at 2:12 AM, F21 <[email protected]
<javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
Just another update. Even though the logs says that the
transaction manager is not running, it is actually running.
I confirmed this by checking the output of ps and connecting
to the transaction manager:
bash-4.3# ps
PID USER TIME COMMAND
1 root 0:01 bash /run-hbase-phoenix.sh
137 hadoop 0:19 /usr/lib/jvm/java-1.8-openjdk/bin/java
-Dproc_master -XX:OnOutOfMemoryError=kill -9 %p
-XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:Ma
189 hadoop 0:08 /usr/lib/jvm/java-1.8-openjdk/bin/java
-XX:+UseConcMarkSweepGC -cp
/opt/hbase/bin/../lib/*:/opt/hbase/bin/../conf/:/opt/hbase/phoenix-c
542 root 0:00 /bin/bash
9035 root 0:00 sleep 1
9036 root 0:00 ps
bash-4.3# wget localhost:15165
Connecting to localhost:15165 (127.0.0.1:15165
<http://127.0.0.1:15165>)
wget: error getting response: Connection reset by peer
On 31/08/2016 3:25 PM, F21 wrote:
This only seems to be a problem when I have HBase running
in fully distributed mode (1 master, 1 regionserver and 1
zookeeper node in different docker images).
If I have HBase running in standalone mode with HBase and
Phoenix and the Query server in 1 docker image, it works
correctly.
On 31/08/2016 11:21 AM, F21 wrote:
I have HBase 1.2.2 and Phoenix 4.8.0 running on my
HBase master running on alpine linux with OpenJDK JRE 8.
This is my hbase-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl"
href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://mycluster/hbase</value>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>m9edd51-zookeeper.m9edd51</value>
</property>
<property>
<name>data.tx.snapshot.dir</name>
<value>/tmp/tephra/snapshots</value>
</property>
<property>
<name>data.tx.timeout</name>
<value>60</value>
</property>
<property>
<name>phoenix.transactions.enabled</name>
<value>true</value>
</property>
</configuration>
I am able to start the master correctly. I am also
able to create non-transactional table.
However, if I create a transactional table, I get
this error: ERROR [TTransactionServer-rpc-0]
thrift.ProcessFunction: Internal error processing
startShort
This is what I see in the logs:
2016-08-31 01:08:33,560 WARN
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Session 0x156de22abec0004 for
server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-31 01:08:33,616 INFO [DefaultMetricsCollector
STOPPING] metrics.DefaultMetricsCollector: Stopped
metrics reporter
2016-08-31 01:08:33,623 INFO [ThriftRPCServer]
tephra.TransactionManager: Took 170.7 ms to stop
2016-08-31 01:08:33,623 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: RPC server for
TTransactionServer stopped.
2016-08-31 01:08:34,776 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Opening socket connection to
server m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>. Will not attempt to
authenticate using SASL (unknown error)
2016-08-31 01:08:34,777 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Socket connection established
to m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>, initiating session
2016-08-31 01:08:34,778 INFO
[main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
zookeeper.ClientCnxn: Session establishment complete
on server m9edd51-zookeeper.m9edd51/172.18.0.2:2181
<http://172.18.0.2:2181>, sessionid =
0x156de22abec0004, negotiated timeout = 40000
2016-08-31 01:08:34,783 INFO
[leader-election-tx.service-leader]
zookeeper.LeaderElection: Connected to ZK, running
election: m9edd51-zookeeper.m9edd51 for
/tx.service/leader
2016-08-31 01:08:34,815 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: Starting RPC server for
TTransactionServer
2016-08-31 01:08:34,815 INFO [ThriftRPCServer]
rpc.ThriftRPCServer: Running RPC server for
TTransactionServer
2016-08-31 01:08:34,816 INFO [ThriftRPCServer]
server.TThreadedSelectorServerWithFix: Starting
TThreadedSelectorServerWithFix
2016-08-31 01:08:34,822 INFO
[leader-election-tx.service-leader]
distributed.TransactionService: Transaction Thrift
Service started successfully on
m9edd51-hmaster1.m9edd51/172.18.0.12:15165
<http://172.18.0.12:15165>
2016-08-31 01:10:42,830 ERROR
[TTransactionServer-rpc-0] thrift.ProcessFunction:
Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager
is not running.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at
org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at
org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown
Source)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at
org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-31 01:10:42,936 ERROR
[TTransactionServer-rpc-0] thrift.ProcessFunction:
Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager
is not running.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at
org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at
org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at
org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown
Source)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at
org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at
org.apache.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This seems very similar to PHOENIX-2504
(https://issues.apache.org/jira/browse/PHOENIX-2504
<https://issues.apache.org/jira/browse/PHOENIX-2504>),
but that appears to be fixed in 4.8.0?
Any ideas why this might be happening?
Cheers,
Francis
