Francis Chuang created PHOENIX-4374:
---------------------------------------
Summary: Flakyness with Phoenix 4.13.0 and HBase 1.3.1:
RuntimeException: org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.NamespaceNotFoundException:
Key: PHOENIX-4374
URL: https://issues.apache.org/jira/browse/PHOENIX-4374
Project: Phoenix
Issue Type: Bug
Affects Versions: 4.13.0
Reporter: Francis Chuang
I am using the Phoenix Query Server via my [Go Avatica SQL
driver|https://github.com/Boostport/avatica].
In terms of my set up I am running Phoenix 4.13.0 and HBase 1.3.1 in docker
with a single node HBase using local storage. The dockerfile is available here:
https://github.com/Boostport/hbase-phoenix-all-in-one
Today, I updated one of my projects to use the latest version of the above
image (Phoenix 4.13.0 and HBase 1.3.1) and my integration tests against Phoenix
+ HBase has become extremely flaky. The tests use a mix of transactional and
non-transactional tables.
The flakyness is that random tests will fail with the same error. If I rerun
the tests, they sometimes pass and sometimes fail, and it is not clear why this
is happening.
In all of these tests, I am doing the following:
1. Create the schema.
2. Create tables.
3. Insert, delete and read data.
4. Delete the tables and schemas.
This is the error I get when trying to drop the schema:
{code:java}
An error was encountered while processing your request: RuntimeException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
at
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
at
org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
at
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
-> PhoenixIOException:
org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
at
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
at
org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
at
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
-> NamespaceNotFoundException:
org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
at
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
at
org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
at
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
-> RemoteWithExtrasException:
org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
at
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
at
org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
at
org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
at
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
at
org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
{code}
I have also attached the following:
- Logs from tephra
- Logs from PQS/Avatica
- Logs from HBase
[^attachment-name.zip]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
