[
https://issues.apache.org/jira/browse/PHOENIX-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314054#comment-16314054
]
Thomas D'Silva commented on PHOENIX-4374:
-----------------------------------------
Looking through the logs it looks like delete namespace clienttest is being
called only once:
{code}
120826:2018-01-05 11:25:19,108 TRACE [ProcedureExecutor-1]
procedure.DeleteNamespaceProcedure: DeleteNamespaceProcedure
(Namespace=CLIENTTEST) id=178 owner=root
state=RUNNABLE:DELETE_NAMESPACE_PREPARE execute state=DELETE_NAMESPACE_PREPARE
120839:2018-01-05 11:25:19,218 TRACE [ProcedureExecutor-1]
procedure.DeleteNamespaceProcedure: DeleteNamespaceProcedure
(Namespace=CLIENTTEST) id=178 owner=root
state=FINISHED:DELETE_NAMESPACE_PREPARE rollback state=DELETE_NAMESPACE_PREPARE
120840:2018-01-05 11:25:19,319 INFO [ProcedureExecutor-1]
procedure2.ProcedureExecutor: Rolledback procedure DeleteNamespaceProcedure
(Namespace=CLIENTTEST) id=178 owner=root state=ROLLEDBACK exec-time=212msec
exception=org.apache.hadoop.hbase.NamespaceNotFoundException: CL
{code}
HMaster logs also show that all the tables are being deleted before the
namespace delete is called
{code}
120064:2018-01-05 11:24:57,166 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST:AGGREGATE
120474:2018-01-05 11:25:00,633 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST:EVENT
120669:2018-01-05 11:25:04,073 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST:IEVENT
120711:2018-01-05 11:25:07,526 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST:SNAPSHOT
120760:2018-01-05 11:25:10,895 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST:SUBSCRIPTION
120779:2018-01-05 11:25:14,321 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST:TOPIC_SUBSCRIPTION
120819:2018-01-05 11:25:17,777 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST:SCHEMA_MIGRATION
120825:2018-01-05 11:25:19,006 INFO
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=39389] master.HMaster:
Client=root//172.18.0.4 delete CLIENTTEST
{code}
I wonder if this is an HBase issue, were you able to repro the failure in HBase
1.2 ?
> Flakyness with Phoenix 4.13.0 and HBase 1.3.1: RuntimeException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.hadoop.hbase.NamespaceNotFoundException:
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-4374
> URL: https://issues.apache.org/jira/browse/PHOENIX-4374
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.13.0
> Reporter: Francis Chuang
> Attachments: 2017-12-29.m9edd51.log, 2018-01-03.m9edd51.log,
> 2018-01-05-hbase1.3.m9edd51.log,
> hbase--master-m9edd51-phoenix.m9edd51-2017-11-20-second-run.log,
> hbase--master-m9edd51-phoenix.m9edd51-2017-11-20.log,
> hbase--master-m9edd51-phoenix.m9edd51.log,
> protobufs-dump-2017-11-20-second-run.zip, protobufs-dump-2017-11-20.zip,
> root-queryserver-2017-11-20-second-run.log, root-queryserver-2017-11-20.log,
> root-queryserver.log,
> tephra-service--m9edd51-phoenix.m9edd51-2017-11-20-second-run.log,
> tephra-service--m9edd51-phoenix.m9edd51-2017-11-20.log,
> tephra-service--m9edd51-phoenix.m9edd51.log
>
>
> I am using the Phoenix Query Server via my [Go Avatica SQL
> driver|https://github.com/Boostport/avatica].
> In terms of my set up I am running Phoenix 4.13.0 and HBase 1.3.1 in docker
> with a single node HBase using local storage. The dockerfile is available
> here: https://github.com/Boostport/hbase-phoenix-all-in-one
> Today, I updated one of my projects to use the latest version of the above
> image (Phoenix 4.13.0 and HBase 1.3.1) and my integration tests against
> Phoenix + HBase have become extremely flaky. The tests use a mix of
> transactional and non-transactional tables.
> The flakyness is that random tests will fail with the same error. If I rerun
> the tests, they sometimes pass and sometimes fail, and it is not clear why
> this is happening.
> In all of these tests, I am doing the following:
> 1. Create the schema.
> 2. Create tables.
> 3. Insert, delete and read data.
> 4. Delete the tables and schema.
> This is the error I get when trying to drop the schema:
> {code:java}
> An error was encountered while processing your request: RuntimeException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
> INITTEST
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
> -> PhoenixIOException:
> org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
> INITTEST
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
> -> NamespaceNotFoundException:
> org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
> INITTEST
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
> -> RemoteWithExtrasException:
> org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
> INITTEST
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
> {code}
> I have also attached the following:
> - Logs from tephra
> - Logs from PQS/Avatica
> - Logs from HBase
> Further update:
> Rolled back to the HBase 1.2 Phoenix 4.11 image and tests are rock solid.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
