刘珍 created IOTDB-4703:
-------------------------
Summary: 3rep/3C3D , stop 1 datanode ,delete storage group root.**
failed
Key: IOTDB-4703
URL: https://issues.apache.org/jira/browse/IOTDB-4703
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Gaofei Cao
m_1019_f2ffb49
3rep ,3C3D
stop ip73 datanode , ip72 cli execute "delete storage group root.**"
ip72 datanode error log：
2022-10-21 09:16:09,286
[pool-26-IoTDB-ClientRPC-Processor-987$20221021_011534_06996_3] ERROR
o.a.i.d.m.p.e.c.e.ClusterConfigTaskExecutor:248 - Failed to execute delete
storage group [root.**] in config node, status is TSStatus(code:506,
subStatus:[TSStatus(code:400, {color:#DE350B}message:Invalidate cache failed),
TSStatus(code:400, message:Invalidate cache failed), TSStatus(code:400,
message:Invalidate cache failed)]).{color}
2022-10-21 09:16:09,288 [pool-23-IoTDB-MPPCoordinator-17] ERROR
o.a.i.d.m.p.e.c.ConfigExecution:124 - Failures happened during running
ConfigExecution.
org.apache.iotdb.commons.exception.IoTDBException: null
at
org.apache.iotdb.db.mpp.plan.execution.config.executor.ClusterConfigTaskExecutor.deleteStorageGroup(ClusterConfigTaskExecutor.java:252)
at
org.apache.iotdb.db.mpp.plan.execution.config.metadata.DeleteStorageGroupTask.execute(DeleteStorageGroupTask.java:41)
at
org.apache.iotdb.db.mpp.plan.execution.config.ConfigExecution.start(ConfigExecution.java:99)
at
org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:146)
at
org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatement(ClientRPCServiceImpl.java:601)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3037)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3017)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
ip72 confignode error log：
2022-10-21 09:22:03,847 [ProcExecWorker-2] ERROR
o.a.i.c.c.s.SyncDataNodeClientPool:113 - INVALIDATE_SCHEMA_CACHE failed on
DataNode TEndPoint(ip:192.168.10.73, port:9003)
java.io.IOException: Borrow client from pool for node
TEndPoint(ip:192.168.10.73, port:9003) failed.
at
org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:61)
at
org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool.sendSyncRequestToDataNodeWithRetry(SyncDataNodeClientPool.java:68)
at
org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv.invalidateCache(ConfigNodeProcedureEnv.java:147)
at
org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:90)
at
org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:47)
at
org.apache.iotdb.confignode.procedure.impl.statemachine.StateMachineProcedure.execute(StateMachineProcedure.java:186)
at
org.apache.iotdb.confignode.procedure.Procedure.doExecute(Procedure.java:365)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:414)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:373)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor.access$300(ProcedureExecutor.java:50)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:741)
Caused by: net.sf.cglib.core.CodeGenerationException:
org.apache.thrift.transport.TTransportException-->java.net.ConnectException:
Connection refused (Connection refused)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:235)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
at net.sf.cglib.proxy.Enhancer.createUsingReflection(Enhancer.java:639)
at net.sf.cglib.proxy.Enhancer.firstInstance(Enhancer.java:538)
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:231)
at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
at
org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.newErrorHandler(SyncThriftClientWithErrorHandler.java:48)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:127)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:105)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
at
org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:50)
... 10 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:243)
at
org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:91)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient.<init>(SyncDataNodeInternalServiceClient.java:63)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$5e1ad4cd.<init>(<generated>)
at
jdk.internal.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:228)
... 23 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
at
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
at
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:608)
at org.apache.thrift.transport.TSocket.open(TSocket.java:238)
... 30 common frames omitted
2022-10-21 09:22:10,230 [ProcExecWorker-14] ERROR
o.a.i.c.p.e.ConfigNodeProcedureEnv:158 - Invalidate cache failed, invalidate
partition cache status is TSStatus(code:709, message:All retry failed due to:
Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003)
failed.), invalidate schema cache status is TSStatus(code:709, message:All
retry failed due to: Borrow client from pool for node
TEndPoint(ip:192.168.10.73, port:9003) failed.)
2022-10-21 09:22:10,235 [ProcExecWorker-1] ERROR
o.a.i.c.c.s.SyncDataNodeClientPool:113 - INVALIDATE_PARTITION_CACHE failed on
DataNode TEndPoint(ip:192.168.10.73, port:9003)
java.io.IOException: Borrow client from pool for node
TEndPoint(ip:192.168.10.73, port:9003) failed.
at
org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:61)
at
org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool.sendSyncRequestToDataNodeWithRetry(SyncDataNodeClientPool.java:68)
at
org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv.invalidateCache(ConfigNodeProcedureEnv.java:153)
at
org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:90)
at
org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:47)
at
org.apache.iotdb.confignode.procedure.impl.statemachine.StateMachineProcedure.execute(StateMachineProcedure.java:186)
at
org.apache.iotdb.confignode.procedure.Procedure.doExecute(Procedure.java:365)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:414)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:373)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor.access$300(ProcedureExecutor.java:50)
at
org.apache.iotdb.confignode.procedure.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:741)
Caused by: net.sf.cglib.core.CodeGenerationException:
org.apache.thrift.transport.TTransportException-->java.net.ConnectException:
Connection refused (Connection refused)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:235)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
at net.sf.cglib.proxy.Enhancer.createUsingReflection(Enhancer.java:639)
at net.sf.cglib.proxy.Enhancer.firstInstance(Enhancer.java:538)
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:231)
at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
at
org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.newErrorHandler(SyncThriftClientWithErrorHandler.java:48)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:127)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:105)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
at
org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:50)
... 10 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:243)
at
org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:91)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient.<init>(SyncDataNodeInternalServiceClient.java:63)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$5e1ad4cd.<init>(<generated>)
at
jdk.internal.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:228)
... 23 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
at
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
at
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:608)
at org.apache.thrift.transport.TSocket.open(TSocket.java:238)
... 30 common frames omitted
2022-10-21 09:22:10,236 [ProcExecWorker-1] ERROR
o.a.i.c.p.e.ConfigNodeProcedureEnv:158 - Invalidate cache failed, invalidate
partition cache status is TSStatus(code:709, message:All retry failed due to:
Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003)
failed.), invalidate schema cache status is TSStatus(code:709, message:All
retry failed due to: Borrow client from pool for node
TEndPoint(ip:192.168.10.73, port:9003) failed.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
