刘珍 created IOTDB-3659:
-------------------------
Summary: ERROR o.a.i.d.s.t.i.InternalServiceImpl:182 - Something
wrong happened while calling consensus layer's write API.
Key: IOTDB-3659
URL: https://issues.apache.org/jira/browse/IOTDB-3659
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Reporter: 刘珍
Assignee: Yongzao Dan
Attachments: image-2022-06-27-14-04-38-537.png
StandAloneConsensus， 3C3D
启动顺序
node1="172.20.70.13"
node2="172.20.70.16"
node3="172.20.70.14"
RegionID=1的dataregion在ip16，但是ip16报：
2022-06-27 10:48:11,852 [pool-1-IoTDB-InternalServiceRPC-Client-93]
{color:red}*ERROR o.a.i.d.s.t.i.InternalServiceImpl:182 - Something wrong
happened while calling consensus layer's write API.
org.apache.iotdb.consensus.exception.ConsensusGroupNotExistException: The
consensus group {color:red}DataRegion[1]{color} doesn't exist*{color}
at
org.apache.iotdb.consensus.ratis.RatisConsensus.write(RatisConsensus.java:161)
at
org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.sendPlanNode(InternalServiceImpl.java:170)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendPlanNode.getResult(InternalService.java:1284)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendPlanNode.getResult(InternalService.java:1264)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
复现流程
2022-06-25 14:19:11,809启动集群，ip 16 ，datanode启动失败，但是加入集群成功。
继续启动benchmark， 连ip14 执行写入和查询（常规长测配置）。
ip 14 还一直去连ip16：
2022-06-26 22:16:02,274 [20220626_141602_01235_5.1.0-72] WARN
o.a.i.c.c.ClientManager:60 - Borrow client from pool for node
TEndPoint(ip:172.20.70.16, port:9003) failed.
net.sf.cglib.core.CodeGenerationException:
org.apache.thrift.transport.TTransportException-->java.net.ConnectException:
Connection refused (Connection refused)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:235)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
at net.sf.cglib.proxy.Enhancer.createUsingReflection(Enhancer.java:639)
at net.sf.cglib.proxy.Enhancer.firstInstance(Enhancer.java:538)
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:231)
at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
at
org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.newErrorHandler(SyncThriftClientWithErrorHandler.java:48)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:127)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:105)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
at
org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:50)
at
org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:175)
at
org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:163)
at
org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:144)
at
org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:93)
at
org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:95)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:215)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:169)
at
org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:131)
at
org.apache.iotdb.db.service.thrift.impl.DataNodeTSIServiceImpl.insertTablet(DataNodeTSIServiceImpl.java:911)
at
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:3328)
at
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:3308)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:243)
at
org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:91)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient.<init>(SyncDataNodeInternalServiceClient.java:63)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$1eb1dc11.<init>(<generated>)
at sun.reflect.GeneratedConstructorAccessor11.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:228)
... 30 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:238)
... 37 common frames omitted
2022-06-27 10:40 左右kill ip68的datanode，再重启，ip68 ：
2022-06-27 10:48:11,852 [pool-1-IoTDB-InternalServiceRPC-Client-93] ERROR
o.a.i.d.s.t.i.InternalServiceImpl:182 -{color:red} Something wrong happened
while calling consensus layer's write API.
org.apache.iotdb.consensus.exception.ConsensusGroupNotExistException: The
consensus group *DataRegion[1] *doesn't exist{color}
at
org.apache.iotdb.consensus.ratis.RatisConsensus.write(RatisConsensus.java:161)
at
org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.sendPlanNode(InternalServiceImpl.java:170)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendPlanNode.getResult(InternalService.java:1284)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendPlanNode.getResult(InternalService.java:1264)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ip14 cli执行show regions
!image-2022-06-27-14-04-38-537.png!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
