刘珍 created IOTDB-4556:
-------------------------
Summary: [ remove datanode ] ERROR
o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:287 - The consensus group
DataRegion[24] doesn't exist
Key: IOTDB-4556
URL: https://issues.apache.org/jira/browse/IOTDB-4556
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Jinrui Zhang
m_0929_71d5f65
SchemaRegion : ratis
DataRegion : multiLeader
均为3副本,3C5D
启动客户端bm写入,缩容期间写入不停。
bm运行40分钟,缩容节点1(ip72),1小时38分钟缩容成功。
mv 节点1的data ,logs,再上线。
缩容节点2(ip73,开始缩容的时间09-29 14:10),此节点不包含DataRegion[24]
DataRegion[24]在ip74,ip75,ip76
但是ip73 error :
2022-09-29 14:23:39,273
[pool-24-IoTDB-ClientRPC-Processor-2$20220929_062339_48081_4.1.0]
{color:#DE350B}*ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:287 - The
consensus group DataRegion[24] doesn't exist*{color}
2022-09-29 14:23:39,275 [MultiLeaderConsensusClientPool-selector-98] ERROR
o.a.i.c.m.l.IndexController:111 - {color:#DE350B}*failed to flush sync index.
cannot find previous version file. previous: 93500*{color}
2022-09-29 14:23:39,179 [pool-24-IoTDB-ClientRPC-Processor-45] WARN
o.a.i.d.u.ErrorHandlingUtils:62 - Status code: EXECUTE_STATEMENT_ERROR(400),
operation: insertTablet failed
java.lang.RuntimeException: org.apache.iotdb.commons.exception.IoTDBException:
There are no available RegionGroups currently, please check the status of
cluster DataNodes
at
org.apache.iotdb.db.mpp.plan.analyze.ClusterPartitionFetcher.getOrCreateDataPartition(ClusterPartitionFetcher.java:280)
at
org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitInsertTablet(AnalyzeVisitor.java:1236)
at
org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitInsertTablet(AnalyzeVisitor.java:150)
at
org.apache.iotdb.db.mpp.plan.statement.crud.InsertTabletStatement.accept(InsertTabletStatement.java:121)
at
org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:98)
at
org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:40)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:236)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.<init>(QueryExecution.java:138)
at
org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:100)
at
org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:133)
at
org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:160)
at
org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertTablet(ClientRPCServiceImpl.java:996)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:3512)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:3492)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.iotdb.commons.exception.IoTDBException: There are no
available RegionGroups currently, please check the status of cluster DataNodes
... 20 common frames omitted
测试环境
1. 192.168.10.72/73/74/75/76 48CPU384GB
3C : 72,73,74
5D : 72 ,73,74,75,76
集群配置
ConfigNode
MAX_HEAP_SIZE="8G"
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
schema_replication_factor=3
data_replication_factor=3
connection_timeout_ms=120000
DataNode
MAX_HEAP_SIZE="256G"
MAX_DIRECT_MEMORY_SIZE="32G"
connection_timeout_ms=120000
max_connection_for_internal_service=200
max_waiting_time_when_insert_blocked=600000
query_timeout_threshold=36000000
2. benchmark配置见附件
3. bm运行40分钟 缩容ip72
等待ip72 缩容完成,datanode进程退出
mv data logs
再次启动ip72
4. 缩容ip73 ,出现问题描述的ERROR 信息
--
This message was sent by Atlassian Jira
(v8.20.10#820010)