刘珍 created IOTDB-3965:
-------------------------
Summary: [ratis md5 ] ERROR
o.a.i.c.r.FileInfoWithDelayedMd5Computing:64 - compute file digest for
.tsfile.resource.md5.tmp failed due to {}
Key: IOTDB-3965
URL: https://issues.apache.org/jira/browse/IOTDB-3965
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Song Ziyang
RatisConsensus 3副本3C5D
缩容1个datanode(包含2个dataregion),缩容节点的datanode 报错,add new peer 失败:
2022-07-26 15:05:47,271 [ForkJoinPool.commonPool-worker-13] ERROR
o.a.i.c.r.FileInfoWithDelayedMd5Computing:64 - compute file digest for
/data2/cluster_test/shrink_0726_ratis/datanode/sbin/../data/consensus/data_region/47474747-4747-4747-4747-000100000004/sm/1_400002/sequence/root.test.g0_0/4/0/1658816468388-9-0-0.tsfile.md5.tmp
failed due to {}
java.io.FileNotFoundException:
/data2/cluster_test/shrink_0726_ratis/datanode/sbin/../data/consensus/data_region/47474747-4747-4747-4747-000100000004/sm/1_400002/sequence/root.test.g0_0/4/0/1658816468388-9-0-0.tsfile.md5.tmp
(No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at
org.apache.ratis.util.MD5FileUtil.computeMd5ForFile(MD5FileUtil.java:125)
at
org.apache.iotdb.consensus.ratis.FileInfoWithDelayedMd5Computing.lambda$new$0(FileInfoWithDelayedMd5Computing.java:61)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
2022-07-26 14:54:48,514 [pool-16-IoTDB-Region-Migrate-Pool-1] ERROR
o.a.i.d.s.RegionMigrateService$RegionMigrateTask:348 - add new peer
TEndPoint(ip:192.168.130.4, port:40010) for region DataRegion[5] failed, resp:
ConsensusGenericResponse{success=false}
exception=org.apache.iotdb.consensus.exception.RatisRequestFailedException:
Ratis request failed
复现流程:
1. 192.168.130.1/2/3/4/5
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
schema_replication_factor=3
data_replication_factor=3
confignode:
MAX_HEAP_SIZE="4G"
datanode:
MAX_HEAP_SIZE="16G"
max_waiting_time_when_insert_blocked=3600000
query_timeout_threshold=36000000
2. benchmark 在ip2
配置文件见附件
3. 缩容ip5 ,包含2个dataregion
./sbin/remove-datanode.sh 192.168.130.5:6667
ip5有1个dataregion做了1次snapshot
有1个dataregion leader 做了1次snapshot,ip5同步慢,还没有发生snapshot
查看ip5的log。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)