刘珍 created IOTDB-3855:
-------------------------
Summary: [ MultiLeaderConsensus +wal ] After “No space left on
device”,wal file has empty file
Key: IOTDB-3855
URL: https://issues.apache.org/jira/browse/IOTDB-3855
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Haiming Zhu
Attachments: config.properties, image-2022-07-18-10-07-03-735.png,
ip62_datanode_log_all.tar.gz, ip66_datanode_logs.tar.gz,
ip68_datanode_logs.tar.gz
master_0715_8cd42fa
MultiLeaderConsensus,3副本3节点
磁盘空间不足(所有leader在1个节点上),(ip68)爆磁盘节点的wal文件有空文件:
!image-2022-07-18-10-07-03-735.png!
复现流程
1. 192.168.10.62/66/68 72C256G
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
schema_replication_factor=3
data_replication_factor=3
max_waiting_time_when_insert_blocked=3600000
query_timeout_threshold=36000000
2. benchmark 运行附件中的脚本
3.ip68爆磁盘
ip68磁盘空间不足:
2022-07-16 01:52:43,590 [pool-35-IoTDB-WAL-Sync(node-root.bm68_dn62.g_0-2)-1]
ERROR o.a.i.d.w.b.WALBuffer$SyncBufferTask:414 - Fail to sync wal
node-root.bm68_dn62.g_0-2's buffer, change system mode to read-only.
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
at org.apache.iotdb.db.wal.io.LogWriter.write(LogWriter.java:59)
at
org.apache.iotdb.db.wal.buffer.WALBuffer$SyncBufferTask.run(WALBuffer.java:412)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)