FengQingxin created IOTDB-714:
---------------------------------
Summary: [No disk space]Can not restart IoTDB after disk space
cleanup
Key: IOTDB-714
URL: https://issues.apache.org/jira/browse/IOTDB-714
Project: Apache IoTDB
Issue Type: Bug
Reporter: FengQingxin
Hi IoTDB Developers：
I got an issue when I using iotdb-master[0.10.0-snapshot] to run aging
test.After almost 7 days[152Hours] there is not enough disk space available for
this operation. So we got this log:
{color:#ff8b00}_2020-05-25 00:09:12,933 [pool-7-IoTDB-RPC-Client-thread-53]
ERROR org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode:226 - Log node
root.group_11-1590335956489-7-0.tsfile sync failed, change system mode to
read-only_{color}
{color:#ff8b00}_java.io.IOException: No space left on device_{color}
{color:#ff8b00} _at sun.nio.ch.FileDispatcherImpl.write0(Native Method)_{color}
{color:#ff8b00} _at
sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)_{color}
{color:#ff8b00} _at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)_{color}
{color:#ff8b00} _at sun.nio.ch.IOUtil.write(IOUtil.java:65)_{color}
{color:#ff8b00} _at
sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.writelog.io.LogWriter.write(LogWriter.java:78)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode.sync(ExclusiveWriteLogNode.java:224)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode.putLog(ExclusiveWriteLogNode.java:106)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode.write(ExclusiveWriteLogNode.java:86)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.insertTablet(TsFileProcessor.java:203)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTabletToTsFileProcessor(StorageGroupProcessor.java:730)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTablet(StorageGroupProcessor.java:681)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.StorageEngine.insertTablet(StorageEngine.java:279)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:969)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.service.TSServiceImpl.insertTablet(TSServiceImpl.java:1171)_{color}
{color:#ff8b00} _at
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:21982020-05-25
00:09:14,769 [pool-1-IoTDB-Flush-ServerServiceImpl-thread-27] ERROR
org.apache.iotdb.db.engine.storagegroup.TsFileProcessor:542 - root.group_17:
1590336165770-7-0.tsfile meet error when flushing a memtable, change system
mode to read-only_{color}
{color:#ff8b00}_java.util.concurrent.ExecutionException:
org.apache.iotdb.db.exception.runtime.FlushRunTimeException:
java.io.IOException: No space left on device_{color}
{color:#ff8b00} _at
java.util.concurrent.FutureTask.report(FutureTask.java:122)_{color}
{color:#ff8b00} _at
java.util.concurrent.FutureTask.get(FutureTask.java:192)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:109)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:540)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:91)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.concurrent.WrappedRunnable.run(WrappedRunnable.java:32)_{color}
{color:#ff8b00} _at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)_{color}
{color:#ff8b00} _at
java.util.concurrent.FutureTask.run(FutureTask.java:266)_{color}
{color:#ff8b00} _at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)_{color}
{color:#ff8b00} _at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)_{color}
{color:#ff8b00} _at java.lang.Thread.run(Thread.java:748)_{color}
{color:#ff8b00}_Caused by:
org.apache.iotdb.db.exception.runtime.FlushRunTimeException:
java.io.IOException: No space left on device_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.flush.MemTableFlushTask.lambda$new$0(MemTableFlushTask.java:243)_{color}
{color:#ff8b00} _... 5 common frames omitted_{color}
{color:#ff8b00}_Caused by: java.io.IOException: No space left on device_{color}
{color:#ff8b00} _at java.io.FileOutputStream.writeBytes(Native Method)_{color}
{color:#ff8b00} _at
java.io.FileOutputStream.write(FileOutputStream.java:326)_{color}
{color:#ff8b00} _at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)_{color}
{color:#ff8b00} _at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)_{color}
{color:#ff8b00} _at
org.apache.iotdb.tsfile.write.writer.LocalTsFileOutput.getPosition(LocalTsFileOutput.java:61)_{color}
{color:#ff8b00} _at
org.apache.iotdb.tsfile.write.writer.TsFileIOWriter.getPos(TsFileIOWriter.java:325)_{color}
{color:#ff8b00} _at
org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl.writeAllPagesOfChunkToTsFile(ChunkWriterImpl.java:311)_{color}
{color:#ff8b00} _at
org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl.writeToFileWriter(ChunkWriterImpl.java:220)_{color}
{color:#ff8b00} _at
org.apache.iotdb.db.engine.flush.MemTableFlushTask.lambda$new$0(MemTableFlushTask.java:236)_{color}
{color:#ff8b00} _... 5 common frames omitted_{color}
At this time I can use start-cli.sh to connect to server. I tried to use
stop-server.sh to stop iotdb service,but it's failed.I saw it when I using
command 'jps' .
After I deleted some useless file in this server,I restart the server.Then I
tried to restart iotdb.But I got this error:
{color:#ff8b00}_2020-05-26 10:18:39,463
[pool-6-IoTDB-Recovery-Thread-Pool-thread-1] INFO
org.apache.iotdb.tsfile.read.TsFileSequenceReader:921 - TsFile
/data/fengqingxin/incubator-iotdb_local/server/target/iotdb-server-0.10.0-SNAPSHOT/sbin/./../data/data/sequence/root.group_9/2549/1590335934313-7-0.tsfile
self-check cannot proceed at position 526831142 recovered, because : reach the
end of the file._{color}
{color:#de350b}2020-05-26 10:18:39,818 [main] ERROR
org.apache.iotdb.db.concurrent.IoTDBDefaultThreadExceptionHandler:31 -
Exception in thread main-1{color}
{color:#de350b}java.lang.ExceptionInInitializerError: null{color}
{color:#de350b} at
org.apache.iotdb.db.engine.StorageEngine.getInstance(StorageEngine.java:103){color}
{color:#de350b} at
org.apache.iotdb.db.service.UpgradeSevice.countUpgradeFiles(UpgradeSevice.java:107){color}
{color:#de350b} at
org.apache.iotdb.db.service.UpgradeSevice.start(UpgradeSevice.java:60){color}
{color:#de350b} at
org.apache.iotdb.db.service.RegisterManager.register(RegisterManager.java:48){color}
{color:#de350b} at
org.apache.iotdb.db.service.IoTDB.setUp(IoTDB.java:100){color}
{color:#de350b} at
org.apache.iotdb.db.service.IoTDB.active(IoTDB.java:72){color}
{color:#de350b} at org.apache.iotdb.db.service.IoTDB.main(IoTDB.java:58){color}
{color:#de350b}Caused by:
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException:
StorageEngine failed to recover.{color}
{color:#de350b} at
org.apache.iotdb.db.engine.StorageEngine.<init>(StorageEngine.java:164){color}
{color:#de350b} at
org.apache.iotdb.db.engine.StorageEngine.<init>(StorageEngine.java:73){color}
{color:#de350b} at
org.apache.iotdb.db.engine.StorageEngine$InstanceHolder.<clinit>(StorageEngine.java:99){color}
{color:#de350b} ... 7 common frames omitted{color}
{color:#de350b}Caused by: java.util.concurrent.ExecutionException:
java.nio.BufferUnderflowException{color}
{color:#de350b} at
java.util.concurrent.FutureTask.report(FutureTask.java:122){color}
{color:#de350b} at
java.util.concurrent.FutureTask.get(FutureTask.java:192){color}
{color:#de350b} at
org.apache.iotdb.db.engine.StorageEngine.<init>(StorageEngine.java:162){color}
{color:#de350b} ... 9 common frames omitted{color}
{color:#de350b}Caused by: java.nio.BufferUnderflowException: null{color}
{color:#de350b} at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151){color}
{color:#de350b} at
org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readStringWithLength(ReadWriteIOUtils.java:529){color}
{color:#de350b} at
org.apache.iotdb.db.qp.physical.PhysicalPlan.readString(PhysicalPlan.java:125){color}
{color:#de350b} at
org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan.deserializeFrom(InsertTabletPlan.java:321){color}
{color:#de350b} at
org.apache.iotdb.db.qp.physical.PhysicalPlan$Factory.create(PhysicalPlan.java:153){color}
{color:#de350b} at
org.apache.iotdb.db.writelog.io.BatchLogReader.readLogs(BatchLogReader.java:52){color}
{color:#de350b} at
org.apache.iotdb.db.writelog.io.BatchLogReader.<init>(BatchLogReader.java:44){color}
{color:#de350b} at
org.apache.iotdb.db.writelog.io.SingleFileLogReader.hasNext(SingleFileLogReader.java:96){color}
{color:#de350b} at
org.apache.iotdb.db.writelog.io.MultiFileLogReader.hasNext(MultiFileLogReader.java:61){color}
{color:#de350b} at
org.apache.iotdb.db.writelog.recover.LogReplayer.replayLogs(LogReplayer.java:91){color}
{color:#de350b} at
org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:196){color}
{color:#de350b} at
org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:133){color}
{color:#de350b} at
org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:516){color}
{color:#de350b} at
org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:283){color}
{color:#de350b} at
org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.<init>(StorageGroupProcessor.java:263){color}
{color:#de350b} at
org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:152){color}
{color:#de350b} at
java.util.concurrent.FutureTask.run(FutureTask.java:266){color}
{color:#de350b} at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149){color}
{color:#de350b} at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624){color}
{color:#de350b} at java.lang.Thread.run(Thread.java:748){color}
IoTDB can not be started again!
Please help me to check this.
Thanks!
commit edea3a4c5d066cca16dba3797e9c0ce92d390345
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
