Jialin Qiao created IOTDB-3795:
----------------------------------

             Summary: target file is not complete, and some source files is lost
                 Key: IOTDB-3795
                 URL: https://issues.apache.org/jira/browse/IOTDB-3795
             Project: Apache IoTDB
          Issue Type: Bug
            Reporter: Jialin Qiao
            Assignee: Liuxuxin


Meet error in compaction, then set system mode to read-only.
{code:java}
2022-07-09 17:45:20,943 [pool-39-IoTDB-RPC-Client-7] WARN  
o.a.i.d.metadata.MManager:2009 - meet error when check 
root.DT.BJ.FW.KT_J0CRR11FA010_CL, message: Path 
[root.DT.BJ.FW.KT_J0CRR11FA010_CL] does not exist 
2022-07-09 17:45:20,944 [pool-39-IoTDB-RPC-Client-7] WARN  
o.a.i.d.u.ErrorHandlingUtils:146 - Exception occurred while processing 
non-query.  
org.apache.iotdb.db.exception.BatchProcessException: Batch process 
failed:[TSStatus(code:304, 
message:org.apache.iotdb.db.exception.metadata.PathNotExistException: Path 
[KT_J0CRR11FA010_CL] does not exist)]
    at 
org.apache.iotdb.db.qp.executor.PlanExecutor.insert(PlanExecutor.java:1632)
    at 
org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:284)
    at 
org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:53)
    at 
org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeOperationSync(TSServiceImpl.java:2165)
    at 
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeOperationSync.getResult(TSIService.java:3989)
    at 
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeOperationSync.getResult(TSIService.java:3969)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
    at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:830)
2022-07-09 17:45:21,653 [pool-7-IoTDB-Compaction-10] INFO  
o.a.i.d.e.c.i.s.SizeTieredCompactionTask:95 - root.DT.HX-0 [Compaction] 
starting compaction task with 30 files 
2022-07-09 17:45:21,654 [pool-7-IoTDB-Compaction-10] INFO  
o.a.i.d.e.c.i.s.SizeTieredCompactionTask:112 - root.DT.HX-0 
[SizeTiredCompactionTask] Close the logger 
2022-07-09 17:45:21,654 [pool-7-IoTDB-Compaction-10] INFO  
o.a.i.d.e.c.i.s.SizeTieredCompactionTask:114 - root.DT.HX-0 [Compaction] 
compaction with [file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657151026844-6-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657158229818-7-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657165428104-8-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657172631386-9-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657179827208-10-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657187027013-11-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657194231019-12-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657201430483-13-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657208626434-14-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657215828924-15-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657223028454-16-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657230224566-17-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657237428049-18-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657244627481-19-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657251827900-20-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657259027626-21-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657266227844-22-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657273423548-23-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657280623088-24-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657287827377-25-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657295023561-26-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657302227862-27-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657309423854-28-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657316627249-29-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657323823301-30-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657331023417-31-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657338227391-32-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657345427054-33-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657352623586-34-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657359823237-35-0-0.tsfile,
 status: ] 
2022-07-09 17:45:21,670 [pool-7-IoTDB-Compaction-10] ERROR 
o.a.i.d.e.c.i.s.SizeTieredCompactionTask:196 - root.DT.HX-0 [Compaction] 
Throwable is caught during execution of SizeTieredCompaction, {} 
java.lang.IllegalArgumentException: Negative position
    at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:785)
    at 
org.apache.iotdb.tsfile.read.reader.LocalTsFileInput.read(LocalTsFileInput.java:90)
    at 
org.apache.iotdb.tsfile.read.TsFileSequenceReader.readTailMagic(TsFileSequenceReader.java:230)
    at 
org.apache.iotdb.tsfile.read.TsFileSequenceReader.loadMetadataSize(TsFileSequenceReader.java:203)
    at 
org.apache.iotdb.tsfile.read.TsFileSequenceReader.<init>(TsFileSequenceReader.java:140)
    at 
org.apache.iotdb.tsfile.read.TsFileSequenceReader.<init>(TsFileSequenceReader.java:123)
    at 
org.apache.iotdb.db.query.control.FileReaderManager.get(FileReaderManager.java:129)
    at 
org.apache.iotdb.db.engine.compaction.inner.utils.MultiTsFileDeviceIterator.<init>(MultiTsFileDeviceIterator.java:88)
    at 
org.apache.iotdb.db.engine.compaction.CompactionUtils.compact(CompactionUtils.java:98)
    at 
org.apache.iotdb.db.engine.compaction.inner.sizetiered.SizeTieredCompactionTask.doCompaction(SizeTieredCompactionTask.java:121)
    at 
org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.call(AbstractCompactionTask.java:67)
    at 
org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.call(AbstractCompactionTask.java:38)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:830)
2022-07-09 17:45:21,670 [pool-7-IoTDB-Compaction-10] WARN  
o.a.i.d.e.c.i.s.SizeTieredCompactionTask:200 - root.DT.HX-0 [Compaction] Start 
to handle exception 
2022-07-09 17:45:21,671 [pool-7-IoTDB-Compaction-10] INFO  
o.a.i.d.e.c.t.CompactionExceptionHandler:63 - root.DT.HX-0 
[Compaction][ExceptionHandler] inner space compaction start handling exception, 
source seqFiles is [], source unseqFiles is [file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657151026844-6-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657158229818-7-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657165428104-8-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657172631386-9-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657179827208-10-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657187027013-11-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657194231019-12-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657201430483-13-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657208626434-14-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657215828924-15-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657223028454-16-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657230224566-17-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657237428049-18-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657244627481-19-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657251827900-20-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657259027626-21-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657266227844-22-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657273423548-23-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657280623088-24-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657287827377-25-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657295023561-26-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657302227862-27-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657309423854-28-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657316627249-29-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657323823301-30-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657331023417-31-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657338227391-32-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657345427054-33-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657352623586-34-0-0.tsfile,
 status: , file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657359823237-35-0-0.tsfile,
 status: ]. 
2022-07-09 17:45:21,672 [pool-7-IoTDB-Compaction-10] ERROR 
o.a.i.d.e.c.t.CompactionExceptionHandler:237 - root.DT.HX-0 
[Compaction][ExceptionHandler] target file file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657359823237-35-1-0.inner,
 status:  is not complete, and some source files [file is 
/home/xdb/xdb_home/apache-iotdb-0.13.1-all-bin/sbin/../data/data/unsequence/root.DT.HX/0/0/1657359823237-35-0-0.tsfile,
 status: ] is lost, do nothing. Set allowCompaction to false 
2022-07-09 17:45:21,672 [pool-7-IoTDB-Compaction-10] ERROR 
o.a.i.d.e.c.t.CompactionExceptionHandler:99 - [Compaction][ExceptionHandler] 
Fail to handle inner space compaction exception, set allowCompaction to false 
in root.DT.HX-0 
2022-07-09 17:45:21,674 [pool-39-IoTDB-RPC-Client-7] WARN  
o.a.i.d.u.ErrorHandlingUtils:155 - Exception occurred while processing 
non-query.  
org.apache.iotdb.db.exception.StorageEngineReadonlyException: Database is 
read-only, and does not accept non-query operation now
    at 
org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:51)
    at 
org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeOperationSync(TSServiceImpl.java:2165)
    at 
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeOperationSync.getResult(TSIService.java:3989)
    at 
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeOperationSync.getResult(TSIService.java:3969)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
    at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:830)
2022-07-09 17:45:21,674 [pool-39-IoTDB-RPC-Client-6] WARN  
o.a.i.d.u.ErrorHandlingUtils:155 - Exception occurred while processing 
non-query.  
org.apache.iotdb.db.exception.StorageEngineReadonlyException: Database is 
read-only, and does not accept non-query operation now
    at 
org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:51)
    at 
org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeOperationSync(TSServiceImpl.java:2165)
    at 
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeOperationSync.getResult(TSIService.java:3989)
    at 
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeOperationSync.getResult(TSIService.java:3969)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
    at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:830) {code}
TODO:

(1) In the IoTDBConfig.setReadOnly() function, if it receive a true, we need to 
log the stack trace with the information "change system mode to read-only". To 
avoid missing the log outside.

if (readOnly) {

logger.error("change system mode to read-only", new RuntimeException());

}

(2) Check all places that set read-only, this is a dangerous operation in the 
deployment environment, and we should avoid using this if not necessary.

(3) Check why compaction lost file.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to