[
https://issues.apache.org/jira/browse/IOTDB-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558939#comment-17558939
]
Yuan Tian commented on IOTDB-3483:
----------------------------------
这个异常只要不影响查询正确性，应该就没有问题。因为目前我们并没有一个优雅的结束查询的方式，所以有可能下游查询结束了，去close整个查询的时候，上游的FragmentInstance还在运行，这时候强行终止的时候，就会抛这个异常。
count devices那个问题，可以复测一下，上周钰坤解了一个元数据查询那边的问题，不确定能不能fix这个issue，如果还有问题，可以转给钰坤
> Internal error processing onEndOfDataBlockEvent / count devices returns
> "Empty set"
> ------------------------------------------------------------------------------------
>
> Key: IOTDB-3483
> URL: https://issues.apache.org/jira/browse/IOTDB-3483
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
> Affects Versions: 0.14.0-SNAPSHOT
> Reporter: 刘珍
> Assignee: Yuan Tian
> Priority: Major
> Attachments: IOTDB-3483.zip, config.properties,
> image-2022-06-14-15-20-52-189.png, iotdb-confignode.properties,
> iotdb-engine.properties
>
>
> master_0613_38f3332
> 3节点3副本
> DEVICE_NUMBER=1000
> SENSOR_NUMBER=6000
> CLIENT_NUMBER=100
> GROUP_NUMBER=1
> BATCH_SIZE_PER_WRITE=10
> LOOP=10000
> OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0
> 3小时20分钟后，datanode error：
> 2022-06-14 14:17:30,613 [20220614_061627_19086_2.1.0-38] ERROR
> o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class
> org.apache.thrift.TApplicationException, message Internal error processing
> onEndOfDataBlockEvent
> 2022-06-14 14:17:30,614 [20220614_061657_19097_2.1.0-39] ERROR
> o.a.i.d.m.e.driver.Driver:195 - Failed to execute fragment instance
> 20220614_061657_19097_2.1.0
> java.lang.IllegalStateException: Sink handle is aborted.
> at
> org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at
> org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at
> org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> 2022-06-14 14:17:30,614 [20220614_061627_19086_2.1.0-38] ERROR
> o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal
> error processing onEndOfDataBlockEvent, LocalizedMessage Internal error
> processing onEndOfDataBlockEvent,
> org.apache.thrift.TApplicationException: Internal error processing
> onEndOfDataBlockEvent
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$receiveBase$20(<generated>)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> at
> org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.receiveBase(<generated>)
> at
> org.apache.iotdb.mpp.rpc.thrift.DataBlockService$Client.recv_onEndOfDataBlockEvent(DataBlockService.java:136)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$recv_onEndOfDataBlockEvent$13(<generated>)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> at
> org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.recv_onEndOfDataBlockEvent(<generated>)
> at
> org.apache.iotdb.mpp.rpc.thrift.DataBlockService$Client.onEndOfDataBlockEvent(DataBlockService.java:123)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$onEndOfDataBlockEvent$16(<generated>)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> at
> org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> at
> org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.onEndOfDataBlockEvent(<generated>)
> at
> org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.sendEndOfDataBlockEvent(SinkHandle.java:174)
> at
> org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.setNoMoreTsBlocks(SinkHandle.java:198)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.closeAndDestroyOperators(Driver.java:338)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.destroyIfNecessary(Driver.java:312)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:277)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at
> org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at
> org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> 集群时间节点的细节信息：
> 2022-06-14 10:50:55 启动benchmark
> 2022-06-14 10:58:30,935 bm创建元数据成功
> 查看3个节点的mlog.bin , ip62的元数据先落盘了一部分 (其余2个节点完全落盘）
> !image-2022-06-14-15-20-52-189.png!
> 50多分钟后ip62的元数据刷盘：
> liuzhen@fit-62:/data/liuzhen_test/master_0613_38f3332/datanode$ find ./data
> -name mlog.bin|xargs ls -lort
> -rw-rw-r-- 1 liuzhen 430230000 Jun 14 11:45
> ./data/system/schema/root.bm68_dn62.g_0/0/mlog.bin
> count devices;
> count timeseries ； 结果正确。
> 出现异常后：
> {color:red}count devices 返回Empty set.{color}
> 还有其它异常：
> 2022-06-14 14:17:30,614 [20220614_061657_19097_2.1.0-39] ERROR
> o.a.i.d.m.e.driver.Driver:195 - Failed to execute fragment instance
> 20220614_061657_19097_2.1.0
> java.lang.IllegalStateException: Sink handle is aborted.
> at
> org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at
> org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at
> org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> 2022-06-14 14:17:30,617 [Worker-Thread-3] ERROR
> o.a.i.d.m.e.s.AbstractDriverThread:61 - Executor Worker-Thread-3 processes
> failed
> java.lang.RuntimeException: Driver was interrupted
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:207)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at
> org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at
> org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
> at
> org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
> ... 5 common frames omitted
> Caused by: java.lang.Exception: Interrupted By
> at java.lang.Thread.getStackTrace(Thread.java:1559)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:444)
> at
> org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:150)
> at
> org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:114)
> at
> org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
> at
> org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 复现流程：
> 1. 192.168.10.62/66/68 72C256G
> 2. bm 连ip62 （配置见附件）
> 3. 各节点datanode log见附件。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
