刘珍 created IOTDB-3365:
-------------------------

             Summary: Internal error processing fetchFragmentInstanceState
                 Key: IOTDB-3365
                 URL: https://issues.apache.org/jira/browse/IOTDB-3365
             Project: Apache IoTDB
          Issue Type: Bug
          Components: mpp-cluster
    Affects Versions: 0.14.0-SNAPSHOT
            Reporter: 刘珍
            Assignee: Jinrui Zhang
         Attachments: config.properties.create-schema, 
config.properties.insert, node3-fetchFIState-NPE_log-all-20220531.3.log.gz, 
node3_oom_log_all.log

master_0530_95884ad
3副本3节点,benchmark先创建好元数据,再启动benchmark写入数据,
1小时后,node3 持续抛异常:
2022-05-31 18:31:41,843 [pool-1-IoTDB-InternalServiceRPC-Client-32] ERROR 
o.a.t.ProcessFunction:47 - Internal error processing fetchFragmentInstanceState
java.lang.NullPointerException: null
        at 
org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.fetchFragmentInstanceState(InternalServiceImpl.java:177)
        at 
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$fetchFragmentInstanceState.getResult(InternalService.java:1220)
        at 
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$fetchFragmentInstanceState.getResult(InternalService.java:1200)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
        at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2022-05-31 18:31:41,845 [pool-6-IoTDB-MPPCoordinatorScheduled-1] ERROR 
o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class 
org.apache.thrift.TApplicationException, message Internal error processing 
fetchFragmentInstanceState
2022-05-31 18:31:41,853 [pool-6-IoTDB-MPPCoordinatorScheduled-1] ERROR 
o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal 
error processing fetchFragmentInstanceState, LocalizedMessage Internal error 
processing fetchFragmentInstanceState,
org.apache.thrift.TApplicationException: Internal error processing 
fetchFragmentInstanceState
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
        at 
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a.CGLIB$receiveBase$60(<generated>)
        at 
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a$$FastClassByCGLIB$$47a8a47d.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at 
org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
        at 
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a.receiveBase(<generated>)
        at 
org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_fetchFragmentInstanceState(InternalService.java:228)
        at 
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a.CGLIB$recv_fetchFragmentInstanceState$10(<generated>)


17.5小时后,node3 OOM:
2022-06-01 11:00:48,432 [pool-1-IoTDB-InternalServiceRPC-Client-1340] ERROR 
o.a.t.ProcessFunction:47 - Internal error processing sendFragmentInstance
java.lang.RuntimeException: cannot fetch schema, status is: 411, msg is: 
{color:red}*java.lang.OutOfMemoryError: Java heap space*{color}
        at 
org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:105)
        at 
org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:92)
        at 
org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchemaWithAutoCreate(ClusterSchemaFetcher.java:154)
        at 
org.apache.iotdb.db.mpp.plan.analyze.SchemaValidator.validate(SchemaValidator.java:52)
        at 
org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.sendFragmentInstance(InternalServiceImpl.java:151)
        at 
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1195)
        at 
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1175)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
        at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

测试流程:
1. 192.168.130.3/4/5   16C32G
记为node1,node2,node3
2. benchmark创建元数据
配置文件见附件,运行2次benchmark创建元数据的操作,因为第1次创建有异常,部分失败。
3. benchmark连node1 执行写入

2022-05-31 17:25:39,199 benchmark开始运行。
{color:red}*node3:2022-05-31 18:31:41,843*{color} 
[pool-1-IoTDB-InternalServiceRPC-Client-32] ERROR o.a.t.ProcessFunction:47 - 
Internal error processing fetchFragmentInstanceState
java.lang.NullPointerException: null
{color:red}*node2:2022-06-01 09:54:03,432*{color} 
[pool-1-IoTDB-InternalServiceRPC-Client-40] ERROR o.a.t.ProcessFunction:47 - 
Internal error processing fetchFragmentInstanceState
java.lang.NullPointerException: null

{color:red}*node3:2022-06-01 11:00:48,432*{color} 
[pool-1-IoTDB-InternalServiceRPC-Client-1340] ERROR o.a.t.ProcessFunction:47 - 
Internal error processing sendFragmentInstance
java.lang.RuntimeException: cannot fetch schema, status is: 411, msg is: 
java.lang.OutOfMemoryError: Java heap space




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Reply via email to