刘珍 created IOTDB-3365:
-------------------------
Summary: Internal error processing fetchFragmentInstanceState
Key: IOTDB-3365
URL: https://issues.apache.org/jira/browse/IOTDB-3365
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Jinrui Zhang
Attachments: config.properties.create-schema,
config.properties.insert, node3-fetchFIState-NPE_log-all-20220531.3.log.gz,
node3_oom_log_all.log
master_0530_95884ad
3副本3节点,benchmark先创建好元数据,再启动benchmark写入数据,
1小时后,node3 持续抛异常:
2022-05-31 18:31:41,843 [pool-1-IoTDB-InternalServiceRPC-Client-32] ERROR
o.a.t.ProcessFunction:47 - Internal error processing fetchFragmentInstanceState
java.lang.NullPointerException: null
at
org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.fetchFragmentInstanceState(InternalServiceImpl.java:177)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$fetchFragmentInstanceState.getResult(InternalService.java:1220)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$fetchFragmentInstanceState.getResult(InternalService.java:1200)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-31 18:31:41,845 [pool-6-IoTDB-MPPCoordinatorScheduled-1] ERROR
o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class
org.apache.thrift.TApplicationException, message Internal error processing
fetchFragmentInstanceState
2022-05-31 18:31:41,853 [pool-6-IoTDB-MPPCoordinatorScheduled-1] ERROR
o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal
error processing fetchFragmentInstanceState, LocalizedMessage Internal error
processing fetchFragmentInstanceState,
org.apache.thrift.TApplicationException: Internal error processing
fetchFragmentInstanceState
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a.CGLIB$receiveBase$60(<generated>)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a$$FastClassByCGLIB$$47a8a47d.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at
org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a.receiveBase(<generated>)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_fetchFragmentInstanceState(InternalService.java:228)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$4ea7c99a.CGLIB$recv_fetchFragmentInstanceState$10(<generated>)
17.5小时后,node3 OOM:
2022-06-01 11:00:48,432 [pool-1-IoTDB-InternalServiceRPC-Client-1340] ERROR
o.a.t.ProcessFunction:47 - Internal error processing sendFragmentInstance
java.lang.RuntimeException: cannot fetch schema, status is: 411, msg is:
{color:red}*java.lang.OutOfMemoryError: Java heap space*{color}
at
org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.executeSchemaFetchQuery(ClusterSchemaFetcher.java:105)
at
org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchema(ClusterSchemaFetcher.java:92)
at
org.apache.iotdb.db.mpp.plan.analyze.ClusterSchemaFetcher.fetchSchemaWithAutoCreate(ClusterSchemaFetcher.java:154)
at
org.apache.iotdb.db.mpp.plan.analyze.SchemaValidator.validate(SchemaValidator.java:52)
at
org.apache.iotdb.db.service.thrift.impl.InternalServiceImpl.sendFragmentInstance(InternalServiceImpl.java:151)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1195)
at
org.apache.iotdb.mpp.rpc.thrift.InternalService$Processor$sendFragmentInstance.getResult(InternalService.java:1175)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
测试流程:
1. 192.168.130.3/4/5 16C32G
记为node1,node2,node3
2. benchmark创建元数据
配置文件见附件,运行2次benchmark创建元数据的操作,因为第1次创建有异常,部分失败。
3. benchmark连node1 执行写入
2022-05-31 17:25:39,199 benchmark开始运行。
{color:red}*node3:2022-05-31 18:31:41,843*{color}
[pool-1-IoTDB-InternalServiceRPC-Client-32] ERROR o.a.t.ProcessFunction:47 -
Internal error processing fetchFragmentInstanceState
java.lang.NullPointerException: null
{color:red}*node2:2022-06-01 09:54:03,432*{color}
[pool-1-IoTDB-InternalServiceRPC-Client-40] ERROR o.a.t.ProcessFunction:47 -
Internal error processing fetchFragmentInstanceState
java.lang.NullPointerException: null
{color:red}*node3:2022-06-01 11:00:48,432*{color}
[pool-1-IoTDB-InternalServiceRPC-Client-1340] ERROR o.a.t.ProcessFunction:47 -
Internal error processing sendFragmentInstance
java.lang.RuntimeException: cannot fetch schema, status is: 411, msg is:
java.lang.OutOfMemoryError: Java heap space
--
This message was sent by Atlassian Jira
(v8.20.7#820007)