刘珍 created IOTDB-4986:
-------------------------
Summary: Too many IoTDB-DataNodeInternalRPC-Processor threads are
open
Key: IOTDB-4986
URL: https://issues.apache.org/jira/browse/IOTDB-4986
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
m_1118_3d5eeae
1. 启动3副本3C21D 集群
2. 顺序启动7Benchmark
3. 某个节点的datanode IoTDB-DataNodeInternalRPC-Processor 线程会开的很多，2k+
（慢慢会降下来），但是会偶现OOM
2022-11-18 14:26:48,320
[pool-22-IoTDB-DataNodeInternalRPC-Processor-374$20221118_062422_29227_16.1.0]
ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:234 - write locally failed.
TSStatus: TSStatus(code:506, subStatus:[]), message: null
2022-11-18 14:29:44,568 [DataNodeInternalRPC-Service]{color:red}* ERROR
o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread
DataNodeInternalRPC-Service-40
java.lang.OutOfMemoryError: unable to create native thread: possibly out of
memory or process/resource limits reached*{color}
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:803)
at
java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
at
java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
at
org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:155)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139)
at
org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258)
2022-11-18 14:29:53,751 [ClientRPC-Service] ERROR
o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread
ClientRPC-Service-42
java.lang.OutOfMemoryError: unable to create native thread: possibly out of
memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:803)
at
java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
at
java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
at
org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:155)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139)
at
org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258)
2022-11-18 14:30:11,736 [pool-6-IoTDB-Flush-4] ERROR
o.a.i.d.e.s.TsFileProcessor:1095 - root.test.g_0-6:
/data/iotdb/m_1118_3d5eeae/sbin/../data/datanode/data/unsequence/root.test.g_0/6/2538/1668752675355-5-0-0.tsfile
meet error when flushing a memtable, change system mode to error
java.lang.OutOfMemoryError: unable to create native thread: possibly out of
memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:803)
at
java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
at
java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
at
java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at
org.apache.iotdb.db.rescon.AbstractPoolManager.submit(AbstractPoolManager.java:56)
at
org.apache.iotdb.db.engine.flush.MemTableFlushTask.<init>(MemTableFlushTask.java:88)
at
org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:1082)
at
org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:108)
at
org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-11-18 14:30:11,736 [pool-6-IoTDB-Flush-4] ERROR
o.a.i.c.e.HandleSystemErrorStrategy:37 - Unrecoverable error occurs! Change
system status to read-only because handle_system_error is CHANGE_TO_READ_ONLY.
Only query statements are permitted!
java.lang.RuntimeException: System mode is set to READ_ONLY
at
org.apache.iotdb.commons.enums.HandleSystemErrorStrategy.handle(HandleSystemErrorStrategy.java:37)
at
org.apache.iotdb.commons.conf.CommonConfig.handleUnrecoverableError(CommonConfig.java:282)
at
org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:1100)
at
org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:108)
at
org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
查看哪个节点线程增多，可以通过日志，如
ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:193 - can't connect to node
TEndPoint(ip:172.16.2.15, port:9003)
就去ip15上去找。
测试环境
1. 私有云1期 172.16.2.2 ~ 32
启动3副本3C21D
2. 启动7benchmark
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
