xinhuishuang opened a new issue, #12531: URL: https://github.com/apache/skywalking/issues/12531
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues. ### Apache SkyWalking Component OAP server (apache/skywalking) ### What happened version：skywalking oap 10.0.1 After the oap service is running for some time, the service is abnormal, with both binary deployment and docker deployment. ```java 2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.core.analysis.worker.MetricsAggregateWorker 145 [DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.6.Thread] ERROR [] - [10.0.1-6a9d727] Java heap space java.lang.OutOfMemoryError: Java heap space 2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.storage.plugin.jdbc.common.dao.JDBCHistoryDeleteDAO 55 [pool-7-thread-1] INFO [] - Deleting history data, ttl: 3, now: 1723599511829. Keep [20240812, 20240815] 2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.storage.plugin.jdbc.common.dao.JDBCHistoryDeleteDAO 55 [pool-7-thread-1] INFO [] - Deleting history data, ttl: 3, now: 1723599511829. Keep [20240812, 20240815] 2024-08-14 09:38:31,829 org.apache.skywalking.oap.server.storage.plugin.jdbc.common.dao.JDBCHistoryDeleteDAO 55 [pool-7-thread-1] INFO [] - Deleting history data, ttl: 3, now: 1723599511829. Keep [20240812, 20240815] 2024-08-14 09:38:31,830 io.netty.channel.DefaultChannelPipeline 1152 [armeria-eventloop-epoll-5-29] WARN [] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.channel.ChannelPipelineException: com.linecorp.armeria.server.HttpServerPipelineConfigurator.handlerAdded() has thrown an exception; removed. at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:624) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.108.Final.jar:4.1.108.Final] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.OutOfMemoryError: Java heap space Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.22.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.15.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.11.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.24.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.27.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.5.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.1.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.23.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.20.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HikariPool-1 housekeeper" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.2.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.18.Thread" 2024-08-14 09:40:36,979 io.netty.channel.DefaultChannelPipeline 1152 [armeria-eventloop-epoll-5-32] WARN [] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.channel.ChannelPipelineException: com.linecorp.armeria.server.HttpServerPipelineConfigurator.handlerAdded() has thrown an exception; removed. at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:624) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) [netty-transport-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.108.Final.jar:4.1.108.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.108.Final.jar:4.1.108.Final] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.OutOfMemoryError: Java heap space Exception in thread "pool-2-thread-1" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.28.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.31.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.7.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.16.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-2-thread-1" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.22.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.6.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.8.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-4-thread-1" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-10-thread-1" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.0.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-9-thread-1" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.31.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "grpc-default-executor-2433" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-2-thread-2" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-7-thread-1" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.1.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.7.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.28.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "armeria-eventloop-epoll-5-21" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.29.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.16.Thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "armeria-eventloop-epoll-5-29" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.13.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.29.Thread" Exception in thread "DataCarrier.METRICS_L1_AGGREGATION.BulkConsumePool.13.Thread" Exception in thread "pool-5-thread-1" Exception in thread "armeria-eventloop-epoll-5-22" Exception in thread "armeria-eventloop-epoll-5-27" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2024-08-14 09:43:19,898 io.netty.channel.AbstractChannelHandlerContext 355 [armeria-boss-http-*:12800] WARN [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.lang.OutOfMemoryError: Java heap space 2024-08-14 09:43:19,898 io.netty.channel.AbstractChannelHandlerContext 355 [armeria-eventloop-epoll-5-31] WARN [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.lang.OutOfMemoryError: Java heap space 2024-08-14 09:43:19,898 io.netty.channel.AbstractChannelHandlerContext 355 [grpc-default-worker-ELG-10-1] WARN [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.lang.OutOfMemoryError: Java heap space 2024-08-14 09:43:20,505 io.netty.channel.AbstractChannelHandlerContext 355 [armeria-eventloop-epoll-5-25] WARN [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.lang.OutOfMemoryError: Java heap space 2024-08-14 09:43:20,505 io.netty.util.concurrent.AbstractEventExecutor 168 [armeria-eventloop-epoll-5-28] WARN [] - A task raised an exception. Task: io.netty.channel.AbstractChannel$AbstractUnsafe$1@14f31166 java.lang.OutOfMemoryError: Java heap space 2024-08-14 09:43:21,164 io.netty.util.concurrent.AbstractEventExecutor 168 [armeria-eventloop-epoll-5-26] WARN [] - A task raised an exception. Task: io.netty.channel.AbstractChannel$AbstractUnsafe$1@5289fd5c java.lang.OutOfMemoryError: Java heap space Aug 14, 2024 9:43:23 AM io.grpc.netty.NettyServerHandler onStreamError WARNING: Stream Error io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:523) at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:921) at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:629) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:472) at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:510) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:255) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:166) at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188) at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Aug 14, 2024 9:43:23 AM io.grpc.netty.NettyServerTransport notifyTerminated INFO: Transport failed io.netty.channel.StacklessClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source) Caused by: java.lang.OutOfMemoryError: Java heap space Aug 14, 2024 9:43:23 AM io.grpc.netty.NettyServerHandler onStreamError WARNING: Stream Error io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:523) at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:921) at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:629) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:472) at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:510) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:255) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:166) at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188) at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Aug 14, 2024 9:43:25 AM io.grpc.netty.NettyServerHandler onStreamError WARNING: Stream Error io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517) at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153) at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217) at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432) at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:673) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Aug 14, 2024 9:43:25 AM io.grpc.netty.NettyServerHandler onStreamError WARNING: Stream Error io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036) at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992) at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517) at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153) at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217) at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432) at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:673) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "grpc-default-executor-2441" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "grpc-default-worker-ELG-10-2" 2024-08-14 12:14:34,091 io.netty.channel.AbstractChannelHandlerContext 355 [grpc-default-boss-ELG-9-1] WARN [] - An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.lang.OutOfMemoryError: Java heap space ``` ### What you expected to happen Memory overflow ### How to reproduce After the oap service is running for some time, the service is abnormal, with both binary deployment and docker deployment. Try to configure the jvm parameter in the docker deployment mode, and the same exception occurred. docker yaml `docker run -d --name=skywalking-oap \ --restart=always \ -p 11800:11800 -p 12800:12800 \ -e SW_OAP_JAVA_OPTS='-Xms4g -Xmx4g' \ -e "TZ=Asia/Shanghai" \ apache/skywalking-oap-server:10.0.1` `docker run -d --restart=always \ --name skywalking-ui \ --link skywalking-oap:skywalking-oap \ -p 8080:8080 \ -e SW_OAP_ADDRESS=http://skywalking-oap:12800 \ -e TZ=Asia/Shanghai \ apache/skywalking-ui:10.0.1` ### Anything else _No response_ ### Are you willing to submit a pull request to fix on your own? - [X] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
