thisAmateur opened a new issue, #5147:
URL: https://github.com/apache/servicecomb-java-chassis/issues/5147
### Steps to Reproduce
### 环境信息:
我们业务现网用的jdk:
bisheng jdk 21.0.10+7
我本地用的jdk:
openjdk version "21.0.8" 2025-07-15 LTS
OpenJDK Runtime Environment Microsoft-11933218 (build 21.0.8+9-LTS)
useG1GC
### 问题场景:
微服务A调微服务B的一个sse接口,其中微服务A是一个edge服务。
当这个sse接口的响应报文比较大的时候,比如累计几十KB或者几百KB时,微服务A就会容易触发OOM错误。
微服务A的OOM报错堆栈:
```properties
java.lana.Error: java.lang.reflect.InvocationTargetException
at
io.netty.util.internal.PlatformDependentO.allocateUninitializedArray(PlatformDependent0.java:718)
at
io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:329)
at
io.netty.buffer.UnpooledUnsafeHeapByteBuf.allocateArray(UnpooledUnsafeHeapByteBuf,java:
39)
at io.netty.buffer.AbstractByteBuf.ensureWritable0
(AbstractByteBuf.java: 305)
at
io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java: 305)
at
io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1081)
at
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor.readALineOfBytesFromBuffer(ProduceEventStreamProcessor.java:339)
at
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor.processAllAvailableBufferLines
(ProduceEventStreamProcessor.java:260)
at
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor.bufferReadCacheAndProcessLines
(ProduceEventStreamProcessor.java:175)
at
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor.bufferReadCacheAndProcessLines
(ProduceEventStreamProcessor.java:175)
at
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor.innerLoop(ProduceEventStreamProcessor.java:157)
at
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor.doDecodeResponse
(ProduceEventStreamProcessor.java:125)
at
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor.decodeResponse
(ProduceEventStreamProcessor.java:395)
at
org.apache.servicecomb.transport.rest.client.http.DefaultHttpClientFilter.extractFlowableBody(DefaultHttpClientFilter.java:157)
at
org.apache.servicecomb.transport.rest.client.http.DefaultHttpClientFilter.lambda$extractResponse$0
(DefaultHttpClientFilter.java: 119)
at
io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.drain(FlowableConcatMap.java:
265)
at
io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$BaseConcatMapSubscriber.onNext(FlowableConcatMap.java:158)
at
io.vertx.rxjava3.impl.FlowableReadStream.lambda$subscribeActual$2(FlowableReadStream.java:83)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
at
io.vertx.core.http.impl.HttpEventHandler.handleChunk(HttpEventHandler.java:51)
at
io.vertx.core.http.impl.HttpEventHandler.handleChunk(HttpEventHandler.java:51)
at
io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:239)
at
io.vertx.core.http.impl.HttplxClientConnection$StreamImpl.lambda$new$0(HttplxClientConnection.java:456)
at
io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279)
at
io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(HttplxClientConnection.java:754)
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.
java:166)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:
30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.reflect.InvocationTargetException
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at
io.netty.util.internal.PlatformDependentO.allocateUninitializedArray(PlatformDependent0.java:
714)
... 36 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at
java.base/jdk.internal.misc.Unsafe.allocateUninitializedArray(Unknown Source)
at
java.base/java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(DirectMethodHandle$Holder)
at
java.base/java.lang.invoke.LambdaForm$MH/0x0000007801fc8c00.invoke(LambdaForm$MH)
at
java.base/java.lang.invoke.LambdaForm$MH/0x0000007801002800.invokeExact_MT(LambdaForm$MH)
```
### 问题分析:
org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor#doDecodeResponse中定义个一个128长度的buffer,来从网络流中读数据。while中每读一次,就会走到处理方法中处理。当后端服务返回的数据比较大的时候,128长度里,肯定是没有换行的。需要读非常多次,才能读到换行。
然后
在org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor#readALineOfBytesFromBuffer
这个类里,在匹配一个新行的时候,如果没匹配到,会把已经读过的字节数组数据,重新写回到org.apache.servicecomb.common.rest.codec.produce.ProduceEventStreamProcessor#buffer中,导致buffer的空间急剧膨胀。
实测后端返回170KB左右的报文,这个buffer会占用56MB左右的空间。
这部分空间,在jvm中直接分配到了humongous
region,而我们sse接口,往往一次请求,返回的时间还比较长,当这种大响应的请求并发比较多的时候,jvm堆中老年代内存直接就满了。从而触发OOM。
### Expected Behavior
预期服务正常运行,SSE场景下,ProduceEventStreamProcessor中的变量都分配在eden区,通过younggc即可快速回收,不会导致老年代满,不会导致OOM
### Servicecomb Version
2.9.3
### Additional Context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]