Hi there, I'm working on a proxy server for a set of services that use gRPC. I've been running a load tester with 1 million concurrent clients against 70 server tasks, so the number of sockets per server is not anything crazy. In terms of CPU capacity the servers are over-provisioned and running at about 30% CPU usage.
After the test has been running for a few days, some servers fail. Basically, they get stuck in GC. Logs are filled with the following: ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. We also see the following in the logs (our servers don't shutdown themselves unless they get a SIGTERM and at that point they would have 60 seconds to clean up nicely, so we aren't triggering server shutdowns in our code, since the servers keep running): io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug! at io.grpc.Status.asRuntimeException(Status.java:539) at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:757) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:736) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.util.concurrent.RejectedExecutionException: event executor terminated Has anyone else had similar issues? Best, Edvard -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9a362003-177e-462c-944e-60194637ee96n%40googlegroups.com.
