Hi,
I'm using gRPC in a new Scala service and I'm seeing unexpectedly high CPU
utilization. I see this high utilization in our production workload but
also am able to reproduce via performance tests which I'll describe below.
My setup is using grpc-netty-shaded 1.10 (but i've also repro'd with 1.14).
My performance test uses mTLS to talk to the service. The service is
deployed on a container with 6 cores and 2 gb ram. I've reduced the
footprint of my service to immediately return with a response without doing
any other work to try and identify if it's the application or something to
do with my gRPC configuration.
My performance test is issuing about 250 requests a second using one
Managed Channel to one instance of my service. The data in each request is
about 10 bytes. With this workload, my service is running at about 35% CPU,
which I feel is far too high for this small amount of rps.
Here is how I've constructed my server:
val serverInterceptor =
MonitoringServerInterceptor.create(Configuration.allMetrics())
val realtimeServiceWithMonitoring = ServerInterceptors.intercept(
RealtimePublishGrpc.bindService(realtimeService, ExecutionContext.global),
serverInterceptor)
val rppServiceWithMonitoring = ServerInterceptors.intercept(
RealtimeProxyGrpc.bindService(realtimePublishProxyService,
ExecutionContext.global),
serverInterceptor
)
val keyManagerFactory = GrpcSSLHelper.getKeyManagerFactory(sslConfig)
val trustManagerFactory = GrpcSSLHelper.getTrustManagerFactory(sslConfig)
val serverGrpcSslContexts = GrpcSSLHelper.getServerSslContext
(keyManagerFactory, trustManagerFactory)
NettyServerBuilder
.forPort(8086)
.sslContext(serverGrpcSslContexts)
.addService(realtimeServiceWithMonitoring)
.addService(rppServiceWithMonitoring)
.build()
}
The server interceptor is modeled
after: https://github.com/grpc-ecosystem/java-grpc-prometheus
The managed channel is constructed as such:
private val interceptor =
MonitoringClientInterceptor.create(Configuration.allMetrics())
val trustManagerFactory = GrpcSSLHelper.getTrustManagerFactory(sslConfig)
NettyChannelBuilder
.forAddress(address, *8086*)
.intercept(interceptor)
.negotiationType(NegotiationType.TLS)
.sslContext(GrpcSSLHelper.getClientSslContext(keyManagerFactory,
trustManagerFactory))
.build()
Finally, I use non-blocking stubs to issue the gRPC request.
Any help would be greatly appreciated. Thanks!
-K
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/707e8c32-a76d-48fe-bd23-af3a96c98018%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
