Hi Carl,
Thanks for responding! I've tried a couple different executors and they
don't seem to change the behavior. I've done FixedThreadPool with the
number of threads = # of cores * 2, the ForkJoinPool.commonPool as you
recommended, and the Scala global ExecutionContext which ultimately is a
ForkJoinPool as well. I've set this in the NettyServerBuilder as well as
the call to bind my service.
For some more information, here's results from a gatling test run that
lasted 10 minutes using the CommonPool. Server implementation now looks
like this:
val realtimeServiceWithMonitoring =
ServerInterceptors.intercept(
RealtimePublishGrpc.bindService(realtimeService, ExecutionContext.global),
serverInterceptor)
val rppServiceWithMonitoring = ServerInterceptors.intercept(
RealtimeProxyGrpc.bindService(realtimePublishProxyService,
ExecutionContext.global),
serverInterceptor
)
NettyServerBuilder
.forPort(*8086*)
.sslContext(serverGrpcSslContexts)
.addService(realtimeServiceWithMonitoring)
.addService(batchPublishWithMonitoring)
.addService(rppServiceWithMonitoring)
.executor(ForkJoinPool.commonPool())
.build()
My service implementation immediately returns Future.successful:
override def publish(request: PublishRequest): Future[PublishResponse] = {
logger.debug("Received Publish request: " + request)
Future.successful(PublishResponse())
}
Test Results:
================================================================================
---- Global Information
--------------------------------------------------------
> request count 208686 (OK=208686 KO=0
)
> min response time 165 (OK=165 KO=-
)
> max response time 2997 (OK=2997 KO=-
)
> mean response time 287 (OK=287 KO=-
)
> std deviation 145 (OK=145 KO=-
)
> response time 50th percentile 232 (OK=232 KO=-
)
> response time 75th percentile 324 (OK=324 KO=-
)
> response time 95th percentile 501 (OK=501 KO=-
)
> response time 99th percentile 894 (OK=893 KO=-
)
> mean requests/sec 347.231 (OK=347.231
KO=- )
---- Response Time Distribution
------------------------------------------------
> t < 800 ms 206014 ( 99%)
> 800 ms < t < 1200 ms 1511 ( 1%)
> t > 1200 ms 1161 ( 1%)
> failed 0 ( 0%)
================================================================================
347 Requests/Sec. CPU Utilization hovers between 29% - 35%.
Thanks for your help!
On Tuesday, August 28, 2018 at 12:49:38 PM UTC-7, Carl Mastrangelo wrote:
>
> Can you try setting the executor on both the channel and the server
> builder? I would recommend ForkJoinPool.commonPool().
>
> On Monday, August 27, 2018 at 11:54:19 PM UTC-7, Kos wrote:
>>
>> Hi,
>>
>> I'm using gRPC in a new Scala service and I'm seeing unexpectedly high
>> CPU utilization. I see this high utilization in our production workload but
>> also am able to reproduce via performance tests which I'll describe below.
>>
>> My setup is using grpc-netty-shaded 1.10 (but i've also repro'd with
>> 1.14). My performance test uses mTLS to talk to the service. The service is
>> deployed on a container with 6 cores and 2 gb ram. I've reduced the
>> footprint of my service to immediately return with a response without doing
>> any other work to try and identify if it's the application or something to
>> do with my gRPC configuration.
>>
>> My performance test is issuing about 250 requests a second using one
>> Managed Channel to one instance of my service. The data in each request is
>> about 10 bytes. With this workload, my service is running at about 35% CPU,
>> which I feel is far too high for this small amount of rps.
>>
>> Here is how I've constructed my server:
>>
>> val serverInterceptor =
>> MonitoringServerInterceptor.create(Configuration.allMetrics())
>>
>>
>> val realtimeServiceWithMonitoring = ServerInterceptors.intercept(
>> RealtimePublishGrpc.bindService(realtimeService, ExecutionContext.global),
>> serverInterceptor)
>> val rppServiceWithMonitoring = ServerInterceptors.intercept(
>> RealtimeProxyGrpc.bindService(realtimePublishProxyService,
>> ExecutionContext.global),
>> serverInterceptor
>> )
>>
>>
>> val keyManagerFactory = GrpcSSLHelper.getKeyManagerFactory(sslConfig)
>> val trustManagerFactory = GrpcSSLHelper.getTrustManagerFactory
>> (sslConfig)
>> val serverGrpcSslContexts = GrpcSSLHelper.getServerSslContext
>> (keyManagerFactory, trustManagerFactory)
>>
>> NettyServerBuilder
>> .forPort(8086)
>> .sslContext(serverGrpcSslContexts)
>> .addService(realtimeServiceWithMonitoring)
>> .addService(rppServiceWithMonitoring)
>> .build()
>> }
>>
>>
>> The server interceptor is modeled after:
>> https://github.com/grpc-ecosystem/java-grpc-prometheus
>>
>> The managed channel is constructed as such:
>>
>> private val interceptor =
>> MonitoringClientInterceptor.create(Configuration.allMetrics())
>>
>>
>> val trustManagerFactory = GrpcSSLHelper.getTrustManagerFactory(sslConfig)
>>
>> NettyChannelBuilder
>> .forAddress(address, *8086*)
>> .intercept(interceptor)
>> .negotiationType(NegotiationType.TLS)
>> .sslContext(GrpcSSLHelper.getClientSslContext(keyManagerFactory,
>> trustManagerFactory))
>> .build()
>>
>>
>> Finally, I use non-blocking stubs to issue the gRPC request.
>>
>> Any help would be greatly appreciated. Thanks!
>> -K
>>
>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/8342862b-6968-4c10-ae35-c35ae1c96fb4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
