What is the cpu utilization and load factor during the slowdown ? If a single publisher, and the rate is 10k per 10 sec then every task must complete in 1 ms or they will back up, unless you allow out of order handling.
> On Nov 21, 2018, at 1:53 AM, qplc <[email protected]> wrote: > > Thanks Kun for your reply. > > As you mentioned, I have implemented non streaming async client and server > with limited threads configured. > > Here are the details: > Machine Cores: 8 > Channel Count: 1 > Grpc Client NIO Threads: (Cores-2) * 2 = 12 > Grpc Client App Threads: 30 > Grpc Server NIO Threads: (Cores-2) * 2 = 12 > Grpc Server App Threads: 30 > Grpc Server Max. Concurrent Calls Per Connection = Grpc Server NIO Threads > i.e. 12 > > RabbitMQ Max. Pool Size = 150 > Grpc Client Threadpool size for Submitting RPC Call = 200 > Grpc Server Threadpool size to execute business functionality = 200 (Added > this threadpool to free Grpc Server App Threads) > > I have best tried to finetune grpc client and server but still I have seen a > slowness while submitting tasks for business functionality execution. > > I have seen many messages piled up in the rabbitmq queue. This shows threads > are busy submitting rpc tasks. > > Am I wrongly configuring grpc server/client here? > > On Thursday, October 25, 2018 at 3:49:07 PM UTC+5:30, qplc wrote: >> >> Hi, >> >> I've implemented zookeeper balanced grpc server and client. >> Following are the execution configuration details: >> Grpc Client: >> Channel Count: 1 >> Boss/Acceptor Thread: 1 >> Nio Threads: 100 >> Executor/App Threads: 100 >> >> Grpc Server: >> Nio Threads: 100 >> Executor/App Threads: 400 >> Max conucurrent calls per connection: 100 >> >> Here, I'm using ForkJoinPool while setting executor. I'm sending messages on >> RabbitMQ and forwards to gRPC client. Publisher rate is 10k messages per >> 10sec. >> >> As I've observed each request when reaches server it executes till >> 10seconds. And as configured most 400 tasks are getting executed >> concurrently at a time. Due to this piling up other 9600 requests as waiting >> for application threads to be available. >> >> This is slowing overall process as piling up the requests and I couldn't >> delegate the tasks to other thread pool as executor has already been >> dedicated for the same. >> >> Also, I've given 16gb for application. Increasing the thread count could >> help here as I have already given sufficient configuration. >> How do I improve the execution faster? >> >> >> Thanks, >> qplc >> > > -- > You received this message because you are subscribed to the Google Groups > "grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/e619cf3a-fc77-486e-ad78-72f9186da003%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9F52D676-6A37-40D5-84D2-2F882557E8A1%40earthlink.net. For more options, visit https://groups.google.com/d/optout.
