Re: [grpc-io] Re: gRPC server executor pool slowing process execution
1L? Still, you might want to look at /proc/schedstat 600% utilization is pretty high, especially since there may be only 4 real cores. Also, what is the user cpu % vs system ? You might just be thrashing scheduling all of those threads on 8 cores. > On Nov 21, 2018, at 7:45 AM, qplc wrote: > > Single publisher is sending 1L messages and being distributed over 2mins. > > Observed: > load average: 1.24, 0.48, 0.17 > %CPU: 600 > %MEM: 65 > > > >> On Thursday, October 25, 2018 at 3:49:07 PM UTC+5:30, qplc wrote: >> Hi, >> >> I've implemented zookeeper balanced grpc server and client. >> Following are the execution configuration details: >> Grpc Client: >> Channel Count: 1 >> Boss/Acceptor Thread: 1 >> Nio Threads: 100 >> Executor/App Threads: 100 >> >> Grpc Server: >> Nio Threads: 100 >> Executor/App Threads: 400 >> Max conucurrent calls per connection: 100 >> >> Here, I'm using ForkJoinPool while setting executor. I'm sending messages on >> RabbitMQ and forwards to gRPC client. Publisher rate is 10k messages per >> 10sec. >> >> As I've observed each request when reaches server it executes till >> 10seconds. And as configured most 400 tasks are getting executed >> concurrently at a time. Due to this piling up other 9600 requests as waiting >> for application threads to be available. >> >> This is slowing overall process as piling up the requests and I couldn't >> delegate the tasks to other thread pool as executor has already been >> dedicated for the same. >> >> Also, I've given 16gb for application. Increasing the thread count could >> help here as I have already given sufficient configuration. >> How do I improve the execution faster? >> >> >> Thanks, >> qplc >> > > -- > You received this message because you are subscribed to the Google Groups > "grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to grpc-io+unsubscr...@googlegroups.com. > To post to this group, send email to grpc-io@googlegroups.com. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/7bc25124-5456-4fcd-9ad3-aff2917337dd%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1A899BB6-2FA0-49D2-AE09-D7156565D663%40earthlink.net. For more options, visit https://groups.google.com/d/optout.
[grpc-io] Re: gRPC server executor pool slowing process execution
Single publisher is sending 1L messages and being distributed over 2mins. *Observed:* load average: 1.24, 0.48, 0.17 %CPU: 600 %MEM: 65 On Thursday, October 25, 2018 at 3:49:07 PM UTC+5:30, qplc wrote: > > Hi, > > I've implemented zookeeper balanced grpc server and client. > Following are the execution configuration details: > Grpc Client: > Channel Count: 1 > Boss/Acceptor Thread: 1 > Nio Threads: 100 > Executor/App Threads: 100 > > Grpc Server: > Nio Threads: 100 > Executor/App Threads: 400 > Max conucurrent calls per connection: 100 > > Here, I'm using ForkJoinPool while setting executor. I'm sending messages > on RabbitMQ and forwards to gRPC client. Publisher rate is 10k messages per > 10sec. > > As I've observed each request when reaches server it executes till > 10seconds. And as configured most 400 tasks are getting executed > concurrently at a time. Due to this piling up other 9600 requests as > waiting for application threads to be available. > > This is slowing overall process as piling up the requests and I couldn't > delegate the tasks to other thread pool as executor has already been > dedicated for the same. > > Also, I've given 16gb for application. Increasing the thread count could > help here as I have already given sufficient configuration. > How do I improve the execution faster? > > > Thanks, > qplc > > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/7bc25124-5456-4fcd-9ad3-aff2917337dd%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [grpc-io] Re: gRPC server executor pool slowing process execution
What is the cpu utilization and load factor during the slowdown ? If a single publisher, and the rate is 10k per 10 sec then every task must complete in 1 ms or they will back up, unless you allow out of order handling. > On Nov 21, 2018, at 1:53 AM, qplc wrote: > > Thanks Kun for your reply. > > As you mentioned, I have implemented non streaming async client and server > with limited threads configured. > > Here are the details: > Machine Cores: 8 > Channel Count: 1 > Grpc Client NIO Threads: (Cores-2) * 2 = 12 > Grpc Client App Threads: 30 > Grpc Server NIO Threads: (Cores-2) * 2 = 12 > Grpc Server App Threads: 30 > Grpc Server Max. Concurrent Calls Per Connection = Grpc Server NIO Threads > i.e. 12 > > RabbitMQ Max. Pool Size = 150 > Grpc Client Threadpool size for Submitting RPC Call = 200 > Grpc Server Threadpool size to execute business functionality = 200 (Added > this threadpool to free Grpc Server App Threads) > > I have best tried to finetune grpc client and server but still I have seen a > slowness while submitting tasks for business functionality execution. > > I have seen many messages piled up in the rabbitmq queue. This shows threads > are busy submitting rpc tasks. > > Am I wrongly configuring grpc server/client here? > > On Thursday, October 25, 2018 at 3:49:07 PM UTC+5:30, qplc wrote: >> >> Hi, >> >> I've implemented zookeeper balanced grpc server and client. >> Following are the execution configuration details: >> Grpc Client: >> Channel Count: 1 >> Boss/Acceptor Thread: 1 >> Nio Threads: 100 >> Executor/App Threads: 100 >> >> Grpc Server: >> Nio Threads: 100 >> Executor/App Threads: 400 >> Max conucurrent calls per connection: 100 >> >> Here, I'm using ForkJoinPool while setting executor. I'm sending messages on >> RabbitMQ and forwards to gRPC client. Publisher rate is 10k messages per >> 10sec. >> >> As I've observed each request when reaches server it executes till >> 10seconds. And as configured most 400 tasks are getting executed >> concurrently at a time. Due to this piling up other 9600 requests as waiting >> for application threads to be available. >> >> This is slowing overall process as piling up the requests and I couldn't >> delegate the tasks to other thread pool as executor has already been >> dedicated for the same. >> >> Also, I've given 16gb for application. Increasing the thread count could >> help here as I have already given sufficient configuration. >> How do I improve the execution faster? >> >> >> Thanks, >> qplc >> > > -- > You received this message because you are subscribed to the Google Groups > "grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to grpc-io+unsubscr...@googlegroups.com. > To post to this group, send email to grpc-io@googlegroups.com. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/e619cf3a-fc77-486e-ad78-72f9186da003%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9F52D676-6A37-40D5-84D2-2F882557E8A1%40earthlink.net. For more options, visit https://groups.google.com/d/optout.
[grpc-io] Re: gRPC server executor pool slowing process execution
Thanks Kun for your reply. As you mentioned, I have implemented non streaming async client and server with limited threads configured. Here are the details: Machine Cores: 8 Channel Count: 1 Grpc Client NIO Threads: (Cores-2) * 2 = 12 Grpc Client App Threads: 30 Grpc Server NIO Threads: (Cores-2) * 2 = 12 Grpc Server App Threads: 30 Grpc Server Max. Concurrent Calls Per Connection = Grpc Server NIO Threads i.e. 12 RabbitMQ Max. Pool Size = 150 Grpc Client Threadpool size for Submitting RPC Call = 200 Grpc Server Threadpool size to execute business functionality = 200 (Added this threadpool to free Grpc Server App Threads) I have best tried to finetune grpc client and server but still I have seen a slowness while submitting tasks for business functionality execution. I have seen many messages piled up in the rabbitmq queue. This shows threads are busy submitting rpc tasks. Am I wrongly configuring grpc server/client here? On Thursday, October 25, 2018 at 3:49:07 PM UTC+5:30, qplc wrote: > > Hi, > > I've implemented zookeeper balanced grpc server and client. > Following are the execution configuration details: > Grpc Client: > Channel Count: 1 > Boss/Acceptor Thread: 1 > Nio Threads: 100 > Executor/App Threads: 100 > > Grpc Server: > Nio Threads: 100 > Executor/App Threads: 400 > Max conucurrent calls per connection: 100 > > Here, I'm using ForkJoinPool while setting executor. I'm sending messages > on RabbitMQ and forwards to gRPC client. Publisher rate is 10k messages per > 10sec. > > As I've observed each request when reaches server it executes till > 10seconds. And as configured most 400 tasks are getting executed > concurrently at a time. Due to this piling up other 9600 requests as > waiting for application threads to be available. > > This is slowing overall process as piling up the requests and I couldn't > delegate the tasks to other thread pool as executor has already been > dedicated for the same. > > Also, I've given 16gb for application. Increasing the thread count could > help here as I have already given sufficient configuration. > How do I improve the execution faster? > > > Thanks, > qplc > > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/e619cf3a-fc77-486e-ad78-72f9186da003%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[grpc-io] Re: gRPC server executor pool slowing process execution
The number of threads, especially the nio threads, look way too many. Too many threads only add overhead. You may try limiting your number of threads to 2 times the number of CPU cores. If you are doing blocking operations in your application thread, try switching to a non-blocking asynchronous model which will scale better. On Thursday, October 25, 2018 at 3:19:07 AM UTC-7, qplc wrote: > > Hi, > > I've implemented zookeeper balanced grpc server and client. > Following are the execution configuration details: > Grpc Client: > Channel Count: 1 > Boss/Acceptor Thread: 1 > Nio Threads: 100 > Executor/App Threads: 100 > > Grpc Server: > Nio Threads: 100 > Executor/App Threads: 400 > Max conucurrent calls per connection: 100 > > Here, I'm using ForkJoinPool while setting executor. I'm sending messages > on RabbitMQ and forwards to gRPC client. Publisher rate is 10k messages per > 10sec. > > As I've observed each request when reaches server it executes till > 10seconds. And as configured most 400 tasks are getting executed > concurrently at a time. Due to this piling up other 9600 requests as > waiting for application threads to be available. > > This is slowing overall process as piling up the requests and I couldn't > delegate the tasks to other thread pool as executor has already been > dedicated for the same. > > Also, I've given 16gb for application. Increasing the thread count could > help here as I have already given sufficient configuration. > How do I improve the execution faster? > > > Thanks, > qplc > > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/be7f6eb6-5862-43ce-9dd6-9793d3a3c6b2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
