It's probably the tcp write buffer that is full and there is degradation because of backoff. I would avoid sending so many messages in a burst without application level flow control.
Note that Akka 2.3.x is end-of life and if you are looking for much better remoting performance you should give the new implementation a try (in Akka 2.4 and 2.5): http://doc.akka.io/docs/akka/current/scala/remoting-artery.html Regards, Patrik On Thu, Jul 6, 2017 at 11:30 AM, Gao Yuan <[email protected]> wrote: > I'm using akka 2.3.10 > I have a simple akka communication program. and 5 machines(one for server, > the rest 4 are clients). > Client will send requests by calling method: tell(), each request will be > marked with a timestamp (the time we call method tell()) , and a parameter > for the server: "processDelay". Something like this: > class Request { > int requestId, > long tellingTime = System.currentTimeMillis() > long processDelay = 0; //in ms > ... > } // sizeof(Request): about 2kB > > Server will schedule a response when request comes. Something like this: > ExecutorService es = Executors.newScheduledThreadPool(...) > void onReceive(request) { > es.schedule(new Runable(){ ... process request and send response to the > client ... }, request.processDelay, TimeUnit.MILLISECONDS); > } > > The Response contains information as: > class Response { > Request request; > long requestArriveTime; > long responseSendTime; > ... > } // sizeof(Response): about 3.5kB > > Let's mark clients with name : C1, C2, C3, C4; > Form C1, I will keep 40000 requests (with "processDelay == 5000") in > working. Each Response will trigger another request, so we can keep 40000 > request in working. > And for C2,C3,C4, I will keep 1000 request (with "processDelay == 0") in > working. > As a result, all clients reach a throughput around 5500 tps. (BTW, > machines connected to a same switch, and network bandwidth is enough for > the test) > > The process may cut into steps: > > 1. client call method tell(request) > 2. client's endpointWriter call writeSend > 3. server's onReceive(request) get the request > 4. server make delay base on Request.processDelay > 5. server call method tell(response) > 6. server's endpointWriter call writeSend > 7. client's onReceive(response) get the response > > We expect the whole time cost would be a litter longer than the > Request.processDelay. > On C2,C3,C4, the actual time cost is around 150ms, close to > Request.processDelay(0ms). > But on C1, the actual result is around 7000ms, while the > Request.processDelay == 5000ms. There are 2s lost. C1 should get response > in 5 seconds. > > I add logs to akka code. Seems that between step 5 and 6, we lost the > missing 2s. There are too many messages in EndpointWriter.buffer. 12000 > messages in buffer for C1, while only 800 messages for C2,C3,C4. > > I did trys which makes no big difference: > > 1. use PinnedDispatcher > 2. use thread-pool-executor > 3. change the pool size > 4. change EndpointWriter.SendBufferBatchSize > > Questions: > > 1. Since the response speed pumps to the EndpointWriter are same, why > C1's buffer is much more bigger than others? Is there a reason? > 2. Is there any mechanism that make the longer buffer get higher > process priority? > > Let's make another try: > On all the clients (C1-C4), I will keep 40000 requests (with "processDelay > == 5000") in working. > All client will reach throughput around 5500 tps, and they all lost 2 > seconds between step 5 and 6. Why? > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Akka Tech Lead Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
