Viktor, Here is the latest configuration that allowed to me reach 30msg/sec between 2 x 4cpu hosts in GCE network.
http://goo.gl/sPnjQS On both sides context switches jumped to 30-50K/sec Client: 0 0 0 1528652 94396 794228 0 0 0 4 468 772 0 0 100 0 2 0 0 1533988 94408 794224 0 0 0 5 9051 16555 11 4 85 0 1 0 0 1326560 94412 794224 0 0 0 6 28710 52792 34 13 54 0 2 0 0 1326188 94412 794228 0 0 0 3 30209 59439 26 8 66 0 1 0 0 1325444 94420 794228 0 0 0 4 26194 51556 24 6 70 0 1 0 0 1328172 94424 794228 0 0 0 1 468 725 0 0 100 0 Server: 3 0 0 827812 30372 2066316 0 0 0 3 9762 13188 48 7 45 0 1 0 0 564080 30388 2066316 0 0 0 3 19969 23901 29 13 59 0 4 0 0 416900 30388 2066316 0 0 0 0 24864 31988 39 13 48 0 1 0 0 241876 30388 2066316 0 0 0 3 24113 34808 42 11 47 0 4 0 0 145280 30404 2014432 0 0 0 2 21127 40357 44 6 50 0 2 0 0 106608 30404 1960024 0 0 0 0 19162 36545 47 6 47 0 4 0 0 138560 30328 1867092 0 0 0 6 14574 27803 57 4 39 0 2 0 0 293616 30336 1867092 0 0 0 4 6390 10023 42 12 46 0 Couple more things that concerned me: 1) [WARN] [04/01/2015 12:54:26.125] [systemB-network-dispatcher-5] [akka.tcp://[email protected]:2001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FsystemA%4010.110.47.70%3A2001-0/endpointWriter] [ *473494*] buffered messages in EndpointWriter for [akka.tcp://[email protected]:2001]. *You should probably implement flow control to avoid flooding the remote connection.* Looks like either side is sending more messages than receiving is capable to accept. This however, might be cause by 1M network buffer, which, OTOH is needed for higher throughput. jFTR, GCE network is quite fast (my last measurement gives me 72Mb/sec, while during the test we barely hit 10Mb/sec). 2) Significant heap pressure, apparently caused by 1) and, as a consequence, some GC activity. Profiler gives me estimate of ~700bytes per every message in the queue, quickly growing heap. The slower messages get processed, the more of them are accumulated, the more cpu is needed for GC, chicken-egg. Any configuration advice to achieve better throughput in this scenario of many small messages? On Wednesday, April 1, 2015 at 10:59:32 AM UTC+3, √ wrote: > > Could you share your entire config? > > On Tue, Mar 31, 2015 at 5:31 PM, Ivan Balashov <[email protected] > <javascript:>> wrote: > >> >> >> On Tuesday, March 31, 2015 at 11:23:25 AM UTC+3, √ wrote: >>> >>> Sounds like you're using too many FJ threads. >>> >> >> I wish it was that simple. For both remoting and actor pool the same >> dispatcher is used (4 core box): >> >> type = Dispatcher >>> executor = "fork-join-executor" >>> throughput = 1000 // Does this apply to FJ, or only to >>> ThreadPoolEx? >>> fork-join-executor { >>> parallelism-min = 1 >>> parallelism-max = 4 >>> } >> >> >> I get lower CS values if I set parallelism-max=1, maintaining about the >> same total throughput only with less cpu burn. >> However, it looks like CS should not depend much on whether we have 1 or >> 4 or 10 FJ threads, most switching must be happing on deeper level, e.g. >> controlled by `throughput`. >> >> >> >> >>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Cheers, > √ > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
