Looks like you're using Java Serialization, a good first start would be to switch serializers for your messages.
On Wed, Apr 1, 2015 at 3:49 PM, Ivan Balashov <ibalas...@gmail.com> wrote: > Viktor, > > Here is the latest configuration that allowed to me reach 30msg/sec > between 2 x 4cpu hosts in GCE network. > > http://goo.gl/sPnjQS > > On both sides context switches jumped to 30-50K/sec > > Client: > 0 0 0 1528652 94396 794228 0 0 0 4 468 772 0 0 > 100 0 > 2 0 0 1533988 94408 794224 0 0 0 5 9051 16555 11 4 > 85 0 > 1 0 0 1326560 94412 794224 0 0 0 6 28710 52792 34 13 > 54 0 > 2 0 0 1326188 94412 794228 0 0 0 3 30209 59439 26 8 > 66 0 > 1 0 0 1325444 94420 794228 0 0 0 4 26194 51556 24 6 > 70 0 > 1 0 0 1328172 94424 794228 0 0 0 1 468 725 0 0 > 100 0 > > Server: > 3 0 0 827812 30372 2066316 0 0 0 3 9762 13188 48 7 > 45 0 > 1 0 0 564080 30388 2066316 0 0 0 3 19969 23901 29 13 > 59 0 > 4 0 0 416900 30388 2066316 0 0 0 0 24864 31988 39 13 > 48 0 > 1 0 0 241876 30388 2066316 0 0 0 3 24113 34808 42 11 > 47 0 > 4 0 0 145280 30404 2014432 0 0 0 2 21127 40357 44 6 > 50 0 > 2 0 0 106608 30404 1960024 0 0 0 0 19162 36545 47 6 > 47 0 > 4 0 0 138560 30328 1867092 0 0 0 6 14574 27803 57 4 > 39 0 > 2 0 0 293616 30336 1867092 0 0 0 4 6390 10023 42 12 > 46 0 > > Couple more things that concerned me: > > 1) [WARN] [04/01/2015 12:54:26.125] [systemB-network-dispatcher-5] > [akka.tcp:// > systemB@10.110.112.155:2001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FsystemA%4010.110.47.70%3A2001-0/endpointWriter] > [ > *473494*] buffered messages in EndpointWriter for [akka.tcp:// > systemA@10.110.47.70:2001]. *You should probably implement flow control > to avoid flooding the remote connection.* > > Looks like either side is sending more messages than receiving is capable > to accept. This however, might be cause by 1M network buffer, which, OTOH > is needed for higher throughput. > jFTR, GCE network is quite fast (my last measurement gives me 72Mb/sec, > while during the test we barely hit 10Mb/sec). > > 2) Significant heap pressure, apparently caused by 1) and, as a > consequence, some GC activity. Profiler gives me estimate of ~700bytes per > every message in the queue, quickly growing heap. The slower messages get > processed, the more of them are accumulated, the more cpu is needed for GC, > chicken-egg. > > Any configuration advice to achieve better throughput in this scenario of > many small messages? > > > On Wednesday, April 1, 2015 at 10:59:32 AM UTC+3, √ wrote: >> >> Could you share your entire config? >> >> On Tue, Mar 31, 2015 at 5:31 PM, Ivan Balashov <ibal...@gmail.com> wrote: >> >>> >>> >>> On Tuesday, March 31, 2015 at 11:23:25 AM UTC+3, √ wrote: >>>> >>>> Sounds like you're using too many FJ threads. >>>> >>> >>> I wish it was that simple. For both remoting and actor pool the same >>> dispatcher is used (4 core box): >>> >>> type = Dispatcher >>>> executor = "fork-join-executor" >>>> throughput = 1000 // Does this apply to FJ, or only to >>>> ThreadPoolEx? >>>> fork-join-executor { >>>> parallelism-min = 1 >>>> parallelism-max = 4 >>>> } >>> >>> >>> I get lower CS values if I set parallelism-max=1, maintaining about the >>> same total throughput only with less cpu burn. >>> However, it looks like CS should not depend much on whether we have 1 or >>> 4 or 10 FJ threads, most switching must be happing on deeper level, e.g. >>> controlled by `throughput`. >>> >>> >>> >>> >>>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to akka-user+...@googlegroups.com. >>> To post to this group, send email to akka...@googlegroups.com. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Cheers, >> √ >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Cheers, √ -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
