Kryo and Akka Kryo don't use any threads, the remote-dispatcher is the one doing the work, and Netty threads sending/receiving the bytes from/to Akka remote dispatcher as far as I can tell.
Both Kryo and Akka Kryo have a pool with Kryo instances which also happens to re-use the byte buffer making it very efficient in regards of GC. Kryo pool by default uses a Java ConcurrentQueue (I contributed some of it to Akka Kryo extension) which you can change. Read the section, KryoQueueBuilder: https://github.com/romix/akka-kryo-serialization#kryo-queue-builder-examples which will also improve even further the GC collection. HTH, Guido. On Sunday, July 3, 2016 at 1:28:46 PM UTC+1, Eduardo Fernandes wrote: > > Quite clean. Many thanks Guido! I'll try it out. I have to inventory my > classes so it will take a while. I hope I'll have some numbers this night. > > Anyway, kryo will scale vertically, I suppose. Will kryo use more threads > than I'm using right now? > > Thanks again for your help and knowledge. > > On Sun, Jul 3, 2016 at 2:13 PM, Guido Medina <oxy...@gmail.com > <javascript:>> wrote: > >> I meant to say *"anything that implements Serializable"* >> The classes list is important as Kryo won't write class names on the >> messages but IDs of the classes: >> >> classes = [ >> "com.mypackage.Class1", >> "com.mypackage.Class2" >> ] >> >> Suffice to say every node of the cluster must have the same IDs so that's >> some sort of configuration you agree upon. >> I find convenient to have a package in a "common" jar project with all >> the classes that I'm going to share (Serialize-ables) >> and a "common.conf" with Akka configuration, as I can build my final >> configuration by putting together Akka configurations, >> even using place holders, like this: >> >> config = parseFile(new File(configPath)).withFallback(parseResources( >> "common.conf"))...resolve(); >> >> You can use that fallback call as many times as you wish and it won't to >> resolve place holders until you don't call resolve() >> >> HTH, >> >> Guido. >> >> >> On Sunday, July 3, 2016 at 12:57:21 PM UTC+1, Guido Medina wrote: >>> >>> I have mimic-ed your configuration and corrected some errors, also added >>> Kryo if you want to give it a chance with the configuration I believe will >>> do best. >>> I default the "java" serializer to Kryo, that way, everything that >>> inherits "Serializable" will use Kryo, also, I list every class that I care >>> (performance wise) under Kryo list. >>> >>> Hope this give you better result, also, don't underestimate the the >>> default mailbox you have commented out: >>> >>> akka { >>> extensions = >>> ["com.romix.akka.serialization.kryo.KryoSerializationExtension$"] >>> >>> actor { >>> provider = "akka.cluster.ClusterActorRefProvider" >>> serializers.java = "com.romix.akka.serialization.kryo.KryoSerializer" >>> default-mailbox.mailbox-type = >>> "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" >>> >>> default-dispatcher { >>> type = Dispatcher >>> executor = "fork-join-executor" >>> >>> fork-join-executor { >>> parallelism-min = 4 >>> parallelism-factor = 1 >>> parallelism-max = 8 >>> } >>> } >>> >>> kryo { >>> kryo-reference-map = false >>> idstrategy = "automatic" >>> use-manifests = true >>> buffer-size = 1024 >>> type = "nograph" >>> >>> classes = [ >>> "com.mypackage.Class1", >>> "com.mypackage.Class2" >>> ] >>> } >>> } >>> >>> remote { >>> log-remote-lifecycle-events = off >>> >>> netty.tcp { >>> >>> server-socket-worker-pool { >>> pool-size-min = 4 >>> pool-size-factor = 1 >>> pool-size-max = 8 >>> } >>> >>> client-socket-worker-pool { >>> pool-size-min = 4 >>> pool-size-factor = 1 >>> pool-size-max = 8 >>> } >>> } >>> >>> default-remote-dispatcher { >>> type = Dispatcher >>> executor = "fork-join-executor" >>> >>> fork-join-executor { >>> parallelism-min = 4 >>> parallelism-factor = 1 >>> parallelism-max = 8 >>> } >>> } >>> } >>> >>> cluster { >>> metrics.enabled = off >>> jmx.enabled = off >>> } >>> } >>> >>> >>> >>> On Sunday, July 3, 2016 at 12:45:57 PM UTC+1, Eduardo Fernandes wrote: >>>> >>>> I'm using the configuration below, following Guido suggestions. >>>> >>>> Anyway, everything behaves like Akka/netty add more communication >>>> threads only if a add more endpoints. Adding more actores talking from >>>> host >>>> 1 to host 2 is not improving throughput, despite of the available idle >>>> cpu's. Does this make sense? >>>> >>>> Of course if my processing were not only dedicated to echo back some >>>> characters the processing cpu would hide the serialization performance. It >>>> just for find the maximum throughput (net performance) of a couple of >>>> actorsystem's. >>>> >>>> My topology is (attached file). Each client process has 10.000 internal >>>> sync clients and uses a single thread to dispatch messages. A single >>>> client >>>> reaches around 450.000/s and two reaches around 525.000/s. Each client >>>> talks to a particular actor in host 2 which talks to a particular actor in >>>> host 3. >>>> >>>> Again, many thanks for your help. >>>> >>>> Best regards. >>>> >>>> >>>> [image: Inline image 1] >>>> >>>> >>>> Again, many thanks for your help. >>>> >>>> actor { >>>> provider = "akka.cluster.ClusterActorRefProvider" >>>> default-dispatcher { >>>> throughput = 1024 >>>> fork-join-executor { >>>> parallelism-min = 6 >>>> parallelism-factor = 1 >>>> parallelism-max = 8 >>>> } >>>> } >>>> >>>> # default-mailbox { >>>> # mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" >>>> # } >>>> >>>> } >>>> remote { >>>> log-remote-lifecycle-events = off >>>> netty.tcp { >>>> tcp-nodelay = on >>>> write-buffer-high-water-mark = 40000000b >>>> write-buffer-low-water-mark = 0b >>>> send-buffer-size = 40000000b >>>> receive-buffer-size = 40000000b >>>> >>>> server-socket-worker-pool { >>>> pool-size-min = 4 >>>> pool-size-factor = 1 >>>> pool-size-max = 8 >>>> } >>>> >>>> client-socket-worker-pool { >>>> pool-size-min = 4 >>>> pool-size-factor = 1 >>>> pool-size-max = 8 >>>> } >>>> } >>>> } >>>> >>>> >>>> On Sun, Jul 3, 2016 at 1:05 PM, Roland Kuhn <goo...@rkuhn.info> wrote: >>>> >>>>> Doesn't the classical remoting perform serialization within the single >>>>> actor responsible for each connection? >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On 03 Jul 2016, at 12:57, Viktor Klang <viktor...@gmail.com> wrote: >>>>> >>>>> Eduardo, are you sure that there aren't any synchronized-blocks or >>>>> locks used? (i.e. is this a contention problem rather than a >>>>> paralellization problem?) >>>>> >>>>> On Sun, Jul 3, 2016 at 12:28 PM, Eduardo Fernandes <edu...@gmail.com> >>>>> wrote: >>>>> >>>>>> Ups... sorry for misunderstanding your question. >>>>>> >>>>>> My principal problem is not the overhead itself. My problem is that I >>>>>> can't get more threads serializing objects to a node. Example: one >>>>>> client I >>>>>> have 30%, lets say. If I add other client talking to other actor >>>>>> instance >>>>>> in parallel I would expect around 60% cpu usage in my server (I have 6 >>>>>> threads minimum and I'm pretty sure that the configuration would enable >>>>>> that from workers and netty perspective). Nevertheless I get around 40% >>>>>> of >>>>>> my 8 cores machine working. If I put the actors in different processes I >>>>>> get the 60% I was expecting. When I say server I mean an actorsystem >>>>>> process (a single java process). >>>>>> >>>>>> Thanks again for your help. >>>>>> >>>>>> On Sun, Jul 3, 2016 at 11:55 AM, Viktor Klang <viktor...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Eduardo, >>>>>>> >>>>>>> I meant the overhead of the Java Serialization envelope. >>>>>>> >>>>>>> On Sat, Jul 2, 2016 at 10:12 PM, Eduardo Fernandes <edu...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi Viktor. >>>>>>>> >>>>>>>> I'm using basic (binary) serialization of basic Java types (int, >>>>>>>> String (UTF), long, arrays of basic types, etc...). >>>>>>>> >>>>>>>> The overhead is that depending of internal values there is no send >>>>>>>> to serialize some members and other not. If you merge your functional >>>>>>>> logic >>>>>>>> with the serialization you can make optimizations that a generic >>>>>>>> serializar >>>>>>>> can't do. Example. Suppose that if a member A has a null value you >>>>>>>> don't >>>>>>>> have to serialize other member B. Maybe the member B you don't have to >>>>>>>> serialize could have a null value which is fast to serialize but it is >>>>>>>> even >>>>>>>> faster you don't have even to serialize the null. This type of >>>>>>>> overhead is >>>>>>>> only possible if the serializer knows about your functional logic. >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> On Sat, Jul 2, 2016 at 10:05 PM, Viktor Klang <viktor...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Eduardo, >>>>>>>>> >>>>>>>>> Perhaps I misunderstood, what serialization format are you >>>>>>>>> emitting in your readObject/writeObject? >>>>>>>>> What overhead are you observing compared to using a custom >>>>>>>>> Serializer? >>>>>>>>> >>>>>>>>> On Sat, Jul 2, 2016 at 10:02 PM, Eduardo Fernandes < >>>>>>>>> edu...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi. >>>>>>>>>> >>>>>>>>>> If you have writeObject/readObject defined in your class the Java >>>>>>>>>> plain serialization will invoke those methods. In my case all my >>>>>>>>>> internal >>>>>>>>>> members and class references are also serialized using the very same >>>>>>>>>> technique. So this is equivalent to technologies like kryo and >>>>>>>>>> similars >>>>>>>>>> since there is no overhead if you serialize basic members. In other >>>>>>>>>> words >>>>>>>>>> the pre-compiles classes you get from kryo are already made so there >>>>>>>>>> is no >>>>>>>>>> performance enhancement in this case. The big advantage of kryo is >>>>>>>>>> that you >>>>>>>>>> don't have to create the writeObject/readObject by yourself. In my >>>>>>>>>> particular case I've already done that job and my serialization is >>>>>>>>>> optimized in particular cases where I don't have to serialize all >>>>>>>>>> members >>>>>>>>>> depending of my semantic. I've made some tests and doing this way is >>>>>>>>>> faster >>>>>>>>>> than kryo but you have to burn some calories implementing a >>>>>>>>>> optimized >>>>>>>>>> serialization code. >>>>>>>>>> >>>>>>>>>> Bests regards and thanks for your comment. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Jul 2, 2016 at 9:27 PM, Viktor Klang <viktor...@gmail.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> I'm not sure I understand why write/readObject special methods >>>>>>>>>>> would necessarily be faster? Most of the waste of Java >>>>>>>>>>> Serialization is its >>>>>>>>>>> envelopes and using class names etc. >>>>>>>>>>> >>>>>>>>>>> On Sat, Jul 2, 2016 at 1:14 AM, Eduardo Fernandes < >>>>>>>>>>> edu...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi. >>>>>>>>>>>> >>>>>>>>>>>> I'm using Akka 2.3.13, Java edition. >>>>>>>>>>>> >>>>>>>>>>>> I'm making some performance tests and in the same machine with >>>>>>>>>>>> 8 cores I see that the serialization process is my bottleneck. I >>>>>>>>>>>> know that >>>>>>>>>>>> because after an increment of actor cpu usage the throughput is >>>>>>>>>>>> exactly the >>>>>>>>>>>> same. >>>>>>>>>>>> >>>>>>>>>>>> My actor system talks to 2 other nodes so I see 2 cores >>>>>>>>>>>> dedicated to serialization. Is is possible to increase the number >>>>>>>>>>>> of >>>>>>>>>>>> threads for serialization? >>>>>>>>>>>> >>>>>>>>>>>> I'm using standard Java serialization but I have my own >>>>>>>>>>>> serialization implementation in my write/readObject methods so I >>>>>>>>>>>> think that >>>>>>>>>>>> switching to kryo or similar will not enhance too much the >>>>>>>>>>>> throughput. >>>>>>>>>>>> >>>>>>>>>>>> Many thanks for your help. >>>>>>>>>>>> >>>>>>>>>>>> /Eduardo >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>>> --- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "Akka User List" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to akka-user+...@googlegroups.com. >>>>>>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Cheers, >>>>>>>>>>> √ >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>> in the Google Groups "Akka User List" group. >>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe >>>>>>>>>>> . >>>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>>> to akka-user+...@googlegroups.com. >>>>>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Akka User List" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to akka-user+...@googlegroups.com. >>>>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Cheers, >>>>>>>>> √ >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "Akka User List" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe >>>>>>>>> . >>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>> to akka-user+...@googlegroups.com. >>>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>> https://groups.google.com/group/akka-user >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Akka User List" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to akka-user+...@googlegroups.com. >>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Cheers, >>>>>>> √ >>>>>>> >>>>>>> -- >>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>> >>>>>>>>>> Search the archives: >>>>>>> https://groups.google.com/group/akka-user >>>>>>> --- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "Akka User List" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> akka-user+...@googlegroups.com. >>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: >>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: >>>>>> https://groups.google.com/group/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to akka-user+...@googlegroups.com. >>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> √ >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to akka-user+...@googlegroups.com. >>>>> To post to this group, send email to akka...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Akka User List" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> akka-user+...@googlegroups.com. >>>>> To post to this group, send email to akka...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "Akka User List" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> akka-user+...@googlegroups.com <javascript:>. >> To post to this group, send email to akka...@googlegroups.com >> <javascript:>. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
