No worries: thanks for sharing your findings!
7 aug 2014 kl. 15:00 skrev Sean Zhong <[email protected]>:
> I finally find out it is all out remote dispatcher parallism setting! when I
> change "parallelism-max" to 10, the performance is greatly improved.
>
> "default-remote-dispatcher": {
> "executor": "fork-join-executor",
> "fork-join-executor": {
> "parallelism-max": 10,
> "parallelism-factor" : 2.0
> "parallelism-min": 2
> },
> use-dispatcher: "akka.remote.default-remote-dispatcher",
>
>
> Sorry for taking so long to find out the answer, thanks for your help
>
> On Thursday, August 7, 2014 8:47:26 PM UTC+8, Sean Zhong wrote:
> The default-remote-dispatcher config is:
> ### Default dispatcher for the remoting subsystem
>
> default-remote-dispatcher {
> type = Dispatcher
> executor = "fork-join-executor"
> fork-join-executor {
> # Min number of threads to cap factor-based parallelism number to
> parallelism-min = 2
> parallelism-max = 2
> }
> }
>
>
> Is it possible that the value setting here are too Conservative which impact
> performance. I will tune this and see what happens...
>
>
>
> On Thursday, August 7, 2014 8:35:50 PM UTC+8, Sean Zhong wrote:
> I finally narrowed down the config item: akka.actor.remote.use-dispatcher
>
> The default setting for akka 2.3.4 remote is akka.actor.remote.use-dispatcher
> = "akka.remote.default-remote-dispatcher", when I change it to
> akka.actor.remote.use-dispatcher = "", then the performance is same or better
> with akka 2.2.3.
>
>
>
>
> On Thursday, August 7, 2014 7:57:28 PM UTC+8, √ wrote:
> I hope you didn't try to run the config verbatim as it works differently
> between the versions, but what happened when you updated to use the 2.2.3
> values for the options that were still there in 2.3.4?
>
>
> On Thu, Aug 7, 2014 at 1:55 PM, Sean Zhong <[email protected]> wrote:
> I made a diff, and try to use old akka 2.2.3 config when running with akka
> 2.3.4
>
> Here is the diff:
>
> --- akka.2.3.4.conf.json Thu Aug 7 19:33:37 2014
> +++ akka2.2.3.conf.json Thu Aug 7 19:32:35 2014
> @@ -13,13 +13,10 @@
> },
> "default-dispatcher": {
> "attempt-teamwork": "on",
> - "default-executor": {
> - "fallback": "fork-join-executor"
> - },
> - "executor": "default-executor",
> + "executor": "fork-join-executor",
> "fork-join-executor": {
> "parallelism-factor": 3,
> - "parallelism-max": 4,
> + "parallelism-max": 64,
> "parallelism-min": 8
> },
> "mailbox-capacity": -1,
> @@ -40,14 +37,14 @@
> "task-queue-size": -1,
> "task-queue-type": "linked"
> },
> - "throughput": 1024,
> + "throughput": 5,
> "throughput-deadline-time": "0ms",
> "type": "Dispatcher"
> },
> "default-mailbox": {
> "mailbox-capacity": 1000,
> "mailbox-push-timeout-time": "10s",
> - "mailbox-type": "akka.dispatch.SingleConsumerOnlyUnboundedMailbox",
> + "mailbox-type": "akka.dispatch.UnboundedMailbox",
> "stash-capacity": -1
> },
> "default-stash-dispatcher": {
> @@ -62,7 +59,6 @@
> "resizer": {
> "backoff-rate": 0.1,
> "backoff-threshold": 0.3,
> - "enabled": "off",
> "lower-bound": 1,
> "messages-per-resize": 10,
> "pressure-threshold": 1,
> @@ -110,29 +106,12 @@
> },
> "provider": "akka.remote.RemoteActorRefProvider",
> "reaper-interval": "5s",
> - "router": {
> - "type-mapping": {
> - "balancing-pool": "akka.routing.BalancingPool",
> - "broadcast-group": "akka.routing.BroadcastGroup",
> - "broadcast-pool": "akka.routing.BroadcastPool",
> - "consistent-hashing-group": "akka.routing.ConsistentHashingGroup",
> - "consistent-hashing-pool": "akka.routing.ConsistentHashingPool",
> - "from-code": "akka.routing.NoRouter",
> - "random-group": "akka.routing.RandomGroup",
> - "random-pool": "akka.routing.RandomPool",
> - "round-robin-group": "akka.routing.RoundRobinGroup",
> - "round-robin-pool": "akka.routing.RoundRobinPool",
> - "scatter-gather-group":
> "akka.routing.ScatterGatherFirstCompletedGroup",
> - "scatter-gather-pool":
> "akka.routing.ScatterGatherFirstCompletedPool",
> - "smallest-mailbox-pool": "akka.routing.SmallestMailboxPool"
> - }
> - },
> "serialization-bindings": {
> "[B": "bytes",
> - "akka.actor.ActorSelectionMessage": "akka-containers",
> + "akka.actor.SelectionPath": "akka-containers",
> "akka.remote.DaemonMsgCreate": "daemon-create",
> - "com.google.protobuf.GeneratedMessage": "proto",
> - "java.io.Serializable": "java",
> + "com.google.protobuf_spark.GeneratedMessage": "proto",
> + "java.io.Serializable": "java"
> },
> "serialize-creators": "off",
> "serialize-messages": "off",
> @@ -149,13 +128,9 @@
> "unstarted-push-timeout": "10s"
> },
> "daemonic": "off",
> - "event-handler-startup-timeout": "5s",
> - "event-handlers": [
> - "akka.event.Logging$DefaultLogger"
> - ],
> - "extensions": [
> - "com.romix.akka.serialization.kryo.KryoSerializationExtension$"
> - ],
> + "event-handler-startup-timeout": "-1s",
> + "event-handlers": [],
> + "extensions": [],
> "home": "",
> "io": {
> "default-backlog": 1000,
> @@ -228,37 +203,18 @@
> "gremlin": "akka.remote.transport.FailureInjectorProvider",
> "trttl": "akka.remote.transport.ThrottlerProvider"
> },
> - "backoff-interval": "5 ms",
> - "backoff-remote-dispatcher": {
> - "executor": "fork-join-executor",
> - "fork-join-executor": {
> - "parallelism-max": 2,
> - "parallelism-min": 2
> - },
> - "type": "Dispatcher"
> - },
> + "backoff-interval": "0.01 s",
> "command-ack-timeout": "30 s",
> - "default-remote-dispatcher": {
> - "executor": "fork-join-executor",
> - "fork-join-executor": {
> - "parallelism-max": 2,
> - "parallelism-min": 2
> - },
> - "type": "Dispatcher"
> - },
> "enabled-transports": [
> "akka.remote.netty.tcp"
> ],
> "flush-wait-on-shutdown": "2 s",
> - "gremlin": {
> - "debug": "off"
> - },
> - "initial-system-message-delivery-timeout": "3 m",
> - "log-buffer-size-exceeding": 50000,
> + "gate-invalid-addresses-for": "60 s",
> "log-frame-size-exceeding": "off",
> "log-received-messages": "off",
> "log-remote-lifecycle-events": "on",
> "log-sent-messages": "off",
> + "maximum-retries-in-window": 3,
> "netty": {
> "ssl": {
> "applied-adapters": [],
> @@ -360,26 +316,29 @@
> "write-buffer-low-water-mark": "0b"
> }
> },
> - "prune-quarantine-marker-after": "5 d",
> + "quarantine-systems-for": "60s",
> "require-cookie": "off",
> - "resend-interval": "2 s",
> - "retry-gate-closed-for": "5 s",
> + "resend-interval": "1 s",
> + "retry-gate-closed-for": "0 s",
> + "retry-window": "60 s",
> "secure-cookie": "",
> "shutdown-timeout": "10 s",
> "startup-timeout": "10 s",
> - "system-message-ack-piggyback-timeout": "0.3 s",
> + "system-message-ack-piggyback-timeout": "1 s",
> "system-message-buffer-size": 1000,
> "transport-failure-detector": {
> - "acceptable-heartbeat-pause": "20 s",
> - "heartbeat-interval": "4 s",
> - "implementation-class": "akka.remote.DeadlineFailureDetector"
> + "acceptable-heartbeat-pause": "3 s",
> + "heartbeat-interval": "1 s",
> + "implementation-class": "akka.remote.PhiAccrualFailureDetector",
> + "max-sample-size": 100,
> + "min-std-deviation": "100 ms",
> + "threshold": 7
> },
> - "trusted-selection-paths": [],
> "untrusted-mode": "off",
> - "use-dispatcher": "akka.remote.default-remote-dispatcher",
> + "use-dispatcher": "",
> "use-passive-connections": "on",
> "watch-failure-detector": {
> - "acceptable-heartbeat-pause": "10 s",
> + "acceptable-heartbeat-pause": "4 s",
> "expected-response-after": "3 s",
> "heartbeat-interval": "1 s",
> "implementation-class": "akka.remote.PhiAccrualFailureDetector",
> @@ -396,6 +355,6 @@
> "ticks-per-wheel": 512
> },
> "stdout-loglevel": "WARNING",
> - "version": "2.3.4"
> + "version": "2.2.3"
> }
> }
>
>
>
> On Thursday, August 7, 2014 7:36:27 PM UTC+8, Sean Zhong wrote:
>
>
>
>
>
>
>
>
>
> Cluster: 4 machines, each machine has 1 source actor and 1 target actor.. all
> actor started remotely by a master.
> test scenario: Each source actor will randomly deliver a 100 bytes messge
> at a time to any target actor.
>
> Test result：
>
> akka 2.2.3: 500K 100bytes message /second
> akka 2.3.4: 320K 100bytes message / second
>
> the network bandwidth occupation reflects the message throughput. network
> bandwidth is 3.2x message throughput. (as the actorPath overhead is 221
> bytes, aka 2.2 times the message size in my test)
>
> Network bandwidth usage ratio: akka2.2.3/akka 2.3.4 ~= 1.5
>
> But the CPU ratio akka2.2.3/akka 2.3.4 = 40/15 = 2.6.
>
> akka 2.2.3 use much more CPU for same throughput, does this mean akka 2.3.4
> is more efficient? But why the message throughput drops on akka 2.3.4.
>
> I dumped the effective akka conf for 2.2.3 and 2.3.4 in the attachment.
>
>
> On Thursday, August 7, 2014 6:05:54 PM UTC+8, Sean Zhong wrote:
> Can it be the case that you have a lot of system message traffic between your
> systems? Do you have lots of remote deployed actors maybe?
>
> All actors(4 source, 4 target) are created using remote actors.
>
>
> On Thursday, August 7, 2014 5:45:34 PM UTC+8, drewhk wrote:
> Hi Sean,
>
> This is interesting, we actually measured increase with 2.3.4, in fact the
> relevant changes in that version were directly targeted to increasing
> performance somewhat. One important difference is that 2.3.4 prioritizes
> internal Akka messages over user messages to improve stability. Can it be the
> case that you have a lot of system message traffic between your systems? Do
> you have lots of remote deployed actors maybe?
>
> -Endre
>
>
> On Thu, Aug 7, 2014 at 10:53 AM, Sean Zhong <[email protected]> wrote:
> Hi,
>
> When I upgrade from akka 2.2.3 to akka 2.3.4, I found the message throughput
> drops about 30%.
>
> My benchmark looks like this:
> 4 machines, each machine has 1 source actor and 1 target actor. Each source
> actor will randomly deliver a 100 bytes messge at a time to any target actor.
>
> I use default configuration. Are there default configuration changes in akka
> reference.conf which result in this behavior?
>
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Cheers,
> √
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
