Hi Brian,
On Mon, Jan 27, 2014 at 12:54 PM, Brian Fowler <[email protected]>wrote: > Hi Endre, > > Upping the > akka.remote.transport-failure-detector.acceptable-heartbeat-pause timeout, > made the systems stay connected with each other for longer however they > still failed in the same manner after time. > This indicates load issues though. > Due to our release schedules we are rolling back to 2.1.4. I'll have to > investigate further at another time. > Sorry to hear that but I understand the choice. I think you should investigate the issue a bit more even on 2.1.4 because the fact that the transport Failure Detector triggers means that it misses many heartbeats -- which can indicate high latencies (can be CPU, GC pressure or network load). These problems are not apparent in 2.1.x because its remoting does not monitor response times. > > Thanks for you help, > Thank you for your patience. -Endre > > Brian > > > > On Friday, 24 January 2014 15:59:21 UTC, Akka Team wrote: > >> Hi Brian, >> >> >> >> On Fri, Jan 24, 2014 at 4:44 PM, Brian Fowler <[email protected]>wrote: >> >>> Thanks Endre, I'll try increasing the akka.remote.transport-failure- >>> detector.acceptable-heartbeat-pause >>> >>> However I'm a little concerned in general that the ActorSystems are not >>> recovering their connectivity without restarting the ActorSystem1 JVM. Do I >>> need to change the config or but in a different Supervisor strategy to make >>> this work? >>> >> >> Restart is not needed, unless one of the nodes quarantined the other (but >> that would be in the logs). I suspect something else here: from your log it >> seems that system one connects fine: >> >> akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127. >> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >> >> As I said, I suspect the direction from system2 to system1 to be >> problematic. I think the two systems are trying to connect at the same >> time, and therefore the connection from system1 on system2 to is accepted >> for reading only, but the writing direction (system2 to system1) is failing >> for some reason (Connection Refused). >> >> You can debug what messages are actually getting through: >> >> akka.remote.log-received-messages = on >> akka.remote.log-sent-messages = on >> >> Then you will see the actual remote messages that are attempted to be >> sent and that are actually received in the log. >> >> Btw without a longer log it is hard to see the whole picture. >> >> -Endre >> >> >>> >>> ActorSystem1 seems to make some attempt at it but it doesn't seem to >>> work.. >>> >>> >>> [2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor. >>> default-dispatcher-11] akka.remote.EndpointWriter - restarted >>> [2014-01-24 14:26:14,632] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>> akka.remote.EndpointWriter - restarted >>> [2014-01-24 14:26:14,633] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.AkkaProtocolManager - now supervising >>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>> 1273774939] >>> [2014-01-24 14:26:14,633] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.AkkaProtocolManager - now supervising >>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>> 1273774939] >>> [2014-01-24 14:26:14,634] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.ProtocolStateActor - started >>> (akka.remote.transport.ProtocolStateActor@549ad840) >>> [2014-01-24 14:26:14,634] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.ProtocolStateActor - started >>> (akka.remote.transport.ProtocolStateActor@549ad840) >>> [2014-01-24 14:26:17,126] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - Associated [akka.tcp://[email protected] >>> .0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:17,126] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - Associated [akka.tcp://[email protected] >>> .0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:17,127] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader- >>> akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] >>> [2014-01-24 14:26:17,127] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader- >>> akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@1b >>> d97d0d) >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@1b >>> d97d0d) >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486] >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486] >>> >>> >>> On Friday, 24 January 2014 15:23:05 UTC, Akka Team wrote: >>> >>>> Hi Brian, >>>> >>>> >>>> On Fri, Jan 24, 2014 at 4:14 PM, Brian Fowler <[email protected]>wrote: >>>> >>>>> >>>>> Hi Endre, >>>>> >>>>> No the two systems can comunicate initially, the problem just occurs >>>>> after a quiet period of about 5 minutes. The problem also does not occur >>>>> with Akka 2.1.4 so I don't think it is a network issue. >>>>> >>>> >>>> Akka 2.1.4 had no real recovery mechanism (and other important >>>> features) so it probably never triggers it. >>>> >>>> That the systems can communicate initially is not really a proof >>>> because actor systems reuse the incoming connection when possible. So it >>>> can be like this: >>>> >>>> 1. system1 connects to system2 on address akka.tcp://DataAugmentation@ >>>> server002:2552 (this direction is fine) >>>> 2. communication is fine >>>> 3. for some reason system1 shuts of the connection (most probably >>>> heartbeat misses, you should increase the akka.remote.transport-failure- >>>> detector.acceptable-heartbeat-pause setting to something like 10s -- >>>> but I need more log entries). >>>> 4. Now system2 tries to recover by connecting to system1 on address >>>> 127.0.0.1:60789 >>>> and this direction fails. >>>> >>>> What is suspicious to me is this error: akka.remote.transport.netty.Ne >>>> ttyTransport$$anonfun$associate$1$$anon$2: Connection refused: / >>>> 127.0.0.1:60789 >>>> This is thrown by netty and means the connection is not accepted. >>>> >>>> You can try to force this scenario by setting akka.remote.use-passive- >>>> connections = off on both nodes (this prevents system2 to reuse the >>>> connection from system1). Then we might see what happens. >>>> >>>> -Endre >>>> >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Brian >>>>> >>>>> >>>>> On Friday, 24 January 2014 15:05:12 UTC, Akka Team wrote: >>>>> >>>>>> Hi Brian, >>>>>> >>>>>> >>>>>> On ActorSystem1 I get >>>>>>> >>>>>> >>>>>> Actually the interesting part is more probably earlier in this log. >>>>>> Btw, is the port 60789 firewalled? It seems like ActorSystem2 is trying >>>>>> to >>>>>> connect back to system1 and it cannot. >>>>>> >>>>>> -Endre >>>>>> >>>>>> >>>>>>> >>>>>>> [2014-01-24 14:26:14,620] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>>>> akka.remote.transport.ProtocolStateActor - stopped >>>>>>> [2014-01-24 14:26:14,620] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>>>> akka.remote.transport.ProtocolStateActor - stopped >>>>>>> [2014-01-24 14:26:14,620] [INFO] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.Associa >>>>>>> tionHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] >>>>>>> to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmana >>>>>>> ger.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server0 >>>>>>> 02%3A2552-1#1651503039] was not delivered. [1] dead letters >>>>>>> encountered. This logging can be turned off or adjusted with >>>>>>> configuration >>>>>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>>>>> shutdown'. >>>>>>> [2014-01-24 14:26:14,620] [INFO] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.Associa >>>>>>> tionHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] >>>>>>> to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmana >>>>>>> ger.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server0 >>>>>>> 02%3A2552-1#1651503039] was not delivered. [1] dead letters >>>>>>> encountered. This logging can be turned off or adjusted with >>>>>>> configuration >>>>>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>>>>> shutdown'. >>>>>>> [2014-01-24 14:26:14,621] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>>>> akka.remote.EndpointReader - stopped >>>>>>> [2014-01-24 14:26:14,621] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>>>> akka.remote.EndpointReader - stopped >>>>>>> [2014-01-24 14:26:14,623] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>>>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>>>> erver002:2552] >>>>>>> [2014-01-24 14:26:14,623] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>>>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>>>> erver002:2552] >>>>>>> [2014-01-24 14:26:14,628] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>>>>>> akka.remote.EndpointWriter - restarting >>>>>>> [2014-01-24 14:26:14,628] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>>>>>> akka.remote.EndpointWriter - restarting >>>>>>> [2014-01-24 14:26:14,630] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>>>> erver002:2552] >>>>>>> [2014-01-24 14:26:14,630] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>>>> erver002:2552] >>>>>>> [2014-01-24 14:26:14,630] [INFO] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTr >>>>>>> ansportAdapter$DisassociateUnderlying] from >>>>>>> Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/ >>>>>>> system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F% >>>>>>> 2FDataAugmentation%40server002%3A2552-1#1651503039] was not >>>>>>> delivered. [2] dead letters encountered. This logging can be turned off >>>>>>> or >>>>>>> adjusted with configuration settings 'akka.log-dead-letters' and >>>>>>> 'akka.log-dead-letters-during-shutdown'. >>>>>>> [2014-01-24 14:26:14,630] [INFO] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTr >>>>>>> ansportAdapter$DisassociateUnderlying] from >>>>>>> Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/ >>>>>>> system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F% >>>>>>> 2FDataAugmentation%40server002%3A2552-1#1651503039] was not >>>>>>> delivered. [2] dead letters encountered. This logging can be turned off >>>>>>> or >>>>>>> adjusted with configuration settings 'akka.log-dead-letters' and >>>>>>> 'akka.log-dead-letters-during-shutdown'. >>>>>>> [2014-01-24 14:26:14,632] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>>>>>> akka.remote.EndpointWriter - restarted >>>>>>> [2014-01-24 14:26:14,632] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>>>>>> akka.remote.EndpointWriter - restarted >>>>>>> [2014-01-24 14:26:14,633] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>>>> akka.remote.transport.AkkaProtocolManager - now supervising >>>>>>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmana >>>>>>> ger.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server0 >>>>>>> 02%3A2552-2#1273774939] >>>>>>> [2014-01-24 14:26:14,633] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>>>> akka.remote.transport.AkkaProtocolManager - now supervising >>>>>>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmana >>>>>>> ger.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server0 >>>>>>> 02%3A2552-2#1273774939] >>>>>>> [2014-01-24 14:26:14,634] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>>>> akka.remote.transport.ProtocolStateActor - started >>>>>>> (akka.remote.transport.ProtocolStateActor@549ad840) >>>>>>> [2014-01-24 14:26:14,634] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>>>> akka.remote.transport.ProtocolStateActor - started >>>>>>> (akka.remote.transport.ProtocolStateActor@549ad840) >>>>>>> [2014-01-24 14:26:17,126] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>>>> akka.remote.EndpointWriter - Associated [akka.tcp:// >>>>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>>>> erver002:2552] >>>>>>> [2014-01-24 14:26:17,126] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>>>> akka.remote.EndpointWriter - Associated [akka.tcp:// >>>>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>>>> erver002:2552] >>>>>>> [2014-01-24 14:26:17,127] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>>>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>>>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FD >>>>>>> ataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader- >>>>>>> akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] >>>>>>> [2014-01-24 14:26:17,127] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>>>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>>>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FD >>>>>>> ataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader- >>>>>>> akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] >>>>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>>>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@1b >>>>>>> d97d0d) >>>>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>>>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@1b >>>>>>> d97d0d) >>>>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>>>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>>>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FD >>>>>>> ataAugmentation%40server002%3A2552-0/endpointWriter#226701486] >>>>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>>>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>>>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FD >>>>>>> ataAugmentation%40server002%3A2552-0/endpointWriter#226701486] >>>>>>> >>>>>>> On ActorSystem2 >>>>>>> >>>>>>> [2014-01-24 14:26:16,473] [ERROR] >>>>>>> [DataAugmentation-akka.actor.default-dispatcher-234] >>>>>>> akka.remote.EndpointWriter - AssociationError >>>>>>> [akka.tcp://DataAugmentation@ >>>>>>> server002:2552] -> [akka.tcp://[email protected]:60789]: Error >>>>>>> [Association failed with [akka.tcp://[email protected]:60789]] >>>>>>> [ >>>>>>> akka.remote.EndpointAssociationException: Association failed with >>>>>>> [akka.tcp://[email protected]:60789] >>>>>>> Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$ >>>>>>> associate$1$$anon$2: Connection refused: /127.0.0.1:60789 >>>>>>> >>>>>>> -- >>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>> p/akka-user >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Akka User List" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> >>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Akka Team >>>>>> Typesafe - The software stack for applications that scale >>>>>> Blog: letitcrash.com >>>>>> Twitter: @akkateam >>>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>> p/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>>> >>>> -- >>>> Akka Team >>>> Typesafe - The software stack for applications that scale >>>> Blog: letitcrash.com >>>> Twitter: @akkateam >>>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> Akka Team >> Typesafe - The software stack for applications that scale >> Blog: letitcrash.com >> Twitter: @akkateam >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/groups/opt_out. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
