Hello, I came across this mail thread because we experience the same issue. Currently I'm working an a small project that will consume messages from kafka and will be processed be an Akka cluster with Sharding. According to the documentation it should be possible to send message to the ShardRegion as soon you have an ActorRef, messages will be stashed until the required ShardRegion is available. During test runs we see this behaviour if we start a node for the first time, when we restart the node we're having the 'resolve of path sequence failed' debug message and message are send to DeadLetter instead of an actor in the ShardRegion. If we clean up the underlying persistence storage, the node restart correctly I was able to reproduce this issue with a small setup (see attachements). We're using Akka 2.3.4 and test it with default persistence (LevelDB) and with MongoDB, in both cases the node is not able to restart correctly.
Regards, Johan On Thursday, June 26, 2014 10:16:45 AM UTC+2, Patrik Nordwall wrote: > > Thanks for reporting. Please add your stack trace to ticket > https://github.com/akka/akka/issues/15440 > If you have any hints of how to reproduce that is very welcome, and please > add that to the ticket as well. > > Cheers, > Patrik > > > On Thu, Jun 26, 2014 at 9:58 AM, Carsten Saathoff <[email protected] > <javascript:>> wrote: > >> Hi, >> >> just for the record, I am also seeing the exact same problem with Akka >> 2.3.3. I also use the Cluster Sharding, and the events were stored with >> Akka 2.3.3 (it's a fresh setup from yesterday). This only happened after >> the system was under constant (but not excessively high) load. In between >> one of the nodes died two or three times, and then this strange recovery >> behaviour started to occur. Currently the whole cluster is basically >> unusable. I can try to create a reproducible test case, but that seems not >> so easy. >> >> best >> >> Carsten >> >> Am Dienstag, 13. Mai 2014 16:24:52 UTC+2 schrieb Patrik Nordwall: >>> >>> Thanks for the update. >>> >>> As I mentioned previously, the problem can be explained by the >>> serialization bug in Akka persistence in 2.3.0 and 2.3.1 that was fixed in >>> 2.3.2. Are you 100% sure that the data was not stored with 2.3.0 or 2.3.1 >>> and then later replayed with 2.3.2? >>> >>> Please create a ticket if you see the problem again, even though you >>> don't know exactly how to reproduce it. >>> >>> Regards, >>> Patrik >>> >>> >>> On Tue, May 13, 2014 at 11:21 AM, benoit heinrich <[email protected]> >>> wrote: >>> >>>> Hi Patrik, >>>> >>>> First of all, I'm very sorry to reply so late but I've tried to create >>>> a reproducible test case for you and it seems that I can't easily >>>> reproduce >>>> the conditions which cause that failure. >>>> >>>> Also, since I'm trying to recreate a reproducible test case, it seems >>>> that even my application works and doesn't throw that error anymore. >>>> >>>> I'm a bit puzzled, because I'm not sure if I should open a new issue as >>>> even myself can't reproduce all the time. >>>> >>>> I'm going to spend more time this week to try to reproduce and once >>>> I've got a way to reproduce all the time then I'll open the issue and >>>> attach the test case. >>>> >>>> >>>> Thanks for all, and I'll update that post once I get more news. >>>> >>>> Cheers, >>>> /Benoit >>>> >>>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>>> current/additional/faq.html >>>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>>> group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> >>> Patrik Nordwall >>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>> Twitter: @patriknw >>> JOIN US. REGISTER TODAY! <http://www.scaladays.org/> >>> Scala <http://www.scaladays.org/> >>> Days <http://www.scaladays.org/> >>> June 16th-18th, <http://www.scaladays.org/> >>> Berlin <http://www.scaladays.org/> >>> >>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
[DEBUG] [08/09/2014 06:46:41.109] [main] [EventStream(akka://DoubleIt)] logger log1-Logging$DefaultLogger started [DEBUG] [08/09/2014 06:46:41.110] [main] [EventStream(akka://DoubleIt)] Default Loggers started [INFO] [08/09/2014 06:46:41.168] [main] [Remoting] Starting remoting [INFO] [08/09/2014 06:46:41.362] [main] [Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:2551] [INFO] [08/09/2014 06:46:41.376] [main] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Starting up... [INFO] [08/09/2014 06:46:41.456] [main] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [08/09/2014 06:46:41.456] [main] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Started up successfully [INFO] [08/09/2014 06:46:41.464] [DoubleIt-akka.actor.default-dispatcher-16] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar [INFO] [08/09/2014 06:46:41.466] [DoubleIt-akka.actor.default-dispatcher-16] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Metrics collection has started successfully [DEBUG] [08/09/2014 06:46:41.492] [DoubleIt-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Request shard [0] home [WARN] [08/09/2014 06:46:41.533] [DoubleIt-akka.remote.default-remote-dispatcher-5] [Remoting] Tried to associate with unreachable remote address [akka.tcp://[email protected]:2552]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: /127.0.0.1:2552 [INFO] [08/09/2014 06:46:46.493] [DoubleIt-akka.actor.default-dispatcher-14] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2551] is JOINING, roles [] [INFO] [08/09/2014 06:46:47.480] [DoubleIt-akka.actor.default-dispatcher-14] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2551] to [Up] [DEBUG] [08/09/2014 06:46:47.482] [DoubleIt-akka.actor.default-dispatcher-14] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Coordinator moved from [] to [akka.tcp://[email protected]:2551] [INFO] [08/09/2014 06:46:47.486] [DoubleIt-akka.actor.default-dispatcher-20] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator] Singleton manager [akka.tcp://[email protected]:2551] starting singleton actor [INFO] [08/09/2014 06:46:47.488] [DoubleIt-akka.actor.default-dispatcher-20] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator] ClusterSingletonManager state change [Start -> Oldest] [DEBUG] [08/09/2014 06:46:47.569] [DoubleIt-akka.actor.default-dispatcher-2] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator/singleton/coordinator] ShardRegion registered: [Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]] [DEBUG] [08/09/2014 06:46:47.575] [DoubleIt-akka.persistence.dispatchers.default-plugin-dispatcher-24] [akka.serialization.Serialization(akka://DoubleIt)] Using serializer[akka.persistence.serialization.MessageSerializer] for message [akka.persistence.PersistentImpl] [DEBUG] [08/09/2014 06:46:47.584] [DoubleIt-akka.persistence.dispatchers.default-plugin-dispatcher-24] [akka.serialization.Serialization(akka://DoubleIt)] Using serializer[akka.serialization.JavaSerializer] for message [akka.contrib.pattern.ShardCoordinator$Internal$ShardRegionRegistered] [DEBUG] [08/09/2014 06:46:47.631] [DoubleIt-akka.actor.default-dispatcher-14] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Retry request for shard [0] homes [DEBUG] [08/09/2014 06:46:47.634] [DoubleIt-akka.persistence.dispatchers.default-plugin-dispatcher-24] [akka.serialization.Serialization(akka://DoubleIt)] Using serializer[akka.serialization.JavaSerializer] for message [akka.contrib.pattern.ShardCoordinator$Internal$ShardHomeAllocated] [DEBUG] [08/09/2014 06:46:47.638] [DoubleIt-akka.actor.default-dispatcher-2] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator/singleton/coordinator] Shard [0] allocated at [Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]] [DEBUG] [08/09/2014 06:46:47.638] [DoubleIt-akka.actor.default-dispatcher-14] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Shard [0] located at [Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]] [DEBUG] [08/09/2014 06:46:47.639] [DoubleIt-akka.actor.default-dispatcher-14] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Starting entry [10] in shard [0] [DEBUG] [08/09/2014 06:46:47.640] [DoubleIt-akka.actor.default-dispatcher-14] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Message [DoubleIt$Get] for shard [0] sent to entry Get(20)
DoubleIt.scala
Description: Binary data
[DEBUG] [08/09/2014 06:47:22.638] [main] [EventStream(akka://DoubleIt)] logger log1-Logging$DefaultLogger started [DEBUG] [08/09/2014 06:47:22.638] [main] [EventStream(akka://DoubleIt)] Default Loggers started [INFO] [08/09/2014 06:47:22.696] [main] [Remoting] Starting remoting [INFO] [08/09/2014 06:47:22.838] [main] [Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:2551] [INFO] [08/09/2014 06:47:22.858] [main] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Starting up... [INFO] [08/09/2014 06:47:22.927] [main] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [08/09/2014 06:47:22.927] [main] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Started up successfully [INFO] [08/09/2014 06:47:22.937] [DoubleIt-akka.actor.default-dispatcher-5] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar [INFO] [08/09/2014 06:47:22.938] [DoubleIt-akka.actor.default-dispatcher-5] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Metrics collection has started successfully [DEBUG] [08/09/2014 06:47:22.964] [DoubleIt-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Request shard [0] home [WARN] [08/09/2014 06:47:23.008] [DoubleIt-akka.remote.default-remote-dispatcher-7] [Remoting] Tried to associate with unreachable remote address [akka.tcp://[email protected]:2552]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: /127.0.0.1:2552 [INFO] [08/09/2014 06:47:27.967] [DoubleIt-akka.actor.default-dispatcher-19] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2551] is JOINING, roles [] [INFO] [08/09/2014 06:47:28.950] [DoubleIt-akka.actor.default-dispatcher-19] [Cluster(akka://DoubleIt)] Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2551] to [Up] [DEBUG] [08/09/2014 06:47:28.953] [DoubleIt-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Coordinator moved from [] to [akka.tcp://[email protected]:2551] [INFO] [08/09/2014 06:47:28.955] [DoubleIt-akka.actor.default-dispatcher-17] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator] Singleton manager [akka.tcp://[email protected]:2551] starting singleton actor [INFO] [08/09/2014 06:47:28.957] [DoubleIt-akka.actor.default-dispatcher-17] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator] ClusterSingletonManager state change [Start -> Oldest] [DEBUG] [08/09/2014 06:47:29.028] [DoubleIt-akka.persistence.dispatchers.default-replay-dispatcher-26] [akka.serialization.Serialization(akka://DoubleIt)] Using serializer[akka.persistence.serialization.MessageSerializer] for message [akka.persistence.PersistentRepr] [DEBUG] [08/09/2014 06:47:29.079] [DoubleIt-akka.persistence.dispatchers.default-replay-dispatcher-26] [LocalActorRefProvider(akka://DoubleIt)] resolve of path sequence [/user/sharding/DoubleIt#213253828] failed [DEBUG] [08/09/2014 06:47:29.080] [DoubleIt-akka.persistence.dispatchers.default-replay-dispatcher-26] [LocalActorRefProvider(akka://DoubleIt)] resolve of path sequence [/user/sharding/DoubleIt#213253828] failed [DEBUG] [08/09/2014 06:47:29.084] [DoubleIt-akka.persistence.dispatchers.default-replay-dispatcher-26] [LocalActorRefProvider(akka://DoubleIt)] resolve of path sequence [/user/sharding/DoubleIt#213253828] failed [DEBUG] [08/09/2014 06:47:29.084] [DoubleIt-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator/singleton/coordinator] receiveRecover ShardRegionRegistered(Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]) [DEBUG] [08/09/2014 06:47:29.084] [DoubleIt-akka.persistence.dispatchers.default-replay-dispatcher-26] [LocalActorRefProvider(akka://DoubleIt)] resolve of path sequence [/user/sharding/DoubleIt#213253828] failed [DEBUG] [08/09/2014 06:47:29.085] [DoubleIt-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator/singleton/coordinator] receiveRecover ShardHomeAllocated(0,Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]) [DEBUG] [08/09/2014 06:47:29.091] [DoubleIt-akka.actor.default-dispatcher-22] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator/singleton/coordinator] ShardRegion registered: [Actor[akka://DoubleIt/user/sharding/DoubleIt#1398635940]] [DEBUG] [08/09/2014 06:47:29.095] [DoubleIt-akka.persistence.dispatchers.default-plugin-dispatcher-24] [akka.serialization.Serialization(akka://DoubleIt)] Using serializer[akka.persistence.serialization.MessageSerializer] for message [akka.persistence.PersistentImpl] [DEBUG] [08/09/2014 06:47:29.098] [DoubleIt-akka.persistence.dispatchers.default-plugin-dispatcher-24] [akka.serialization.Serialization(akka://DoubleIt)] Using serializer[akka.serialization.JavaSerializer] for message [akka.contrib.pattern.ShardCoordinator$Internal$ShardRegionRegistered] [DEBUG] [08/09/2014 06:47:29.108] [DoubleIt-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Retry request for shard [0] homes [DEBUG] [08/09/2014 06:47:29.110] [DoubleIt-akka.actor.default-dispatcher-22] [akka.tcp://[email protected]:2551/user/sharding/DoubleItCoordinator/singleton/coordinator] ShardRegion terminated: [Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]] [DEBUG] [08/09/2014 06:47:29.110] [DoubleIt-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Shard [0] located at [Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]] [DEBUG] [08/09/2014 06:47:29.111] [DoubleIt-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Forwarding request for shard [0] to [Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]] [DEBUG] [08/09/2014 06:47:29.111] [DoubleIt-akka.persistence.dispatchers.default-plugin-dispatcher-24] [akka.serialization.Serialization(akka://DoubleIt)] Using serializer[akka.serialization.JavaSerializer] for message [akka.contrib.pattern.ShardCoordinator$Internal$ShardRegionTerminated] [DEBUG] [08/09/2014 06:47:29.112] [DoubleIt-akka.actor.default-dispatcher-5] [akka.tcp://[email protected]:2551/user/sharding/DoubleIt] Region [Actor[akka://DoubleIt/user/sharding/DoubleIt#213253828]] with shards [0] terminated Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after [20 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at DoubleIt$delayedInit$body.apply(DoubleIt.scala:33) at scala.Function0$class.apply$mcV$sp(Function0.scala:40) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:71) at scala.App$$anonfun$main$1.apply(App.scala:71) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32) at scala.App$class.main(App.scala:71) at DoubleIt$.main(DoubleIt.scala:9) at DoubleIt.main(DoubleIt.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
