Is it possible there is some kind of issue using AtLeastOnceDelivery here? I am using at least once delivery between the cluster singleton and the cluster shard Here are some more logs that are happening on the seed node that is sending the message to the shard on the "other" node:
-- so, it looks like the seed can actually see the node here... 01:37:18.015UTC [test] DEBUG a.contrib.pattern.ShardCoordinator akka.tcp://[email protected]:2551/user/sharding/ZoneCoordinator/singleton/coordinator - Shard [4] allocated at [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]] 01:37:18.015UTC [test] DEBUG akka.contrib.pattern.ShardRegion akka.tcp://[email protected]:2551/user/sharding/Zone - Shard [4] located at [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]] 01:37:18.015UTC [test] DEBUG akka.contrib.pattern.ShardRegion akka.tcp://[email protected]:2551/user/sharding/Zone - Forwarding request for shard [4] to [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]] 01:37:18.015UTC [test] DEBUG akka.contrib.pattern.ShardRegion akka.tcp://[email protected]:2551/user/sharding/Zone - Forwarding request for shard [4] to [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]] Then it continues, look like messages are flowing... 15UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.48%3A2551-0/endpointWriter - sending message RemoteMessage: [ZoneChangeSync(3,ZoneChange(Zone(denis.,[email protected],Active,2015-09-20T01:37:17.978Z,None,447a7499-5622-463d-a5d9-2d1b1227b940,Some(ZoneConnection(denis.,denis.,nzisn+4G2ldMn0q1CV3vsg==,172.17.0.46))),system,Create,Complete,2015-09-20T01:37:17.978Z,None,79f40d1a-a9b1-44dc-a3cb-be23dcef1a86))] to [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]]<+[akka.tcp://[email protected]:2551/user/sharding/Zone] from [Actor[akka://Denis/user/zoneManagerSingleton/zoneManager#1297617447]] 01:37:18.015UTC [test] DEBUG a.s.Serialization(akka://Denis) akka.serialization.Serialization(akka://Denis) - Using serializer[akka.serialization.JavaSerializer] for message [com.comcast.denis.domain.ZoneChangeSync] 01:37:18.091UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.48%3A2551-0/endpointWriter - sending message RemoteMessage: [RecordSetChange(Zone(denis.,[email protected],Active,2015-09-20T01:37:17.978Z,None,447a7499-5622-463d-a5d9-2d1b1227b940,Some(ZoneConnection(denis.,denis.,nzisn+4G2ldMn0q1CV3vsg==,172.17.0.46))),RecordSet(447a7499-5622-463d-a5d9-2d1b1227b940,test-create-ok,A,100,Pending,2015-09-20T01:37:17.998Z,None,List(AData(10.1.1.1), AData(10.2.2.2)),7d1ea0e6-a7af-455e-838b-190907083edc),system,Create,Pending,2015-09-20T01:37:18.006Z,None,None,fd69c745-350b-44bc-8000-9b983cf2bed3)] to [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]]<+[akka.tcp://[email protected]:2551/user/sharding/Zone] from [Actor[akka://Denis/temp/$W]] 01:37:18.207UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.48%3A2551-0/endpointWriter - received local message RemoteMessage: [GetShardHome(4)] to [Actor[akka://Denis/user/sharding/ZoneCoordinator/singleton/coordinator#-1961921611]]<+[akka://Denis/user/sharding/ZoneCoordinator/singleton/coordinator] from [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]()] 01:37:18.208UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.48%3A2551-0/endpointWriter - sending message RemoteMessage: [ShardHome(4,Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521])] to [Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-327960521]]<+[akka.tcp://[email protected]:2551/user/sharding/Zone] from [Actor[akka://Denis/user/sharding/ZoneCoordinator/singleton/coordinator#-1961921611]] 01:37:18.208UTC [test] DEBUG a.s.Serialization(akka://Denis) akka.serialization.Serialization(akka://Denis) - Using serializer[akka.serialization.JavaSerializer] for message [akka.contrib.pattern.ShardCoordinator$Internal$ShardHome] And then, it blows up: 01:37:18.239UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.48%3A2551-0/endpointWriter - Disassociated [akka.tcp://[email protected]:2551] <- [akka.tcp://[email protected]:2551] 01:37:18.258UTC [test] WARN a.remote.ReliableDeliverySupervisor akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.48%3A2551-0 - Association with remote system [akka.tcp://[email protected]:2551] has failed, address is now gated for [5000] ms. Reason: [Disassociated] -- and here is what is happening on the node -- *01:37:18*.018UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.47%3A2551-0/endpointWriter - received local message RemoteMessage: [Watch(Actor[akka://Denis/user/sharding/Zone#-327960521],Actor[akka.tcp://[email protected]:2551/user/sharding/Zone#-1469184628])] to [Actor[akka://Denis/user/sharding/Zone#-327960521]]<+[akka://Denis/user/sharding/Zone] from [Actor[akka://Denis/deadLetters]()] *01:37:18*.202UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.47%3A2551-0/endpointWriter - received local message RemoteMessage: [ZoneChangeSync(3,ZoneChange(Zone(denis.,[email protected],Active,2015-09-20T01:37:17.978Z,None,447a7499-5622-463d-a5d9-2d1b1227b940,Some(ZoneConnection(denis.,denis.,nzisn+4G2ldMn0q1CV3vsg==,172.17.0.46))),system,Create,Complete,2015-09-20T01:37:17.978Z,None,79f40d1a-a9b1-44dc-a3cb-be23dcef1a86))] to [Actor[akka://Denis/user/sharding/Zone#-327960521]]<+[akka://Denis/user/sharding/Zone] from [Actor[akka.tcp://[email protected]:2551/user/zoneManagerSingleton/zoneManager#1297617447]()] *01:37:18*.202UTC [test] DEBUG akka.contrib.pattern.ShardRegion akka.tcp://[email protected]:2551/user/sharding/Zone - Request shard [4] home *01:37:18*.204UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.47%3A2551-0/endpointWriter - sending message RemoteMessage: [GetShardHome(4)] to [Actor[akka.tcp://[email protected]:2551/user/sharding/ZoneCoordinator/singleton/coordinator#-1961921611]]<+[akka.tcp://[email protected]:2551/user/sharding/ZoneCoordinator/singleton/coordinator] from [Actor[akka://Denis/user/sharding/Zone#-327960521]] *01:37:18*.204UTC [test] DEBUG a.s.Serialization(akka://Denis) akka.serialization.Serialization(akka://Denis) - Using serializer[akka.serialization.JavaSerializer] for message [akka.contrib.pattern.ShardCoordinator$Internal$GetShardHome] *01:37:18*.218UTC [test] DEBUG akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.47%3A2551-0/endpointWriter - received local message RemoteMessage: [null] to [Actor[akka://Denis/user/sharding/Zone#-327960521]]<+[akka://Denis/user/sharding/Zone] from [Actor[akka.tcp://[email protected]:2551/temp/$W]()] *01:37:18*.227UTC [test] ERROR akka.remote.EndpointWriter akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDenis%40172.17.0.47%3A2551-0/endpointWriter - AssociationError [akka.tcp://[email protected]:2551] -> [akka.tcp://[email protected]:2551]: Error [Message is null] [ akka.actor.InvalidMessageException: Message is null -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
