I run cluster of Actor nodes on AWS Autoscaling groups.
Before migration to version 2.3.4, cluster discovery worked just fine: new
instances join cluster, terminated machines switched to "Unreachable" and
"Down" state.
After switching to 2.3.4, I see that terminated instances never leave a
cluster.
I see a lot of remoting errors, but node status never changed. :
17:46:28.960UTC WARN Remoting Remoting - Tried to associate with
unreachable remote address
[akka.tcp://loadtes...@ec2-54-80-144-23.compute-1.amazonaws.com:2552].
Address is now gated for 5000 ms, all messages to this address will be
delivered to dead letters. Reason: connection timed out:
ec2-54-80-144-23.compute-1.amazonaws.com/10.238.203.225:2552
17:46:37.760UTC WARN akka.remote.EndpointWriter
akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FLoadTester%40ec2-54-211-16-150.compute-1.amazonaws.com%3A2552-237/endpointWriter
- AssociationError
[akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552] ->
[akka.tcp://loadtes...@ec2-54-211-16-150.compute-1.amazonaws.com:2552]:
Error [Invalid address:
akka.tcp://loadtes...@ec2-54-211-16-150.compute-1.amazonaws.com:2552] [
akka.remote.InvalidAssociation: Invalid address:
akka.tcp://loadtes...@ec2-54-211-16-150.compute-1.amazonaws.com:2552
Caused by: akka.remote.transport.Transport$InvalidAssociationException:
connection timed out:
ec2-54-211-16-150.compute-1.amazonaws.com/10.225.5.195:2552
]
17:46:37.760UTC WARN Remoting Remoting - Tried to associate with
unreachable remote address
[akka.tcp://loadtes...@ec2-54-211-16-150.compute-1.amazonaws.com:2552].
Address is now gated for 5000 ms, all messages to this address will be
delivered to dead letters. Reason: connection timed out:
ec2-54-211-16-150.compute-1.amazonaws.com/10.225.5.195:2552
17:46:44.960UTC WARN akka.remote.EndpointWriter
akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FLoadTester%40ec2-54-225-3-180.compute-1.amazonaws.com%3A2552-238/endpointWriter
- AssociationError
[akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552] ->
[akka.tcp://loadtes...@ec2-54-225-3-180.compute-1.amazonaws.com:2552]:
Error [Invalid address:
akka.tcp://loadtes...@ec2-54-225-3-180.compute-1.amazonaws.com:2552] [
akka.remote.InvalidAssociation: Invalid address:
akka.tcp://loadtes...@ec2-54-225-3-180.compute-1.amazonaws.com:2552
Caused by: akka.remote.transport.Transport$InvalidAssociationException:
connection timed out:
ec2-54-225-3-180.compute-1.amazonaws.com/10.236.143.47:2552
]
17:46:44.960UTC WARN Remoting Remoting - Tried to associate with
unreachable remote address
[akka.tcp://loadtes...@ec2-54-225-3-180.compute-1.amazonaws.com:2552].
Address is now gated for 5000 ms, all messages to this address will be
delivered to dead letters. Reason: connection timed out:
ec2-54-225-3-180.compute-1.amazonaws.com/10.236.143.47:2552
17:46:48.558UTC WARN akka.remote.EndpointWriter
akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FLoadTester%40ec2-50-16-180-25.compute-1.amazonaws.com%3A2552-239/endpointWriter
- AssociationError
[akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552] ->
[akka.tcp://loadtes...@ec2-50-16-180-25.compute-1.amazonaws.com:2552]:
Error [Invalid address:
akka.tcp://loadtes...@ec2-50-16-180-25.compute-1.amazonaws.com:2552] [
akka.remote.InvalidAssociation: Invalid address:
akka.tcp://loadtes...@ec2-50-16-180-25.compute-1.amazonaws.com:2552
Caused by: akka.remote.transport.Transport$InvalidAssociationException:
connection timed out:
ec2-50-16-180-25.compute-1.amazonaws.com/10.239.45.179:2552
]
17:46:48.558UTC WARN Remoting Remoting - Tried to associate with
unreachable remote address
[akka.tcp://loadtes...@ec2-50-16-180-25.compute-1.amazonaws.com:2552].
Address is now gated for 5000 ms, all messages to this address will be
delivered to dead letters. Reason: connection timed out:
ec2-50-16-180-25.compute-1.amazonaws.com/10.239.45.179:2552
17:46:50.759UTC WARN akka.remote.EndpointWriter
akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FLoadTester%40ec2-54-80-144-23.compute-1.amazonaws.com%3A2552-240/endpointWriter
- AssociationError
[akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552] ->
[akka.tcp://loadtes...@ec2-54-80-144-23.compute-1.amazonaws.com:2552]:
Error [Invalid address:
akka.tcp://loadtes...@ec2-54-80-144-23.compute-1.amazonaws.com:2552] [
akka.remote.InvalidAssociation: Invalid address:
akka.tcp://loadtes...@ec2-54-80-144-23.compute-1.amazonaws.com:2552
Caused by: akka.remote.transport.Transport$InvalidAssociationException:
connection timed out:
ec2-54-80-144-23.compute-1.amazonaws.com/10.238.203.225:2552
]
Cluster status reported by Mbean. It's interesting that nodes from
"unreachable" section reported as "Up":
{
"self-address":
"akka.tcp://loadtes...@ec2-107-20-67-78.compute-1.amazonaws.com:2552",
"members": [
{
"address":
"akka.tcp://loadtes...@ec2-107-20-67-78.compute-1.amazonaws.com:2552",
"status": "Up"
},
{
"address":
"akka.tcp://loadtes...@ec2-50-16-180-25.compute-1.amazonaws.com:2552",
"status": "Up"
},
{
"address":
"akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552",
"status": "Up"
},
{
"address":
"akka.tcp://loadtes...@ec2-54-211-16-150.compute-1.amazonaws.com:2552",
"status": "Up"
},
{
"address":
"akka.tcp://loadtes...@ec2-54-225-3-180.compute-1.amazonaws.com:2552",
"status": "Up"
},
{
"address":
"akka.tcp://loadtes...@ec2-54-80-144-23.compute-1.amazonaws.com:2552",
"status": "Up"
},
{
"address":
"akka.tcp://loadtes...@ec2-54-89-171-156.compute-1.amazonaws.com:2552",
"status": "Up"
},
{
"address":
"akka.tcp://loadtes...@ec2-75-101-210-243.compute-1.amazonaws.com:2552",
"status": "Up"
}
],
"unreachable": [
{
"node":
"akka.tcp://loadtes...@ec2-50-16-180-25.compute-1.amazonaws.com:2552",
"observed-by": [
"akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-54-89-171-156.compute-1.amazonaws.com:2552"
]
},
{
"node":
"akka.tcp://loadtes...@ec2-54-211-16-150.compute-1.amazonaws.com:2552",
"observed-by": [
"akka.tcp://loadtes...@ec2-107-20-67-78.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-54-89-171-156.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-75-101-210-243.compute-1.amazonaws.com:2552"
]
},
{
"node":
"akka.tcp://loadtes...@ec2-54-225-3-180.compute-1.amazonaws.com:2552",
"observed-by": [
"akka.tcp://loadtes...@ec2-107-20-67-78.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-54-204-110-59.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-54-89-171-156.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-75-101-210-243.compute-1.amazonaws.com:2552"
]
},
{
"node":
"akka.tcp://loadtes...@ec2-54-80-144-23.compute-1.amazonaws.com:2552",
"observed-by": [
"akka.tcp://loadtes...@ec2-107-20-67-78.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-54-89-171-156.compute-1.amazonaws.com:2552",
"akka.tcp://loadtes...@ec2-75-101-210-243.compute-1.amazonaws.com:2552"
]
}
]
}
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
