Public bug reported:
We are running eucalyptus 1.6.2-0ubuntu27 on lucid beta1 in MANAGED-
NOVLAN. I will retest as soon as is feasible with ubuntu30 but as I see
no mention of this issue/fix in the changelog I wanted to get the
information in your hands.
Eucalyptus has trouble allocating additional VMs to existing security
groups in some cases. I tried several tests and saw very similar
results. Eucalyptus allows you to request VMs in a given security
group. Once all the VMs are running an additional euca-run-instances
request for that security group will fail and in some cases the network
associated with that security group will be removed from iptables (even
if there are running VMs within that security group). The network that
was freed up can be re-allocated to another security group but new VMs
requested in that security group fail with the same "failed to add host"
message.
---------------------------------------------------
A typical cycle looks like this (command-line interspersed with snippets of
cc.log):
$ euca-run-instances -n 250 -g default…
[Thu Apr 15 14:14:51 2010][001325][EUCAINFO ] StartNetwork(): called
[Thu Apr 15 14:14:51 2010][001324][EUCAINFO ] ConfigureNetwork(): called
[Thu Apr 15 14:14:51 2010][001324][EUCAINFO ] vnetTableRule(): applying
iptables rule: -A user-default -s 0.0.0.0/0 -d 10.0.8.0/24 -p tcp --dport 22:22
-j ACCEPT
[Thu Apr 15 14:14:51 2010][001327][EUCAINFO ] RunInstances(): called
#….Proceeds to run 250 instances successfully…..
$ euca-run-instances -n 1 -g default….
[Thu Apr 15 14:29:46 2010][001376][EUCAINFO ] StartNetwork(): called
[Thu Apr 15 14:29:46 2010][001368][EUCAINFO ] ConfigureNetwork(): called
[Thu Apr 15 14:29:46 2010][001368][EUCAINFO ] vnetTableRule(): applying
iptables rule: -A user-default -s 0.0.0.0/0 -d 10.0.8.0/24 -p tcp --dport 22:22
-j ACCEPT
[Thu Apr 15 14:29:46 2010][001328][EUCAINFO ] RunInstances(): called
[Thu Apr 15 14:29:46 2010][001328][EUCAERROR ] vnetAddHost(): failed to add
host d0:0d:3B:E6:07:11 on vlan 10
[Thu Apr 15 14:29:46 2010][001328][EUCAERROR ] RunInstances(): could not
find/initialize any free network address, failing doRunInstances()
#…..After 15 minutes instance goes to terminated and TerminateInstance()
is called many times (once per NC?)…….
[Thu Apr 15 14:39:51 2010][005458][EUCAERROR ] ERROR: TerminateInstance() could
not be invoked (check NC host, port, and credentia
ls)
[Thu Apr 15 14:39:51 2010][001326][EUCAINFO ] TerminateInstances(): calling
terminate instance (i-3BE60711) on (192.168.1.2)
[Thu Apr 15 14:39:51 2010][005459][EUCAERROR ] ERROR: TerminateInstance() could
not be invoked (check NC host, port, and credentia
ls)
[Thu Apr 15 14:39:51 2010][001326][EUCAINFO ] TerminateInstances(): calling
terminate instance (i-3BE60711) on (192.168.1.3)
[Thu Apr 15 14:39:51 2010][005460][EUCAERROR ] ERROR: TerminateInstance() could
not be invoked (check NC host, port, and credentia
ls)
[Thu Apr 15 14:39:51 2010][001326][EUCAINFO ] TerminateInstances(): calling
terminate instance (i-3BE60711) on (192.168.1.4)
[Thu Apr 15 14:39:51 2010][005461][EUCAERROR ] ERROR: TerminateInstance() could
not be invoked (check NC host, port, and credentia
ls)
#……It then removes the network allocated for the user's default security
group even though there are 250 running VMs!!!……
[Thu Apr 15 14:40:00 2010][001328][EUCAINFO ] StopNetwork(): called
#iptables shows that the chain user-default has disappeared!
---------------------------------------------------
I tried many different combinations of numbers of nodes, etc.
(ADDRSPERNET is 256)
250 + 1 additional (the 1 additional failed, network was removed and VMs are
inaccessible)
100 + 1 additional (the 1 additional failed, network was removed and VMs are
inaccessible)
20 + 20 additional (the 20 additional failed, network was removed and VMs are
inaccessible)
I did have some success adding to to existing security groups by 10 or
20 nodes at a time. One security group grew to 80 nodes before I
received the "failed to add host" messages. It seemed I was more
successful when I was making requests rapidly (waiting only a few
minutes between requests) rather than waiting for all the nodes to
allocate in a given reservation. I am at a loss to the exact cause
because some security groups are allowed to expand while others are cut
off from receiving additional IPs well before they reach ADDRSPERNET.
** Affects: eucalyptus (Ubuntu)
Importance: Undecided
Status: New
--
Second euca-run-instance request in same security group causes eucalyptus to
remove network assoicated with security group
https://bugs.launchpad.net/bugs/564355
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
