Bug#863385: cloud.debian.org: Latest stretch AMI (ami-772ab761) fails to launch successfully into VPCs

[Andy Sizer]

On Fri, 26 May 2017 at 20:52 Ross Vandegrift  wrote:

> On Fri, May 26, 2017 at 09:15:35AM +0700, Andy Sizer wrote:
> > I tried to launch in instance from the latest stretch ami
> > (ami-772ab761 i.e. us-east-1) into a VPC. It failed - more
> > specifically networking.service failed.
>
> I just tried this AMI and all looks good to me.  Anything unusual about
> your
> VPC?  I'm using a default VPC without IPv6.
>
>
The VPC in question is configured as follows:

* IPv6 CIDR - x.y.z.0/24
* IPv6 CIDR - none
* DHCP options: domain-name = ec2.internal, domain-name-servers =
AmazonProvidedDNS
* Route table - nothing interesting here - just a 'gateway'
* ClassicLink - disabled
* Network ACL - again nothing interesting - the default inbound, outbound
rules, 1 associated subnet (i.e. the same as the VPC's)
* Tenancy - default
* DNS resolution - yes
* DNS hostnames - yes
* ClassicLink DNS support - no

It's not obvious to me that there's anything unusual there. Not sure when
this VPC was created, it predates my time here (> 2 years). FWIW the wheezy
and jessie AMIs launched fine into this VPC.

Anyway, I then did the following:

* launch ami-772ab761 in EC2-classic
* edit /etc/network/interfaces - add an entry for 'ens3' before 'eth0' -
same config
* create a new AMI
* launch new AMI into our VPC
* It came up 'OK' and was accessible via SSH.

The following might be informative:

root@ip-10-42-42-64:/var/log# ip addr
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
2: ens3:  mtu 9001 qdisc mq state UP group
default qlen 1000
link/ether 0e:0c:48:21:5d:14 brd ff:ff:ff:ff:ff:ff
inet 10.42.42.64/24 brd 10.42.42.255 scope global ens3
   valid_lft forever preferred_lft forever
inet6 fe80::c0c:48ff:fe21:5d14/64 scope link
   valid_lft forever preferred_lft forever

root@ip-10-42-42-64:/var/log# ifconfig
ens3: flags=4163  mtu 9001
inet 10.42.42.64  netmask 255.255.255.0  broadcast 10.42.42.255
inet6 fe80::c0c:48ff:fe21:5d14  prefixlen 64  scopeid 0x20
ether 0e:0c:48:21:5d:14  txqueuelen 1000  (Ethernet)
RX packets 806  bytes 81137 (79.2 KiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 673  bytes 123224 (120.3 KiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10
loop  txqueuelen 1  (Local Loopback)
RX packets 0  bytes 0 (0.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 0  bytes 0 (0.0 B)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@ip-10-42-42-64:/var/log# ifquery --read-environment --list
lo
ens3
eth0

[ ens3 AND eth0 ?? ]

root@ip-10-42-42-64:/var/log# head -n 18 /var/log/cloud-init-output.log
Cloud-init v. 0.7.9 running 'init-local' at Mon, 29 May 2017 02:34:12
+. Up 35.35 seconds.
Cloud-init v. 0.7.9 running 'init' at Mon, 29 May 2017 02:34:16 +. Up
39.66 seconds.
ci-info: Net device
info+
ci-info:
++--++-+---+---+
ci-info: | Device |  Up  |Address |   Mask  | Scope |
Hw-Address|
ci-info:
++--++-+---+---+
ci-info: |  lo:   | True |   127.0.0.1|255.0.0.0|   .   |
  . |
ci-info: |  lo:   | True |   .|.|   d   |
  . |
ci-info: | eth0:  | True | 10.233.189.228 | 255.255.255.192 |   .   |
22:00:0a:e9:bd:e4 |
ci-info: | eth0:  | True |   .|.|   d   |
22:00:0a:e9:bd:e4 |
ci-info:
++--++-+---+---+
ci-info: +Route IPv4
info+
ci-info:
+---+++-+---+---+
ci-info: | Route |  Destination   |Gateway | Genmask |
Interface | Flags |
ci-info:
+---+++-+---+---+
ci-info: |   0   |0.0.0.0 | 10.233.189.193 | 0.0.0.0 |
 eth0   |   UG  |
ci-info: |   1   | 10.233.189.192 |0.0.0.0 | 255.255.255.192 |
 eth0   |   U   |
ci-info:
+---+++-+---+---+

Let us know if there's anything else you'd like.

Seems like we have a workaround. But we'd prefer to be provisioning an
'official' AMI - if nothing else the workaround adds additional complexity
to our automation 

Bug#863385: cloud.debian.org: Latest stretch AMI (ami-772ab761) fails to launch successfully into VPCs

[Ross Vandegrift]

On Fri, May 26, 2017 at 09:15:35AM +0700, Andy Sizer wrote:
> I tried to launch in instance from the latest stretch ami
> (ami-772ab761 i.e. us-east-1) into a VPC. It failed - more
> specifically networking.service failed. 

I just tried this AMI and all looks good to me.  Anything unusual about your
VPC?  I'm using a default VPC without IPv6.

$ head -n 18 /var/log/cloud-init-output.log
Cloud-init v. 0.7.9 running 'init-local' at Fri, 26 May 2017 13:35:30 +. Up 
13.26 seconds.
Cloud-init v. 0.7.9 running 'init' at Fri, 26 May 2017 13:35:32 +. Up 16.00 
seconds.
ci-info: +++Net device 
info+++
ci-info: 
++--+---+---+---+---+
ci-info: | Device |  Up  |Address|  Mask | Scope | 
Hw-Address|
ci-info: 
++--+---+---+---+---+
ci-info: |  lo:   | True |   127.0.0.1   |   255.0.0.0   |   .   | .
 |
ci-info: |  lo:   | True |   .   |   .   |   d   | .
 |
ci-info: | eth0:  | True | 172.31.20.116 | 255.255.240.0 |   .   | 
0e:a1:bc:8f:db:d6 |
ci-info: | eth0:  | True |   .   |   .   |   d   | 
0e:a1:bc:8f:db:d6 |
ci-info: 
++--+---+---+---+---+
ci-info: +Route IPv4 
info+
ci-info: 
+---+-+-+---+---+---+
ci-info: | Route | Destination |   Gateway   |Genmask| Interface | 
Flags |
ci-info: 
+---+-+-+---+---+---+
ci-info: |   0   |   0.0.0.0   | 172.31.16.1 |0.0.0.0|eth0   |   UG 
 |
ci-info: |   1   | 172.31.16.0 |   0.0.0.0   | 255.255.240.0 |eth0   |   U  
 |
ci-info: 
+---+-+-+---+---+---+

Ross

Bug#863385: cloud.debian.org: Latest stretch AMI (ami-772ab761) fails to launch successfully into VPCs

[Andy Sizer]

Package: cloud.debian.org
Severity: important

Hi,

I tried to launch in instance from the latest stretch ami (ami-772ab761 i.e. 
us-east-1) into a VPC. It failed - more specifically networking.service failed. 

Here's the relevant output from syslog:

[ [0;1;31mFAILED [0m] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
 Starting Initial cloud-init job (metadata service crawler)...
[ [0;32m  OK   [0m] Reached target Network.
[9.288945] cloud-init[390]: Cloud-init v. 0.7.9 running 'init' at Mon, 22 
May 2017 09:14:57 +. Up 9.19 seconds.
[9.294365] cloud-init[390]: ci-info: +++Net device 
info
[9.299149] cloud-init[390]: ci-info: 
++---+---+---+---+---+
[9.303735] cloud-init[390]: ci-info: | Device |   Up  |  Address  |Mask 
  | Scope | Hw-Address|
[9.308328] cloud-init[390]: ci-info: 
++---+---+---+---+---+
[9.312879] cloud-init[390]: ci-info: | ens3:  | False | . | .   
  |   .   | 0e:3d:ec:0d:c9:56 |
[9.317451] cloud-init[390]: ci-info: |  lo:   |  True | 127.0.0.1 | 
255.0.0.0 |   .   | . |
[9.321656] cloud-init[390]: ci-info: |  lo:   |  True | . | .   
  |   d   | . |
[9.325631] cloud-init[390]: ci-info: 
++---+---+---+---+---+

I guess networking was looking for eth0 but systemd knew better (i.e. ens3). 
Can't provide any more details since I couldn't access the instance :-) .

This is easy to reproduce. Launch a t2.micro from the ami, specify a VPC 
(create if necessary), enable public IP and launch. Try to ssh into the 
instance.

This may be related to Bug#745587 
(https://lists.debian.org/debian-cloud/2017/04/msg00017.html).

In any case, not being able to launch into a VPC seems pretty severe, all 
things considered.