Re: Call for Testing: Stretch Cloud Images on AWS

[Marcin Kulisz]

On 2017-02-24 09:10:17, Noah Meyerhans wrote:
> On Fri, Feb 24, 2017 at 10:19:53AM +0800, gustavo panizzo wrote:

snip

> > and this is not a bug, but a diference in opinion
> > 
> > $ grep security /etc/apt/sources.list
> > deb http://security.debian.org/ jessie/updates main
> > 
> > instead you can use
> > deb http://cloudfront.debian.net/debian-security jessie/updates main
> > 
> > traffic has no cost :)
> 
> Also a good idea. I want to double check that this mirror gets pushes
> from security-master. Otherwise I agree that we should use it.

Noah I'd suggest using http://cdn-aws.deb.debian.org/ instead, this service is
having the same setup as cloudfront.d.n but is DSA maintained.
-- 

|_|0|_|  |
|_|_|0|  "Panta rei" |
|0|0|0|  kuLa    |

gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3


signature.asc
Description: PGP signature

Re: Call for Testing: Stretch Cloud Images on AWS

[Noah Meyerhans]

On Fri, Feb 24, 2017 at 10:19:53AM +0800, gustavo panizzo wrote:
> I found another issue on the image
> 
> $ cat /etc/mailname
> ip-10-0-0-64.us-west-2.compute.internal

Nice catch, thank you!

> and this is not a bug, but a diference in opinion
> 
> $ grep security /etc/apt/sources.list
> deb http://security.debian.org/ jessie/updates main
> 
> instead you can use
> deb http://cloudfront.debian.net/debian-security jessie/updates main
> 
> traffic has no cost :)

Also a good idea. I want to double check that this mirror gets pushes
from security-master. Otherwise I agree that we should use it.

> let me know if you prefer emails or bug reports

Bug reports are easier to keep track of, so I'd prefer that in the
future.

Thank you for the feedback!

noah



signature.asc
Description: PGP signature

Re: Call for Testing: Stretch Cloud Images on AWS

[gustavo panizzo]

Hello Noah

I found another issue on the image

$ cat /etc/mailname
ip-10-0-0-64.us-west-2.compute.internal



and this is not a bug, but a diference in opinion

$ grep security /etc/apt/sources.list
deb http://security.debian.org/ jessie/updates main

instead you can use
deb http://cloudfront.debian.net/debian-security jessie/updates main

traffic has no cost :)


let me know if you prefer emails or bug reports

thanks!

On Thu, Feb 02, 2017 at 04:29:11PM +0800, gustavo panizzo wrote:
> Hello Noah
> 
> I saw your blog post (which I've attached to this email), then the next
> time I needed an EC2 instance I tested the images on a non-IPv6 region
> (SG) and an IPv6 enabled VPC
> 
> overall the image looks fine, no extraneous things, sysctl is clean,
> etc. great job. :)
> 
> 
> however there is a thing I'd like to see changed
> /etc/network/interfaces 
> 
> I remember reading about #846583 on the list but I didn't have anything
> to say back then.
> 
> could you move the configuration for eth1 to eth8 to
> /etc/network/interfaces.d/? also can you _please_ move out of
> /usr/local the helper? 
> 
> i'd like to see an /e/n/interfaces  like this
> 
> # interfaces(5) file used by ifup(8) and ifdown(8)
> # Include files from /etc/network/interfaces.d:
> # other interfaces are independently configured in /etc/network/interfaces.d/
> source-directory /etc/network/interfaces.d
> 
> auto lo
> iface lo inet loopback
> 
> auto eth0
> iface eth0 inet dhcp
> allow-hotplug eth0
> iface eth0 inet6 manual
>   up /lib/inet6-ifup-helper
>   down /lib/inet6-ifup-helper
> 
> $ ls -l /etc/network/interfaces.d/
> eth1
> eth2
> 
> eth8
> 
> - cloud-init complains when net-tools is not installed (it appears to
> work anyway) bug #853926
> 
> - I'd like to see all locales installed (but I understand that is a topic
> for another discussion)
> 
> I know my complains are mostly esthetics, but is part of the user
> experience the first time he/she logins into an instance.
> 
> 
> thanks!
> 
> - Forwarded message from "Noah Meyerhans: Noah Meyerhans" 
>  -
> 
> From: "Noah Meyerhans: Noah Meyerhans" 
> Subject: Call for Testing: Stretch Cloud Images on AWS
> Date: Mon, 30 Jan 2017 15:01:52 -
> User-Agent: rss2email/3.9 (https://github.com/wking/rss2email)
> To: g...@zumbi.com.ar
> Delivered-To: g...@zumbi.com.ar
> X-Spam-Level: 
> X-Spam-Status: No, score=-0.656 tagged_above=-999 required=4 
> tests=[AWL=0.344, BAYES_00=-1.9, DKIM_ADSP_NXDOMAIN=0.9, 
> NO_DNS_FOR_FROM=0.001, NO_RELAYS=-0.001] autolearn=no autolearn_force=no
> MIME-Version: 1.0
> 
> Following up on [Steve McIntyre's writeup](https://lists.debian.org/debian-
> sprints/2016/11/msg00018.html) of the Debian Cloud Sprint that took place in
> Seattle this past November, I'm pleased to announce the availability of
> preliminary Debian stretch AMIs for Amazon EC2. Pre-generated images are
> available in all public AWS regions, or you can use [FAI](http://wiki.fai-
> project.org/wiki/Main_Page) with the [fai-cloud-
> images](https://anonscm.debian.org/cgit/cloud/fai-cloud-images.git/)
> configuration tree to generate your own images. The pre-generated AMIs were
> created on 25 January, shortly after Linux 4.9 entered stretch, and their
> details follow:
> 
> ami-6d017002  |  ap-south-1  
> ---|---  
> ami-cc5540a8  |  eu-west-2  
> ami-43401925  |  eu-west-1  
> ami-870edfe9  |  ap-northeast-2  
> ami-812266e6  |  ap-northeast-1  
> ami-932e4aff  |  sa-east-1  
> ami-34ce7350  |  ca-central-1  
> ami-9f6dd8fc  |  ap-southeast-1  
> ami-829295e1  |  ap-southeast-2  
> ami-42448a2d  |  eu-central-1  
> ami-98c9348e  |  us-east-1  
> ami-57361332  |  us-east-2  
> ami-03386563  |  us-west-1  
> ami-7a27991a  |  us-west-2  
>   
> As with the current jessie images, these use a default username of 'admin',
> with access controlled by the ssh key named in the ec2 `run-instances`
> invocation. They're intended to provide a reasonably complete Debian
> environment without too much bloat. IPv6 addressing should be supported in an
> [appropriately configured VPC
> environment](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-
> migrate-ipv6.html).
> 
> These images were build using Thomas Lange's [FAI](http://wiki.fai-
> project.org/wiki/Main_Page), which has been used for over 15 years for
> provisioning all sorts of server, workstation, and VM systems, but which only
> recently was adapted for use generating cloud disk images. It has proven to be
> well suited to this task though, and image creation is straightforward and
> flexible. I'll describe in a followup post the steps you can follow to create
> and customize your own AMIs based on our recipes. In the meantime, please do
> test these images! You can submit bug reports to the
> [cloud.debian.org](https://bugs.debian.org/cgi-
> bin/pkgreport.cgi?pkg=cloud.debian.org;dist=unstable) metapackage, and
> feedback is welcome via the 

Re: Call for Testing: Stretch Cloud Images on AWS

[Marcin Kulisz]

On 2017-02-03 14:01:35, Sam Hartman wrote:
> I will say that the user experience of the default locale not being
> generated is fairly horrible.
> perl, and a lot of other things spew endless warnings all the time.
> It's not that I need locales, it's that I need perl to be pacified:-)

Haaa agreed we need some locale to be default, so some noisy stuff stop been
so annoying :-)

-- 

|_|0|_|  |
|_|_|0|  "Panta rei" |
|0|0|0|  kuLa    |

gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3


signature.asc
Description: PGP signature

Re: Call for Testing: Stretch Cloud Images on AWS

[Vincent Bernat]

 ❦  2 février 2017 21:42 -0800, Noah Meyerhans  :

>> overall the image looks fine, no extraneous things, sysctl is clean,
>> etc. great job. :)
>
> Interesting that you bring up sysctl. I consider it a bug that we're
> currently running with an unmodified set of sysctl variables. Apparently
> you disagree. My reasoning is that the kernel defaults are intended to
> be very broadly applicable, but the cloud AMI is a more specific use
> case and it should be possible to provide a more appropriate set of
> defaults for various settings. We can tune our sysctl settings towards
> server optimizations because we know we're not running on a device like
> a laptop or mobile device.

There is no such things as an universal sysctl settings for servers. The
ones set for EC2 are quite reasonable but still debatable and different
from default settings.

For example, the change to ip_local_port_range may come to a surprise
for some users if they are using some strict local firewalls. It could
also prevent a daemon to bind to a "medium" port that was expected to be
free because outside of the default range.

Another example is the tuning on tcp_wmem/rmem. A server using a lot of
sockets will suddenly use more memory (12 MB per socket instead of 16
KB). The backlog change is similar. A user may expect its clients to
fail early when the server is unable to dequeue requests fast enough,
notably when the clients are load-balancing reverse proxy and latency is
important.

The kernel documents net.ipv4.tcp_tw_reuse as a dangerous setting to
change (and they don't want to change this wording). I know this is not
true, but some users may feel that if kernel developers say this setting
should stay to 0, why Debian does provide images with a different
setting?

At least, a comment should be added on top of the file stating those
changes are advices from Amazon for their platform.
-- 
Take care to branch the right way on equality.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Call for Testing: Stretch Cloud Images on AWS

[gustavo panizzo]

On Thu, Feb 02, 2017 at 09:42:04PM -0800, Noah Meyerhans wrote:
> On Thu, Feb 02, 2017 at 04:29:11PM +0800, gustavo panizzo wrote:
> > I saw your blog post (which I've attached to this email), then the next
> > time I needed an EC2 instance I tested the images on a non-IPv6 region
> > (SG) and an IPv6 enabled VPC
> > 
> > overall the image looks fine, no extraneous things, sysctl is clean,
> > etc. great job. :)
> 
> Interesting that you bring up sysctl. I consider it a bug that we're
> currently running with an unmodified set of sysctl variables. Apparently
> you disagree. My reasoning is that the kernel defaults are intended to
> be very broadly applicable, but the cloud AMI is a more specific use
> case and it should be possible to provide a more appropriate set of
> defaults for various settings. We can tune our sysctl settings towards
> server optimizations because we know we're not running on a device like
> a laptop or mobile device.

the right thing to do would be ship tuned [1] or similar by default

[1] https://fedorahosted.org/tuned/

As others mentioned before, I think we should avoid surprising the
users. the AMI should be as vanilla as possible.

> 
> > could you move the configuration for eth1 to eth8 to
> > /etc/network/interfaces.d/? also can you _please_ move out of
> > /usr/local the helper? 
> 
> I think moving most interface configs to interfaces.d is reasonable and
> will do that. I had considered it previously but did not, mostly out of
> laziness.
> 
> Where would you prefer the interfaces helper script live, if not
> /usr/local? Because it does not belong to a package, I don't think it
> belongs in a first-level /usr subdirectory. I suppose ideally it will
> get added to a package, but I'm not sure it's worth packaging on its
> own. Maybe it could be added to ifupdown?
ifupdown is the right place for it IMHO, if ifupdown maintainer does not
agree i'd place it in /lib or /usr but never on /usr/local because
/usr/local hierarchy is reserved for local administrator, the script is
an artifact of the OS so it should live among OS artifacts.

again, this may be bikesheeding, i really don't want to do that. so if
you don't agree just go with it.

> 
> > - cloud-init complains when net-tools is not installed (it appears to
> > work anyway) bug #853926
> 
> It's probably best to explicitly install net-tools, at least until
> cloud-init is updated.
> 
> > - I'd like to see all locales installed (but I understand that is a topic
> > for another discussion)
> 
> Thanks for the suggestion. One thing that other distros have done is
> provide a "minimal" AMI that contains the most basic set of tools needed
> to function (i.e. not much more than a bare debootstrap install +
> sshd and cloud-init and their dependencies), and a full-featured
> variant. If we were to do that, maybe it'd make sense to provide locales
> in the featureful variant. OTOH, it should be pretty straightforward for
> a user to configure desired locales via user-data provided to cloud-init
> at launch time, so this may not be necessary.
I think any cloud advanced user would be able to provision the locales 
at creation time using cloud-init, heck they may never provision locales 
as they may not even login into the instances in their lifetime.

I was worried about the not so cloudy user who may treat ec2 as an
standard server whom may not be an english speaker.

I agree that 2 sets of AMIs full and minimal (by default) would be
useful for the not-so-advanced-user.

> 
> > I know my complains are mostly esthetics, but is part of the user
> > experience the first time he/she logins into an instance.
> 
> Noted. Thank you for your feedback.
> 
> noah
> 



-- 
1AE0 322E B8F7 4717 BDEA BF1D 44BB 1BA7 9F6C 6333

keybase: https://keybase.io/gfa

Re: Call for Testing: Stretch Cloud Images on AWS

[Noah Meyerhans]

On Thu, Feb 02, 2017 at 04:29:11PM +0800, gustavo panizzo wrote:
> I saw your blog post (which I've attached to this email), then the next
> time I needed an EC2 instance I tested the images on a non-IPv6 region
> (SG) and an IPv6 enabled VPC
> 
> overall the image looks fine, no extraneous things, sysctl is clean,
> etc. great job. :)

Interesting that you bring up sysctl. I consider it a bug that we're
currently running with an unmodified set of sysctl variables. Apparently
you disagree. My reasoning is that the kernel defaults are intended to
be very broadly applicable, but the cloud AMI is a more specific use
case and it should be possible to provide a more appropriate set of
defaults for various settings. We can tune our sysctl settings towards
server optimizations because we know we're not running on a device like
a laptop or mobile device.

> could you move the configuration for eth1 to eth8 to
> /etc/network/interfaces.d/? also can you _please_ move out of
> /usr/local the helper? 

I think moving most interface configs to interfaces.d is reasonable and
will do that. I had considered it previously but did not, mostly out of
laziness.

Where would you prefer the interfaces helper script live, if not
/usr/local? Because it does not belong to a package, I don't think it
belongs in a first-level /usr subdirectory. I suppose ideally it will
get added to a package, but I'm not sure it's worth packaging on its
own. Maybe it could be added to ifupdown?

> - cloud-init complains when net-tools is not installed (it appears to
> work anyway) bug #853926

It's probably best to explicitly install net-tools, at least until
cloud-init is updated.

> - I'd like to see all locales installed (but I understand that is a topic
> for another discussion)

Thanks for the suggestion. One thing that other distros have done is
provide a "minimal" AMI that contains the most basic set of tools needed
to function (i.e. not much more than a bare debootstrap install +
sshd and cloud-init and their dependencies), and a full-featured
variant. If we were to do that, maybe it'd make sense to provide locales
in the featureful variant. OTOH, it should be pretty straightforward for
a user to configure desired locales via user-data provided to cloud-init
at launch time, so this may not be necessary.

> I know my complains are mostly esthetics, but is part of the user
> experience the first time he/she logins into an instance.

Noted. Thank you for your feedback.

noah



signature.asc
Description: PGP signature