Re: Better arm64 support
On Mon, Apr 27, 2020 at 11:41:10PM +0100, Wookey wrote: > > > > generic is most flexible, genericcloud has physical hardware drivers > > > > disabled, > > > > and nocloud is as minimal as possible. The tools, configs, and docs > > > > are in > > > > this repo: https://salsa.debian.org/cloud-team/debian-cloud-images/ > > > > > > OK. Sounds like arm64 should be enabled for these too. I'll have a look. > > > > What do you mean by "enabled"? "generic" arm64 images are available today > > at e.g. > > https://cloud.debian.org/images/cloud/sid/daily/20200427-245/ and > > https://cloud.debian.org/images/cloud/buster/20200425-243/, both of > > which are two clicks from https://cloud.debian.org/images/cloud/ > > Yes 'generic' is built for arm64 but 'genericcloud' and 'nocloud' > aren't. By 'enabled' I meant build those other two images for arm64 as well. nocloud is built for arm64, see e.g. https://cloud.debian.org/images/cloud/sid/daily/20200427-245/debian-sid-nocloud-arm64-daily-20200427-245.tar.xz It's only genericcloud that isn't. Again, this is because the arm64 cloud kernel was only introduced with linux 5.5, so it wasn't possible until very recently. > -start--- > At Packet you can boot operating systems through our Custom iPXE > setup, described in some detail at: > https://www.packet.com/developers/docs/servers/operating-systems/custom-ipxe/ > > We should be able to boot any "server-ready" operating system that way, so > long as the image contains NIC drivers for the hardware (typically Mellanox > CX-4 > in our Arm-based servers, or Intel x710 in our x86 servers). > > To make a given operating system directly provisionable at Packet > we use "Packer" from Hashicorp, and there are some things we > bake into the image - no daemons, but there are a few helper > scripts for things like iSCSI attach for our block storage that get > added to the base image. > end-- It seems likely that the generic images will work there, though they will try to use things like cloud-init that won't function usefully. Cloud-init typically takes care of setup of user accounts and ssh keys for remote access in environments that use it. How does Packet set this up? Is the expectation that authorized_keys files are baked in to the image, or is there some other mechanism to populate them dynamically? > Ah cool, and I see that https://cloud.debian.org/images/cloud/ has > been updated. That's great, cheers. Yes, I just added a little more detail to it, once I was reminded of where to look to actually do so... noah
Re: Better arm64 support
On 2020-04-27 14:19 -0400, Noah Meyerhans wrote:
> On Mon, Apr 27, 2020 at 01:05:34PM +0100, Wookey wrote:
> > > There's new work in progress at: https://image-finder.debian.net/
> It should be linked from somewhere more prominent when it's actually
> usable. It is not currently.
OK. Noted.
> > > > The 'genericcloud' and 'nocloud' images are amd64 only. Maybe they
> > > > don't make sense for arm64 - what are they for?
> > >
> > > generic is most flexible, genericcloud has physical hardware drivers
> > > disabled,
> > > and nocloud is as minimal as possible. The tools, configs, and docs are
> > > in
> > > this repo: https://salsa.debian.org/cloud-team/debian-cloud-images/
> >
> > OK. Sounds like arm64 should be enabled for these too. I'll have a look.
>
> What do you mean by "enabled"? "generic" arm64 images are available today at
> e.g.
> https://cloud.debian.org/images/cloud/sid/daily/20200427-245/ and
> https://cloud.debian.org/images/cloud/buster/20200425-243/, both of
> which are two clicks from https://cloud.debian.org/images/cloud/
Yes 'generic' is built for arm64 but 'genericcloud' and 'nocloud'
aren't. By 'enabled' I meant build those other two images for arm64 as well.
> > > > I don't know if there are other services that it would make sense to
> > > > support, e.g packet provide arm64 online compute. https://www.packet.com
> > > > They already provide debian 8,9,10 options, apparently built as docker
> > > > images: https://github.com/packethost/packet-images
> > >
> > > I haven't heard that anyone on the team uses packet - but if someone
> > > wanted to
> > > extend support for their platform, I imagine we'd accept it.
> >
> > I've made contact with them to ask about what specific support is
> > needed. It seems like a generic image should work, but there are some
> > API support tools we could usefully add. I'll have a go with this and
> > make changes to /debian-cloud-images/ if appropriate.
>
> Is packet based on OpenStack?
I don't think so.
This is what they told me:
-start---
At Packet you can boot operating systems through our Custom iPXE
setup, described in some detail at:
https://www.packet.com/developers/docs/servers/operating-systems/custom-ipxe/
We should be able to boot any "server-ready" operating system that way, so
long as the image contains NIC drivers for the hardware (typically Mellanox CX-4
in our Arm-based servers, or Intel x710 in our x86 servers).
To make a given operating system directly provisionable at Packet
we use "Packer" from Hashicorp, and there are some things we
bake into the image - no daemons, but there are a few helper
scripts for things like iSCSI attach for our block storage that get
added to the base image.
end--
> > Right. And maybe generic cloud works.
>
> Note that the arm64 cloud kernel is available in bullseye and sid, but
> not buster. Arm64 buster images just use the regular ("generic") arm64
> kernel.
Ah. OK so the 'generic' and 'genericcloud' names refer to the kernel
used? Obviously we can't build an arm64 cloud-kernel version on buster
if the package doesn't exist.
> It's not necessarily the goal of the cloud kernel to support
> every cloud infrastructure in existence, but if there are missing
> drivers that prevent it from doing so, we can consider enabling them.
Understood.
Looks like so long as Mellanox CX-4 is enabled then that may be all
that's needed. That seems to be mlx4_en and mlx4_core.
Ah cool, and I see that https://cloud.debian.org/images/cloud/ has
been updated. That's great, cheers.
Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
signature.asc
Description: PGP signature
Re: Better arm64 support
On Mon, Apr 27, 2020 at 01:05:34PM +0100, Wookey wrote:
> > > Someone asked me for a VM image recently, and I discovered we now
> > > make some available at: https://cloud.debian.org/images/cloud/
> >
> > There's new work in progress at: https://image-finder.debian.net/ That page
> > includes some info on the different image types, but doesn't always have up
> > to
> > date data yet.
>
> OK. That's useful. Boggling long list of AWS images!
> Perhaps that should be linked from /debian-cloud-images/ so people could find
> it.
> (where does the content for that page live?)
It should be linked from somewhere more prominent when it's actually
usable. It is not currently. Information on that site is not current
and shouldn't be relied on to be useful today.
So many images are listed for AWS because it (currently) loads every
daily image of every version we build for, as well as every "release"
ami, in all regions. It then relies on client-side filtering to limit
the view to what's been asked for. That technique is already showing
performance issues, and will only get worse. This is evidence of its
WIP status.
> > > The 'genericcloud' and 'nocloud' images are amd64 only. Maybe they
> > > don't make sense for arm64 - what are they for?
> >
> > generic is most flexible, genericcloud has physical hardware drivers
> > disabled,
> > and nocloud is as minimal as possible. The tools, configs, and docs are in
> > this repo: https://salsa.debian.org/cloud-team/debian-cloud-images/
>
> OK. Sounds like arm64 should be enabled for these too. I'll have a look.
What do you mean by "enabled"? "generic" arm64 images are available today at
e.g.
https://cloud.debian.org/images/cloud/sid/daily/20200427-245/ and
https://cloud.debian.org/images/cloud/buster/20200425-243/, both of
which are two clicks from https://cloud.debian.org/images/cloud/
> > > I don't know if there are other services that it would make sense to
> > > support, e.g packet provide arm64 online compute. https://www.packet.com
> > > They already provide debian 8,9,10 options, apparently built as docker
> > > images: https://github.com/packethost/packet-images
> >
> > I haven't heard that anyone on the team uses packet - but if someone wanted
> > to
> > extend support for their platform, I imagine we'd accept it.
>
> I've made contact with them to ask about what specific support is
> needed. It seems like a generic image should work, but there are some
> API support tools we could usefully add. I'll have a go with this and
> make changes to /debian-cloud-images/ if appropriate.
Is packet based on OpenStack? If so, then yes, the generic images
should work there.
> > > A kosher debian image might be a good idea, but then maybe the generic
> > > ones work already? I just failed to find any info on whether anything
> > > special is needed in their images.
> >
> > The generic image should have a good shot at working, though it likely
> > contains
> > many drivers that won't be useful.
>
> Right. And maybe generic cloud works.
Note that the arm64 cloud kernel is available in bullseye and sid, but
not buster. Arm64 buster images just use the regular ("generic") arm64
kernel. It's not necessarily the goal of the cloud kernel to support
every cloud infrastructure in existence, but if there are missing
drivers that prevent it from doing so, we can consider enabling them.
noah
Re: Better arm64 support
On 2020-04-26 22:24 -0700, Ross Vandegrift wrote: [Cheers for cc:ing, I'm not subscribed here] > On Mon, Apr 20, 2020 at 02:52:40PM +0100, Wookey wrote: > > Someone asked me for a VM image recently, and I discovered we now > > make some available at: https://cloud.debian.org/images/cloud/ > > There's new work in progress at: https://image-finder.debian.net/ That page > includes some info on the different image types, but doesn't always have up to > date data yet. OK. That's useful. Boggling long list of AWS images! Perhaps that should be linked from /debian-cloud-images/ so people could find it. (where does the content for that page live?) > > The 'genericcloud' and 'nocloud' images are amd64 only. Maybe they > > don't make sense for arm64 - what are they for? > > generic is most flexible, genericcloud has physical hardware drivers disabled, > and nocloud is as minimal as possible. The tools, configs, and docs are in > this repo: https://salsa.debian.org/cloud-team/debian-cloud-images/ OK. Sounds like arm64 should be enabled for these too. I'll have a look. > Was your user able to get an image working? Well when the website said everything (except openstack) was amd64 only (https://cloud.debian.org/images/cloud/) (and it still does - how do we change that page?) I just went and made him a rootfs with debootstrap: sudo debootstrap --include=linux-image-arm64 testing arm64rootfs which got him going. > If not, that'd be somewhere to > start. I have minimal experience with arm virtualization, so I'm not really > familiar with what use-cases would be common. Neither am I to be honest, but essentilly the use cases are the same as x86. It looks (or should look) exactly the same to users. It's just that some virtualisation providers don't offer arm64 hardware yet. > > I don't know if there are other services that it would make sense to > > support, e.g packet provide arm64 online compute. https://www.packet.com > > They already provide debian 8,9,10 options, apparently built as docker > > images: https://github.com/packethost/packet-images > > I haven't heard that anyone on the team uses packet - but if someone wanted to > extend support for their platform, I imagine we'd accept it. I've made contact with them to ask about what specific support is needed. It seems like a generic image should work, but there are some API support tools we could usefully add. I'll have a go with this and make changes to /debian-cloud-images/ if appropriate. > > A kosher debian image might be a good idea, but then maybe the generic > > ones work already? I just failed to find any info on whether anything > > special is needed in their images. > > The generic image should have a good shot at working, though it likely > contains > many drivers that won't be useful. Right. And maybe generic cloud works. Wookey -- Principal hats: Linaro, Debian, Wookware, ARM http://wookware.org/ signature.asc Description: PGP signature
Re: Better arm64 support
Hello, On Mon, Apr 20, 2020 at 02:52:40PM +0100, Wookey wrote: > Someone asked me for a VM image recently, and I discovered we now > make some available at: https://cloud.debian.org/images/cloud/ There's new work in progress at: https://image-finder.debian.net/ That page includes some info on the different image types, but doesn't always have up to date data yet. > The 'genericcloud' and 'nocloud' images are amd64 only. Maybe they > don't make sense for arm64 - what are they for? generic is most flexible, genericcloud has physical hardware drivers disabled, and nocloud is as minimal as possible. The tools, configs, and docs are in this repo: https://salsa.debian.org/cloud-team/debian-cloud-images/ > So, I'd like to ask who's behind this fine service, and whether > anything more is needed to get arm parity with x86. Possibly there is > nothing to do except update the front page, but if there is I'm able > to spend time helping update things and I can get us some > hardware or resource to do builds on if that's what's needed, > (because Arm/linaro/Huawei/others would like to see x86_64/arm64 > parity for things like this and understand that kit/cycles need to be > supplied). Was your user able to get an image working? If not, that'd be somewhere to start. I have minimal experience with arm virtualization, so I'm not really familiar with what use-cases would be common. > I don't know if there are other services that it would make sense to > support, e.g packet provide arm64 online compute. https://www.packet.com > They already provide debian 8,9,10 options, apparently built as docker > images: https://github.com/packethost/packet-images I haven't heard that anyone on the team uses packet - but if someone wanted to extend support for their platform, I imagine we'd accept it. > A kosher debian image might be a good idea, but then maybe the generic > ones work already? I just failed to find any info on whether anything > special is needed in their images. The generic image should have a good shot at working, though it likely contains many drivers that won't be useful. Ross signature.asc Description: PGP signature
Re: Better arm64 support
On 2020-04-20 14:52:40 +0100 (+0100), Wookey wrote: [...] > The 'genericcloud' and 'nocloud' images are amd64 only. Maybe they > don't make sense for arm64 - what are they for? [...] Maybe you're looking for the "openstack" images (which in theory ought to work under a variety of hypervisor environments not just OpenStack managed ones): https://cloud.debian.org/images/openstack/current/ Those have arm64 variants as well as amd64. -- Jeremy Stanley signature.asc Description: PGP signature
