Re: Release status of i386 for Bullseye and long term support for 3 years?

[Henrique de Moraes Holschuh]

On Sat, Dec 12, 2020, at 13:09, Adrian Bunk wrote:
> 3. Computers that do support MMX and SSE2, but do not support 64bit.

The "Centrino" Pentium-M that you can find on a reasonably lot of still-working 
ThinkPads (the T4x series and similar X/R series of the time), for example.  
Note that this is "family 6" Pentium-M processors, not "family 15" Pentium4-M.

I wouldn't mind the "i386" port baseline bumped up to i686-with-SSE2 (and MMX), 
with gcc configured accordingly: I recall some reports that telling gcc to use 
SSE2 really improved several workloads.  But I am fine keeping the status-quo 
of current i686 baseline as well: requiring SSE2 would kick out a non-trivial 
number of non-Intel processor models, I think we went over this the last time a 
"i386" baseline bump was considered.

-- 
  Henrique de Moraes Holschuh 

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Craig]

Hello.  32-bit Pentium 4 user reading this out of personal interest
throwing in a comment.

If the choice is to primarily support pae or non pae for 32-bit moving
forward, then I suggest non-pae for the reason that everyone can use it.
If you have more than 4GB of memory you probably have a 64-bit cpu anyway
and really should switch to a 64-bit kernel, and there is no performance
gain by using pae with less than 4GB.  I feel the only people who should
use a 32-bit kernel with bullseye should be those without 64-bit support at
all, and comments in this thread seem to agree.  The popcorn results show
most people using pae, but I suspect that is just because it is default, I
am too, but only have 512MB so certainly don't need it.

As to the kernel version and spectre/meltdown mitigation I think 4.19 in
buster already resolves that.   A possible problem with using a newer
kernel for old 32-bit machines is they may not support the drivers we need,
at least in my specific case, nvidia-304 proprietary.  From nvidia directly
it wont even build for kernel 4.15; with community patches I can build for
4.19 and is also in buster repos.   But it won't build on 5.4 for me (as-is
or with patches that work for 4.19).  So it makes me concerned if it will
be available to me in bullseye because whoever packages it would have to do
extra work which may or may not be trivial.  If upgrading to bullseye means
no more nvidia driver, it makes it less appealing for me.  Although I have
read 5.4 improves floppy disk speed, something 32-bit users are more likely
to still have, but I doubt use very often.

Thanks for continuing to support 32-bit in any form, excited to see what
happens.

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Stephan Seitz]

Am Sa, Dez 12, 2020 at 20:27:16 + schrieb Steve McIntyre:

It's still quite new, but we have a package in the archive for this now:
 https://tracker.debian.org/pkg/debian-crossgrader


Well, yes, but it is only in unstable.

I tried to install it but apt wanted to replace many packages. Using 
aptitude with --without-recommends was less scaring but still not 
something I want to do on a stable system.


Shade and sweet water!

Stephan

--
|If your life was a horse, you'd have to shoot it.|

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Theodore Y. Ts'o]

On Sat, Dec 12, 2020 at 08:27:16PM +, Steve McIntyre wrote:
> Stephan wrote:
> >Am Sa, Dez 12, 2020 at 18:09:02 +0200 schrieb Adrian Bunk:
> >>4. People who wrongly installed i386 on amd64-capable hardware.
> >
> >Well, some releases ago befor multi-arch I used to install i386 even on 
> >am64-capable hardware if ram was quite low (=< 8GB) and if the chance 
> >wasn’t that low that you needed to install ia32-codec to watch ancient 
> >video formats.
> >
> >I wouldn’t do it anymore but at least one system is still in use, and 
> >there isn’t a real supported way to upgrade from i386 to amd64.
> 
> It's still quite new, but we have a package in the archive for this now:
> 
>   https://tracker.debian.org/pkg/debian-crossgrader
> 
> It's targeted at systems exactly like yours, tbh. Maybe give it a try?

I took at look at debian-crossgrader, and while it looks like what I
need --- I have a production server running on an Linode VM that was
originally installed with Debian 6.0 ("squeeze") and upgraded over the
years to Debian 10.0.  It's probably no surprise that it's still
running with i386 binaries, although I am using the Linode-supplied
64-bit kernel.

However, debian-crossgrader is only available on Debian testing, which
means it's not available on Buster.  (And while we could try to
backport it, it's unclear to me whether or not it would be reliable on
Buster packages.)  Furthermore, I'm hesitant to update a production
server to Debian testing, since it doesn't get security updates.  And
so if Debian 11.0 drops i386, it's not clear that i386 production
systems will have a clean upgrade path to Debian 11 and upgrading to
amd64 simultaneously.

So in the ideal world we would keep i386 support for at least one more
release, or the upgrade procedures would have a documented, tested
patch to take i386 Buster systems to amd64 Bullseye.

I'm probably going to be able to make it work, and I might just bite
the bullet and figure out a way to update to Debian testing with i386
during the freeze, and then using Crossgrader, and then updating to
Debian stable.  Or I might just try the manual process at
https://wiki.debian.org/CrossGrading with lots of backups, and
testing, and crossed fingers while on Buster, so the upgrade path might be:

Buster i386 --> Manual Crossgrade --> Buster amd_64 --> Bullseye amd_64

But there may be less experienced sysadmins that might find this to be
a rather scary process.

Cheers,

- Ted

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Michael Stone]

On Wed, Dec 16, 2020 at 02:47:22PM -0500, Devops PK Carlisle LLC wrote:

I understand your point about 32 bit being updated forever,  and perhaps
it does not need to be. Perhaps the happy medium would be to freeze it
at some point, but leave it available as-is so that legacy software with
32 bit dependencies can still be installed and run. In other words, no
longer developing for 32 bit does not mean that it cannot be found.
Perhaps a different repository, with disclaimer(?) so that users can
enable it if desired?


http://archive.debian.org/

:)

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Devops PK Carlisle LLC]

I understand your point about 32 bit being updated forever,  and perhaps
it does not need to be. Perhaps the happy medium would be to freeze it
at some point, but leave it available as-is so that legacy software with
32 bit dependencies can still be installed and run. In other words, no
longer developing for 32 bit does not mean that it cannot be found.
Perhaps a different repository, with disclaimer(?) so that users can
enable it if desired?


On 12/15/20 8:47 AM, Michael Stone wrote:
> On Sun, Dec 13, 2020 at 11:40:29AM -0500, Devops PK Carlisle LLC wrote:
>> Being philosophically opposed to throwing a good machine into a
>> landfill, I tend to hang on to equipment for a long time. My
>> play/experimentation and last-ditch backup box is a 10 year old laptop.
> 
> I hear that, but at least around here it's literally possible to grab
> machines that are less than 10 years old that are on their way to the
> landfill. So scratch your itch by saving a machine less than 10 years
> old, then throw the really old one away instead. I'm unconvinced that
> running a stable of unneeded old machines is either great from a waste
> standpoint or something that should dictate the direction of the
> project. Debian isn't devoted specifically to supporting functionally
> obsolete hardware indefinitely, and when obsolete hardware makes it hard
> to move forward we need to just drop it. There are other projects which
> do strive to support old hardware indefinitely, and I highly recommend
> looking at one of those if hardware you want to use is no longer
> supported by debian. I personally run netbsd on some of my nostalgia
> hardware, but there are other options that may work better for you
> depending on what you're trying to do.
> 

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Michael Stone]

On Sun, Dec 13, 2020 at 11:40:29AM -0500, Devops PK Carlisle LLC wrote:

Being philosophically opposed to throwing a good machine into a
landfill, I tend to hang on to equipment for a long time. My
play/experimentation and last-ditch backup box is a 10 year old laptop.


I hear that, but at least around here it's literally possible to grab 
machines that are less than 10 years old that are on their way to the 
landfill. So scratch your itch by saving a machine less than 10 years 
old, then throw the really old one away instead. I'm unconvinced that 
running a stable of unneeded old machines is either great from a waste 
standpoint or something that should dictate the direction of the 
project. Debian isn't devoted specifically to supporting functionally 
obsolete hardware indefinitely, and when obsolete hardware makes it hard 
to move forward we need to just drop it. There are other projects which 
do strive to support old hardware indefinitely, and I highly recommend 
looking at one of those if hardware you want to use is no longer 
supported by debian. I personally run netbsd on some of my nostalgia 
hardware, but there are other options that may work better for you 
depending on what you're trying to do.

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Christian Kastner]

On 15.12.20 01:55, Russ Allbery wrote:
> Increasingly most of the people who work on Debian don't have i386
> hardware lying around, particularly i386 hardware that requires an i386
> kernel or that exercises the full range of older boot processes.  If you
> do, testing and reporting good bugs would probably be helpful.

FWIW, most people probably have amd64 hardware around, and can therefore
use KVM-accelerated i386 emulation using QEMU. That emulation can be
configured with a fine grain, down to CPU models/features.

And until at least a few years ago, that emulation was quite realistic.
For my bachelor's thesis, I looked into portability of binaries, and I
used autopkgtest + QEMU to hunt for bugs where the buildd environment
affected the build (for example, by picking up CPU flags of the buildd
machine).

I found #781998 like that (i386 binary assuming SSE is present), and
confirmed it on real hardware.

So, technically, I think i386 is quite easy to test. To me, even easier
than arm64, where I need to get useful hardware first.

Using QEMU, it's trivial to build packages for i386 on amd64 (using
sbuild, or the qemu-sbuild-tools wrapper I'm dabbling on, which will be
absorbed by sbuild soon), and autopkgtest using QEMU has been trivial
since forever.

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Andrey Rahmatullin]

On Mon, Dec 14, 2020 at 05:13:54PM -0500, Calum McConnell wrote:
> Since (AFAIK) there is a substantial speed penalty to installing a non-pae
> kernel on a -pae processor,
The penalty is not using more than 4 Gb of RAM (the only related speed
penalty I know about is using PAE vs not using it).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Paul Wise]

On Mon, Dec 14, 2020 at 11:36 PM Adrian Bunk wrote:

> A bigger worry for i386 would be the availability of microcode updates

This is also a big problem for amd64, since only the newest
generations of Intel processors get BIOS/UEFI and or microcode
updates, so lots of amd64 users (including myself) do not have
microcode updates.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Russ Allbery]

Calum McConnell  writes:

> A very fair point, and quite equitably put.  If I was remotely
> comfortable tweaking kernels, or used a 32 bit machine regularly, I
> would be more comfortable volunteering.  As it is, I have only really
> learned to maintain packages in the past few months, and I feel nowhere
> near confident enough about my future to make a three-year commitment.

It sounds like from Adrien's message that one of the missing pieces is
more people testing d-i regularly on i386 hardware to confirm that it
works properly.  That's something that doesn't require much kernel
tweaking, and may be a quick way to help.

Increasingly most of the people who work on Debian don't have i386
hardware lying around, particularly i386 hardware that requires an i386
kernel or that exercises the full range of older boot processes.  If you
do, testing and reporting good bugs would probably be helpful.

It sounds like there are a fair number of people want the i386
architecture to survive, which is great.  That probably means the
resources are there.  One of the things a porter does is coordinate the
effort, so people who are willing to invest time into that coordination
can help even if they don't think they can fix tricky kernel bugs.

> But I would like to say that, while it is a significant workload, it
> isn't one that isn't being done.  There is still only one porter, it's
> true, but that's also currently the case for ppc64el, and s390x has no
> confirmed porters.

My intuition is also that i386, although becoming less popular, was
starting from such a huge install base that the resources are probably out
there somewhere.

> Further, unless "sudden death of most porters" can be added to the list
> of bad events of 2020, I feel confident in saying that there are still
> probably some more people who simply haven't gotten around to confirming
> that they can be a porter.

I agree.  Most of my point is just that they should do that.  :)  Now's
the time.

> While I agree that i386 kernel support should be phased out, and might
> even need to be dropped altogether, I strongly disagree with the
> original premise of this thread, that all i386 support should be dropped
> for Bullseye.

I may be able to reassure you a bit there.  Someone pointing out that we
don't have enough confirmed resources for a port happens semi-regularly,
and the usual outcome is that enough resources step forward.  We're not
very eager to drop things that people want to support.  The point is to
prod people into stepping forward and volunteering for the things they
care about.

What's perhaps more significant is that i386 is now getting to the point
where it requires such prodding, instead of being an assumed default
architecture.  That means that the folks who care about it should probably
start thinking about building more organization and structure around the
work, recruiting people, building a task list, and so forth, instead of
just assuming "oh, everything will work on i386, it always has."
Volunteering to do that sort of coordination is helpful even if you aren't
debugging FTBFS problems.

-- 
Russ Allbery (r...@debian.org)  

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Adrian Bunk]

On Mon, Dec 14, 2020 at 02:54:37PM -0800, Russ Allbery wrote:
> 
> The quantity of hardware is useful data, but I think this is also a place
> where it's important to stress the specific problem that Debian has,
> namely that we need people to do the work.
>...

The list of Debian release architecture that did have a non-zero number 
of porters committed by the bullseye porter roll call deadline is 
exactly the list of release architectures not supported by Ubuntu.

Among the architectures also supported by Ubuntu,
none had a porter committed for bullseye.

cu
Adrian

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Calum McConnell]

On Mon, 2020-12-14 at 14:54 -0800, Russ Allbery wrote:
> Calum McConnell  writes:
> > The point I'm making is that i386 processors are still incredibly
> > common, and we shouldn't abandon their users.
> 
> Not abandoning users is a powerful motivating force, but it still has to
> succeed in motivating people.  Debian can't make a decision on the basis
> of not abandoning users.  We have to make a decision on the basis that
> someone is volunteering to do the work.  Perhaps they're volunteering to
> do that work so that we're not abandoning users, and that's great, but
> that additional step is important.
> 
> I think it's therefore useful in this sort of thread to be very clear
> whether your conclusion is "and therefore I am volunteering to do the
> work to keep i386 alive" or whether your conclusion is "and therefore I
> am asking other people to volunteer to keep i386 alive," and be aware
> that the latter may not be successful because volunteer time is a
> limited resource and there are many worthy things that we could all be
> working on to make the lives of users better.

A very fair point, and quite equitably put.  If I was remotely comfortable
tweaking kernels, or used a 32 bit machine regularly, I would be more
comfortable volunteering.  As it is, I have only really learned to
maintain packages in the past few months, and I feel nowhere near
confident enough about my future to make a three-year commitment.

But I would like to say that, while it is a significant workload, it isn't
one that isn't being done.  There is still only one porter, it's true, but
that's also currently the case for ppc64el, and s390x has no confirmed
porters.  In fact, no architecture has any more than 2 porters. Plus, in
other areas i386 is in a better position than most: it has more archive
coverage than any other (non-amd64) port, and still has good upstream
support. 

Further, unless "sudden death of most porters" can be added to the list of
bad events of 2020, I feel confident in saying that there are still
probably some more people who simply haven't gotten around to confirming
that they can be a porter.  Every port but arm64 has less than half the
porters that it had for Buster, and many have a third or a fourth the
people. So while it's true that I am asking others to give up their time,
without backing it up with my own commitment as Johannes has, I feel that
it is a request that will be met.  

> The reason why I'm focusing on the kernel is because the upstream kernel
> developers have been signaling rather strongly for a while that i386 is
> a semi-deprecated architecture that you should avoid running if you can,
> and the amount of resources and attention that it is getting are
> steadily dropping.  Maybe the resources and attention it gets are still
> something we consider "good enough" (although we're already at the point
> where if you care about kernel security, you should put serious effort
> into getting onto the amd64 kernel even if you keep an i386 userspace),
> but at some point it seems likely they will no longer be.  That means it
> may be time to push our users a bit harder to switch to the amd64 kernel
> if they can.

I agree, but I don't think we are yet at that point where dropping support
should be an issue.  If the debate was whether i386 should be maintained
as an LTS architecture [1], I would be more willing to agree.  Perhaps for
Bookworm, a reasonable compromise would be to drop security support for
i386 kernels.  But that discussion is at least a year down the road. While
the kernel upstream may be discouraging i386 users, it is still (mostly)
supported by them for the time being: as long as that remains the case, I
don't think we should drop the i386 kernel.

While I agree that i386 kernel support should be phased out, and might
even need to be dropped altogether, I strongly disagree with the original
premise of this thread, that all i386 support should be dropped for
Bullseye.  There is still a massive library of software that is only
available as 32-bit: indeed, in my experience it has only been in the past
few years that 64-bit has been the default.  While keeping old binaries
running isn't the responsibility of Debian, I do think that, after 20
years of recommending i386 as the most compatible target for code, we
should try to support them at least a little longer.

[1]: I mean a true LTS, not just the three years mentioned in the original
subject



signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Adrian Bunk]

In practice, whether or not i386 will be dropped as release archticture 
in bullseye will likely be decided by whether I will stay the only 
committed porter, or whether other people will ASAP send (belatedly)
replies to the proter roll call [1].

This discussion started due to lack of people testing debian-installer 
on older i386 hardware.[2] "test d-i regularly" and related items from 
the porter roll call would be suitable for people who are looking for
a way to contribute to Debian.

cu
Adrian

[1] https://lists.debian.org/debian-devel/2020/11/msg00025.html
[2] https://lists.debian.org/debian-cd/2020/12/msg0.html

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Russ Allbery (2020-12-14 23:54:37)
> > The point I'm making is that i386 processors are still incredibly common,
> > and we shouldn't abandon their users.
> 
> Not abandoning users is a powerful motivating force, but it still has to
> succeed in motivating people.  Debian can't make a decision on the basis
> of not abandoning users.  We have to make a decision on the basis that
> someone is volunteering to do the work.  Perhaps they're volunteering to
> do that work so that we're not abandoning users, and that's great, but
> that additional step is important.
> 
> I think it's therefore useful in this sort of thread to be very clear
> whether your conclusion is "and therefore I am volunteering to do the work
> to keep i386 alive" or whether your conclusion is "and therefore I am
> asking other people to volunteer to keep i386 alive," and be aware that
> the latter may not be successful because volunteer time is a limited
> resource and there are many worthy things that we could all be working on
> to make the lives of users better.
> 
> If we can confirm that the volunteer resources are there, we can ask what
> they need from the rest of the project to be successful.

I cannot donate my time (I'm also lacking the skill) but I'm willing to put my
money somewhere if it means to keep i386 alive for longer. I privately own
perfectly working old hardware that I would hate to send to the landfill just
because software support is ending.  Should i386 be discontinued I should
probably only keep using the hardware in a separate network disconnected from
the internet but that would make the hardware much less useful.

If somebody could direct me to organizations or people who just need financial
support to keep i386 alive, that would be great. In the light of a planet with
limited resources, I think it's worth my money to help keeping old hardware
around and avoid the waste of resources and energy that manufacturing new
equipment incurs.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Adrian Bunk]

On Mon, Dec 14, 2020 at 01:22:11PM +0100, Ben Hutchings wrote:
> On Sun, 2020-12-13 at 01:53 -0800, Steve Langasek wrote:
> [...]
> > While the ongoing
> > costs of maintaining a full port were a consideration, of equal concern was
> > the fact that we believed we would not be able to provide security support
> > for the architecture as a whole at par with other architectures, due to,
> > among other things, lack of adequate support from the upstream
> > kernel/toolchain community.  I'm not sure if i386 has caught up and now has
> > adequate mitigation for Spectre etc, but it definitely wasn't available on
> > an equivalent timeline as amd64.
> 
> I agree that kernel security support for i386 is seriously lacking.
> 
> The Spectre mitigations were actually available for both x86
> architectures at the same time, but the initial Meltdown mitigation was
> amd64-specific and was not extended to i386 until Linux 4.19.  The
> implementation used in stable kernel branches (KAISER) was sufficiently
> different from that used upstream, that i386 support has not been added
> to it.

If using Spectre/Meltdown as metric, how does kernel security support 
for architectures like arm64 or ppc64el compare to kernel security 
support for i386?

When it comes to security support, i386 often has the benefit that code 
is shared with amd64 so fixes are available early (like for Spectre).

I am not saying that there was no problem on i386, but if this was meant 
to register a security concern for release architectures we have to look 
at all architectures.

> As a result, stretch:i386 is still vulnerable when running the default
> (4.9-based) kernel.

A bigger worry for i386 would be the availability of microcode updates 
for Spectre, but this has little practical impact as long as noone cares 
enough about Spectre to start a GR that would allow us to not leave our 
amd64 users vulnerable by default even in bullseye.

> Ben.

cu
Adrian

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Russ Allbery]

Calum McConnell  writes:

> As I showed in my (slightly over dramatic, very over-long) email this
> morning, there are more people with i386 kernels than there are total
> users of every other release architecture.  Even if you only look at
> non-pae kernels, its still about double the total installs for any two
> other release architectures.

The quantity of hardware is useful data, but I think this is also a place
where it's important to stress the specific problem that Debian has,
namely that we need people to do the work.  That's both packaging work
inside Debian and enough upstream work that the software we're packaging
remains supportable on i386.

> The point I'm making is that i386 processors are still incredibly
> common, and we shouldn't abandon their users.

Not abandoning users is a powerful motivating force, but it still has to
succeed in motivating people.  Debian can't make a decision on the basis
of not abandoning users.  We have to make a decision on the basis that
someone is volunteering to do the work.  Perhaps they're volunteering to
do that work so that we're not abandoning users, and that's great, but
that additional step is important.

I think it's therefore useful in this sort of thread to be very clear
whether your conclusion is "and therefore I am volunteering to do the work
to keep i386 alive" or whether your conclusion is "and therefore I am
asking other people to volunteer to keep i386 alive," and be aware that
the latter may not be successful because volunteer time is a limited
resource and there are many worthy things that we could all be working on
to make the lives of users better.

If we can confirm that the volunteer resources are there, we can ask what
they need from the rest of the project to be successful.

The reason why I'm focusing on the kernel is because the upstream kernel
developers have been signaling rather strongly for a while that i386 is a
semi-deprecated architecture that you should avoid running if you can, and
the amount of resources and attention that it is getting are steadily
dropping.  Maybe the resources and attention it gets are still something
we consider "good enough" (although we're already at the point where if
you care about kernel security, you should put serious effort into getting
onto the amd64 kernel even if you keep an i386 userspace), but at some
point it seems likely they will no longer be.  That means it may be time
to push our users a bit harder to switch to the amd64 kernel if they can.

-- 
Russ Allbery (r...@debian.org)  

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Calum McConnell]

On Mon, 2020-12-14 at 10:02 -0800, Russ Allbery wrote:

> One possible intermediate option shy of dropping the i386 architecture
> would be to drop the i386 kernel and instead help all i386 installs
> switch
> to the amd64 kernel while still running i386 binaries.  (That said, this
> will obviously not work on actual i386 hardware, and I don't know if it
> has other issues that I personally happened not to notice.)

As I showed in my (slightly over dramatic, very over-long) email this
morning, there are more people with i386 kernels than there are total
users of every other release architecture.  Even if you only look at non-
pae kernels, its still about double the total installs for any two other
release architectures.

Since (AFAIK) there is a substantial speed penalty to installing a non-pae
kernel on a -pae processor, I don't see there being a lot of users running
that.  Further, because installations of old distributions still report
popcon information, I can also tell you that the i486, which is even older
than the i586 that was dropped in Stretch, and was deprecated in 2014,
still has more installs than all but 3 of the release architectures:
armel, armhf, amd64, and i386.  

Keep in mind, too, that these are wildly unfair comparisons: I'm comparing
"people with x specific kernel package" to "people with some package from
that architecture".  The point I'm making is that i386 processors are
still incredibly common, and we shouldn't abandon their users.




signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Russ Allbery]

Ben Hutchings  writes:

> I agree that kernel security support for i386 is seriously lacking.

> The Spectre mitigations were actually available for both x86
> architectures at the same time, but the initial Meltdown mitigation was
> amd64-specific and was not extended to i386 until Linux 4.19.  The
> implementation used in stable kernel branches (KAISER) was sufficiently
> different from that used upstream, that i386 support has not been added
> to it.

> As a result, stretch:i386 is still vulnerable when running the default
> (4.9-based) kernel.

It may be worth separating the kernel from the rest of the distribution.
Switching an existing i386 deploy to use the amd64 kernel is fairly easy.
I did that on my legacy i386 installations quite some time ago, and thus
am running a kernel with proper upstream security support.  It's far
easier and less intimidating than crossgrading the rest of the system to
amd64.

One possible intermediate option shy of dropping the i386 architecture
would be to drop the i386 kernel and instead help all i386 installs switch
to the amd64 kernel while still running i386 binaries.  (That said, this
will obviously not work on actual i386 hardware, and I don't know if it
has other issues that I personally happened not to notice.)

-- 
Russ Allbery (r...@debian.org)  

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Ben Hutchings]

On Sun, 2020-12-13 at 01:53 -0800, Steve Langasek wrote:
[...]
> While the ongoing
> costs of maintaining a full port were a consideration, of equal concern was
> the fact that we believed we would not be able to provide security support
> for the architecture as a whole at par with other architectures, due to,
> among other things, lack of adequate support from the upstream
> kernel/toolchain community.  I'm not sure if i386 has caught up and now has
> adequate mitigation for Spectre etc, but it definitely wasn't available on
> an equivalent timeline as amd64.

I agree that kernel security support for i386 is seriously lacking.

The Spectre mitigations were actually available for both x86
architectures at the same time, but the initial Meltdown mitigation was
amd64-specific and was not extended to i386 until Linux 4.19.  The
implementation used in stable kernel branches (KAISER) was sufficiently
different from that used upstream, that i386 support has not been added
to it.

As a result, stretch:i386 is still vulnerable when running the default
(4.9-based) kernel.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.


signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Devops PK Carlisle LLC]

Being philosophically opposed to throwing a good machine into a
landfill, I tend to hang on to equipment for a long time. My
play/experimentation and last-ditch backup box is a 10 year old laptop.

During COVID I spent a little time updating and upgrading it and came up
with this: https://go.carlisle.pk/5boot

Architecture:x86_64
CPU op-mode(s):  32-bit, 64-bit
Byte Order:  Little Endian
Address sizes:   40 bits physical, 48 bits virtual
CPU(s):  1
On-line CPU(s) list: 0
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):   1
NUMA node(s):1
Vendor ID:   AuthenticAMD
CPU family:  15
Model:   124
Model name:  AMD Athlon(tm) Processor TF-36
Stepping:2
CPU MHz: 1600.000
CPU max MHz: 2000.
CPU min MHz: 800.
BogoMIPS:3192.06
Virtualization:  AMD-V
L1d cache:   64K
L1i cache:   64K
L2 cache:256K
NUMA node0 CPU(s):   0
Flags:   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext
fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl cpuid extd_apicid pni
cx16 lahf_lm svm extapic cr8_legacy 3dnowprefetch vmmcall lbrv

The hardware is 64 bit, but as I still have some truecrypt volumes in
use, and the truecrypt installer is 32 bit, I run elements of each (this
is on both Debian and an Ubuntu tower).

I can run everything from G-Suite to Netflix -- maybe what will
eventually kill it is requirements of cloud services.

I would also argue that the ability to support equipment even once
Microsoft decides to no longer support it has long been one of the
arguments given out in favor of Linux over Windows.



On 12/13/20 2:14 AM, Calum McConnell wrote:
> Hi all,
> As someone who runs amd64/i386 multiarch, this statement from Adrian:
> 
> 
>> i386 hardware is so numerous and widely spread, that every tiny fraction
>> of i386 users might be more users than half of our release architectures
>> combined. It is not even clear whether this is just an exaggeration or 
>> might be literally true:
> 
> intrested me.  I wondered just how many there were.  Popcon lists 17281
> people with i386 installations, but I bet that includes those who (like
> me) installed multiarch.  So I grep'ed through the popcon output a bit,
> looking for kernel packages.  I figure that, if you have an i386 kernel
> pacakge, you don't belong in the first group of people.
> 
> Obviously you all can easily replicate this, and this only applies to
> users with popularity-contest installed, but here are my results:
> 
> For a baseline, there are 181,863 amd64 users who are regularally sending
> popcon reports.  Of those, 171,916 have the linux-image-amd64 package
> installed.  I assume the remaining 5.4 percent are selecting what kernel
> package they are running manually, or perhaps are in a VM.
> 
> The 13th most popular linux-image package is linux-image-686-pae, at
> 12,736 installs.  It places ahead of every single 5.x kernel, indicating
> that there are more people running i386 (with some extensions) than there
> are running Testing or Unstable.  
> 
> Continuing down the list, the standard linux-image-686 package (no PAE)
> has 877 popcon installs.  None of the other release archetecures have
> appeared yet: which isn't supprising, since every other popcon archetecure
> has a combined total of 1636 installs, the largest being armhf at 636
> installs.  I assume these people are the ones who would lose support:
> while some of them may have PAE capable computers, I don't think it's a
> significant fraction.
> 
> Clearly, I have already proved Adrian's point: I can say, with certainty,
> that there are an order of magnitude more people with i386 kernels (and
> thus presumably i386 hardware) than there are for every other non-amd64
> release archetecture combined.  Further, there are more people with old
> i386 hardware than there are for any other arch.  My point is that we
> would lose less people if we dropped all ARM support than if we dropped
> the oldest supported i386 kernel[1].
> 
> But lets keep going!  See, we haven't seen any arm kernal images yet, so
> who knows if they even exist?  Remember, the ARM archectures are the
> biggest ones after i386.
> 
> Next up, we hit linux-image-586, with 403 installs.  That means there are
> 403 people who were unable to upgrade to stretch, but are still running
> Debian and popcon.  That's presumably the lower limit for the number
> Adrian referenced as people who were upset with the increase in baseline,
> since again, all of those 403 people have used their 586 machine in the
> past month.
> 
> Continuing down, we see linux-image-486, 310 installs.  That's still more
> installations than arm64's total popcon.  It's also been unsupported since
> 2014, but hey.   
> 
> Then we hit linux-image-marvell, which (as I understand it) is one of the
> arm versions.  

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Steve Langasek]

On Sat, Dec 12, 2020 at 06:09:02PM +0200, Adrian Bunk wrote:
> > Ubuntu have chosen to support the first use-case, and only the first
> > use-case. They longer ship a complete, bootable i386 operating system;
> > instead, they have an i386 second-class-citizen architecture that
> > is sufficient to provide graphics drivers and other shared libraries
> > for legacy 32-bit proprietary binaries.
> >...

> Ubuntu has a business-minded focus, which is fair enough.
> But Debian should not blindly follow whatever Canonical
> does with Ubuntu for business reasons.[3]

> It does make sense for Debian to differenciate by providing support for 
> communities whose hardware is not or no longer supported by Ubuntu.

It's obviously entirely appropriate for Debian to make its own decision here
regarding what they want to support, but FTR the dropping of i386 was
largely not a "business" focused decision for Ubuntu.  While the ongoing
costs of maintaining a full port were a consideration, of equal concern was
the fact that we believed we would not be able to provide security support
for the architecture as a whole at par with other architectures, due to,
among other things, lack of adequate support from the upstream
kernel/toolchain community.  I'm not sure if i386 has caught up and now has
adequate mitigation for Spectre etc, but it definitely wasn't available on
an equivalent timeline as amd64.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Calum McConnell]

Hi all,
As someone who runs amd64/i386 multiarch, this statement from Adrian:


> i386 hardware is so numerous and widely spread, that every tiny fraction
> of i386 users might be more users than half of our release architectures
> combined. It is not even clear whether this is just an exaggeration or 
> might be literally true:

intrested me.  I wondered just how many there were.  Popcon lists 17281
people with i386 installations, but I bet that includes those who (like
me) installed multiarch.  So I grep'ed through the popcon output a bit,
looking for kernel packages.  I figure that, if you have an i386 kernel
pacakge, you don't belong in the first group of people.

Obviously you all can easily replicate this, and this only applies to
users with popularity-contest installed, but here are my results:

For a baseline, there are 181,863 amd64 users who are regularally sending
popcon reports.  Of those, 171,916 have the linux-image-amd64 package
installed.  I assume the remaining 5.4 percent are selecting what kernel
package they are running manually, or perhaps are in a VM.

The 13th most popular linux-image package is linux-image-686-pae, at
12,736 installs.  It places ahead of every single 5.x kernel, indicating
that there are more people running i386 (with some extensions) than there
are running Testing or Unstable.  

Continuing down the list, the standard linux-image-686 package (no PAE)
has 877 popcon installs.  None of the other release archetecures have
appeared yet: which isn't supprising, since every other popcon archetecure
has a combined total of 1636 installs, the largest being armhf at 636
installs.  I assume these people are the ones who would lose support:
while some of them may have PAE capable computers, I don't think it's a
significant fraction.

Clearly, I have already proved Adrian's point: I can say, with certainty,
that there are an order of magnitude more people with i386 kernels (and
thus presumably i386 hardware) than there are for every other non-amd64
release archetecture combined.  Further, there are more people with old
i386 hardware than there are for any other arch.  My point is that we
would lose less people if we dropped all ARM support than if we dropped
the oldest supported i386 kernel[1].

But lets keep going!  See, we haven't seen any arm kernal images yet, so
who knows if they even exist?  Remember, the ARM archectures are the
biggest ones after i386.

Next up, we hit linux-image-586, with 403 installs.  That means there are
403 people who were unable to upgrade to stretch, but are still running
Debian and popcon.  That's presumably the lower limit for the number
Adrian referenced as people who were upset with the increase in baseline,
since again, all of those 403 people have used their 586 machine in the
past month.

Continuing down, we see linux-image-486, 310 installs.  That's still more
installations than arm64's total popcon.  It's also been unsupported since
2014, but hey.   

Then we hit linux-image-marvell, which (as I understand it) is one of the
arm versions.  At 225 installs, its not terribly impressive.  Its also the
first non-amd64/i386 kernel that I hit on this list, and where I stop. 
This is supported as a first-class Debian citizen: and yet, the now
dropped 486 still has more installations.

Of course, the pace of technology marches on, and the 586 is an ancient
chip.  We were right to end support for it: it's not like any modern
software would run well on such a processor.  But there is still a large
section of the debian userbase using the older 686 versions.  Adrian is
right to say that ending support for them isn't right.

Wall of text meticulously analyzing the output of two commands aside, this
was a bit fun to make!  Now I'm off to bed in my bed: thanks for reading! 

Calum M

[1]: Okay, that's not strictly true: the total number of people reporting
packages from each of the arm architectures is 1256.  However, that
involves three seperate sub-archetecures, and I am willing to bet there
are a fair number of multi-arch arm users.  But for strict correctness,
pretend I said "all armhf and arm64 support", since those two total to
only 10 more than the subset of i386 in question.


signature.asc
Description: This is a digitally signed message part

Re: Release status of i386 for Bullseye and long term support for 3 years?

[peter green]

   Then there was the short netbook boom, but AFAIR some early ones
   had 64bit CPUs but 32bit-only firmware.

My memory is that at the height of the boom the dominant processors
were the N270 and N280, which are 32-bit only. By the time 64-bit
netbook processors showed up the boom was on the decline.


There are at least two more:


5. People running Debian on virtual machines.

You can run an i386 VM with vmware or virtualbox with no special
hardware support. An x86-64 VM on the other hand requires VT-x
(or the AMD equivilent). While processor support for this is
the norm nowadays it's still often disabled by default
which can be a pain if you need to get IT support to access
bios setup on a machine.

i386 hardware is so numerous and widely spread, that every tiny fraction 
of i386 users might be more users than half of our release architectures 
combined. It is not even clear whether this is just an exaggeration or 
might be literally true:


i386 still gives 17281 popcon submissions, that is about
a tenth of amd64, but it's also over 10 times the next highest port

Now that probably doesn't reflect true usage, in particular
users who install using images tend to miss out on the popcon
question, but I still suspect that i386 is up there in the top
few most used architectures.

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Simon McVittie]

On Sat, 12 Dec 2020 at 18:09:02 +0200, Adrian Bunk wrote:
> 3. Computers that do support MMX and SSE2, but do not support 64bit.

Right, that's basically the second use-case I mentioned, but moving the
boundary for what we do and don't support to be more modern. We could
put the boundary anywhere we want, with higher baselines letting us make
more assumptions but excluding more old hardware.

> 4. People who wrongly installed i386 on amd64-capable hardware.

You're right that this doesn't match either of the use-cases I gave. If
this one is important to us, then that's a reason to keep a complete
bootable i386 system (with bootloaders and kernels), but we could move
its baseline as high as amd64's.

> Ubuntu has a business-minded focus, which is fair enough.
> But Debian should not blindly follow whatever Canonical
> does with Ubuntu for business reasons.[3]
> [3] neither should Debian blindly do the opposite

I agree - we need to weigh up the same advantages and disadvantages
that Canonical/Ubuntu did, but we might come to a different conclusion
because our priorities (and infrastructure) are different.

smcv

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Steve McIntyre]

Stephan wrote:
>Am Sa, Dez 12, 2020 at 18:09:02 +0200 schrieb Adrian Bunk:
>>4. People who wrongly installed i386 on amd64-capable hardware.
>
>Well, some releases ago befor multi-arch I used to install i386 even on 
>am64-capable hardware if ram was quite low (=< 8GB) and if the chance 
>wasn’t that low that you needed to install ia32-codec to watch ancient 
>video formats.
>
>I wouldn’t do it anymore but at least one system is still in use, and 
>there isn’t a real supported way to upgrade from i386 to amd64.

It's still quite new, but we have a package in the archive for this now:

  https://tracker.debian.org/pkg/debian-crossgrader

It's targeted at systems exactly like yours, tbh. Maybe give it a try?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"You can't barbecue lettuce!" -- Ellie Crane

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Stephan Seitz]

Am Sa, Dez 12, 2020 at 18:09:02 +0200 schrieb Adrian Bunk:

4. People who wrongly installed i386 on amd64-capable hardware.


Well, some releases ago befor multi-arch I used to install i386 even on 
am64-capable hardware if ram was quite low (=< 8GB) and if the chance 
wasn’t that low that you needed to install ia32-codec to watch ancient 
video formats.


I wouldn’t do it anymore but at least one system is still in use, and 
there isn’t a real supported way to upgrade from i386 to amd64.


Stephan

--
|If your life was a horse, you'd have to shoot it.|

Re: Release status of i386 for Bullseye and long term support for 3 years?

[Adrian Bunk]

On Tue, Dec 08, 2020 at 05:46:26PM +, Simon McVittie wrote:
>...
> I think it's necessary to consider what the purpose of the i386 port is,
> and set expectations and an appropriate baseline based on that.
> 
> I see two possible use-cases for i386:
> 
> 1. It's a compatibility layer for legacy 32-bit binaries on x86_64 CPUs:
>in particular, proprietary 32-bit binaries that we cannot recompile,
>32-bit builds of Wine, and the dependency stacks for those
> 
> 2. It's something you can genuinely run on older (or more-embedded)
>32-bit x86 CPUs that do not support x86_64 (down to some arbitrary
>baseline, currently i686 without MMX or SSE)

There are at least two more:

3. Computers that do support MMX and SSE2, but do not support 64bit.
   AFAIR the last 32bit-only CPU from Intel was released in 2007.[1]
   Then there was the short netbook boom, but AFAIR some early ones
   had 64bit CPUs but 32bit-only firmware.
   Surprisingly many netbooks are still in use even in first-world 
   countries.[2]

4. People who wrongly installed i386 on amd64-capable hardware.
   The existing userbase with this setup is large, and even though
   crossgrades are now finally possible it is better to wait until
   there are fewer users in this setup and more potential crossgrade 
   issues resolved.

> Ubuntu have chosen to support the first use-case, and only the first
> use-case. They longer ship a complete, bootable i386 operating system;
> instead, they have an i386 second-class-citizen architecture that
> is sufficient to provide graphics drivers and other shared libraries
> for legacy 32-bit proprietary binaries.
>...

Ubuntu has a business-minded focus, which is fair enough.
But Debian should not blindly follow whatever Canonical
does with Ubuntu for business reasons.[3]

It does make sense for Debian to differenciate by providing support for 
communities whose hardware is not or no longer supported by Ubuntu.

>...
> we could raise the
> baseline to include those and stop patching packages to avoid using them,
> which would remove i386's major oddity when compared with other
> architectures (i387 extended-precision floating point).

While this specific oddity is unique to the i386 port,
it is worth mentioning that every port has its own oddities.

> Also, if we are only supporting i386 for my first use-case, then I think
> we should seriously consider waiving the archive-completeness metric
> for i386: if big packages like web browsers and Libreoffice can't be
> compiled on 32-bit, then so be it. We only need to be able to compile
> a library stack, plus enough programs to be able to build and test that
> library stack on virtual or real hardware.

There is also a cost associated with having to modify packages for 
handling such port-specific omissions.

One current example for missing archive-completeness would be s390x,
and I guess you are aware how much pain its headless nature has caused 
in some places.

> If the second use-case is supported, then we are going to need an i386
> porter team analogous to any other architecture porting team, that can
> take responsibility for choosing an achievable baseline for the oldest
> i386 CPU that they intend to test and support, triaging i386-specific
> bugs, and providing patches where necessary to keep packages below that
> baseline (which will require an increasing amount of work over time if
> they choose a baseline that is suitable for historical x86 CPUs, since
> increasingly many upstreams refuse to support the i387 FPU). I don't
> think it's reasonable any more to expect individual package maintainers
> to understand the finer points of the historical x86 instruction set.

This is correct.

This is exactly porter work, and this is what I have already been doing 
for years for i386.

A large part of porter work is being a one-trick pony, often submitting
the same fix for many packages. For i386 this is usually some variant of
ifneq (,$(filter $(DEB_HOST_ARCH), i386))
export DEB_CXXFLAGS_MAINT_APPEND += -ffloat-store
endif

> One factor that makes the second use-case difficult to support is
> that even if developers still have old 32-bit x86 hardware, it's very
> likely to be considerably newer than our current baseline. For example,
> mainstream Intel and AMD 32-bit CPUs gained SSE support around 20 years
> ago (Pentium III and Athlon XP). I know there are somewhat newer x86
> embedded CPUs with lesser capabilities, but most developers would have
> to have deliberately chosen to obtain those, rather than having access
> to old machines just because they haven't disposed of them yet.

It is not realistic to expect porters to have hardware matching exactly 
the port baseline.

How many people do have hardware matching exactly our amd64 baseline?

Does hardware matching exactly our amd64 baseline even exist?

> I don't think it's realistic to drop i386 completely, and I want to
> be able to continue to run legacy 32-bit games; so if an i386 porter