Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
Hi Adrian,
Sorry for not replying early, busy with preparing the updates.
On Fri, Sep 29, 2023 at 03:41:15AM +0300, Adrian Bunk wrote:
> On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote:
> >...
> > Note that the last time the problem arised already earlier in
> > experimental and Ben workarounded it there with
> > https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
> > We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
> > the performance of kallsyms_lookup_name()") was in fact backported to
> > 6.1.42. So this is next I would try and disable MPTCP and
> > FUNCTION_TRACER.
> >...
>
> Yes, that looks reasonable.
Great thanks, this landed now for the point release udpates and in
fact we have the armel builds back.
> Additionally, one generic cause of bloat is:
> debian/changelog: * Enable UNICODE. (closes: #985689)
> debian/config/config:CONFIG_UNICODE=y
>
> That's 74 kB uncompressed, and there doesn't seem to be any
> justification for not making it modular. It's not urgent since
> Bens change handles the immediate problem, but worth changing
> in unstable.
Could you fill a bug against src:linux for it, so this might be
further addressed in unstable for armel?
Regards,
Salvatore
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On 2023-09-09 08:15, Salvatore Bonaccorso wrote: Hi all, ... I doupt anybody is sensibly using armel nowdays under bookworm, so my proposed course of action for unblock the bookworm-security update is: ... FYI, An old, headless, raspberry pi (armel) with GPS and camera, on Debian testing, was being sensibly used for long-term, slow, time-lapse photography, where it was good enough for the intended purpose. Another pi was slated for print server duties, but was mostly just idling because the particular printer was not supported. Both stopped responding over the network after upgrades some months ago, so this decision wasn't involved; the problem was probably caused by systemd network interface renaming. The hope was to get them going again with a little debugging, or a re-install if necessary, as builds are still being made[1]. It's always sad to see support have to be dropped for old functioning hardware still sensibly serving purposes they can accomplish. But, understandable with the size growth required to support newer "better" hardware. Thanks for keeping them alive, and up to date, as long as they were. [1] https://raspi.debian.net/daily-images/
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote:
>...
> Note that the last time the problem arised already earlier in
> experimental and Ben workarounded it there with
> https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
> We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
> the performance of kallsyms_lookup_name()") was in fact backported to
> 6.1.42. So this is next I would try and disable MPTCP and
> FUNCTION_TRACER.
>...
Yes, that looks reasonable.
Additionally, one generic cause of bloat is:
debian/changelog: * Enable UNICODE. (closes: #985689)
debian/config/config:CONFIG_UNICODE=y
That's 74 kB uncompressed, and there doesn't seem to be any
justification for not making it modular. It's not urgent since
Bens change handles the immediate problem, but worth changing
in unstable.
cu
Adrian
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sun, Sep 10, 2023 at 05:13:52PM +0200, Ben Hutchings wrote: > On Sat, 2023-09-09 at 11:13 +0200, Paul Gevers wrote: > > Hi Salvatore, > > > > On 09-09-2023 10:15, Salvatore Bonaccorso wrote: > > > but should have been support for armel been > > > dropped earlier and should we do it for trixie > > > > The kernel for armel went over some hardware limits before (I was > > affected with my NAS, where I couldn't upgrade the kernel to bullseye as > > documented in the release notes [1]). Is the current situation reaching > > the limit for all armel devices, or "just" for some and are the others > > probably fine for some years to come? > > The issue exists with many devices supported by the "marvell" flavour. > We also have an "rpi" flavour for armel that supports Raspberry Pi > models with Arm v6 CPUs, and that doesn't have any size limits that > we've needed to worry about yet. I happily delayed the issue by changing the layout of the mtd partitions of my QNAP with a nice script [0]. It's able to handle a good number of models and I think it's a great solution for the short/medium term. > > If we're now reaching the final limit and if it was foreseeable that we > > would reach that limit, then yes it would have made sense to drop armel > > *before* the bookworm release, but alas. If the kernel team can't > > support the kernel on armel, than armel shouldn't be a release > > architecture for trixie. If it's only some devices, than we "just" need > > to communicate that clearly. > [...] > > I would be quite happy to drop the "marvell" flavour for trixie. The > size limits are a recurring problem, and the supported SoCs and devices > seem to have been out of production for a long time. In contrast, the > Raspberry Pi models with v6 CPUs are still in production. I would find reasonable if trixie dropped "marvell" flavor. The bell for the armel QNAP devices has been warning for years already, anybody caring of their data should move it somewhere else. > (I do have my doubts as to whether it will be possible to continue > supporting a useful user-space for armel, but I certainly can't speak > authoritatively about that.) > > Ben. > > -- > Ben Hutchings - Debian developer, member of kernel, installer and LTS > teams [0] https://github.com/amouiche/qnap_mtd_resize_for_bullseye -- rsa4096: 3B10 0CA1 8674 ACBA B4FE FCD2 CE5B CF17 9960 DE13 ed25519: FFB4 0CC3 7F2E 091D F7DA 356E CC79 2832 ED38 CB05 signature.asc Description: PGP signature
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sun, Sep 10, 2023 at 10:55:49AM +0800, Paul Wise wrote: > On Sat, 2023-09-09 at 11:42 +0200, Bastian Blank wrote: > > > The first one is the one with included size limitations, because those > > load the kernel from a pre-defined flash partition, whose size can't be > > easily changed by the user. This one is now overflowing for the second > > to last documented one in the kernel package config. > > Seems like this would be solvable by writing a bootloader to the flash > partition that would be able to load Linux from a normal filesystem? Yes, if the support for the given device is mainlined. I doubt this is the case for my QNAP though, would not dare to try. -- rsa4096: 3B10 0CA1 8674 ACBA B4FE FCD2 CE5B CF17 9960 DE13 ed25519: FFB4 0CC3 7F2E 091D F7DA 356E CC79 2832 ED38 CB05 signature.asc Description: PGP signature
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sat, Sep 09, 2023 at 12:43:20PM +0200, Salvatore Bonaccorso wrote: > Hi, > > On Sat, Sep 09, 2023 at 11:49:11AM +0300, Adrian Bunk wrote: > > On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote: > > >... > > > - Relese the DSA without armel builds. This is not optimal and for the > > > point release > > > we need to have to have all builds, but this gives people who still are > > > interested > > > in this architecture to step up and propose a fix for the problem, > > > otherwise then > > > disable the image size check, and then effectively dropping some > > > support. > > >... > > > armel people, can you have ideally look at it ASAP on the comments > > > please, I would not like to delay the DSA for linux on > > > bookworm-security too much. > > > > Releasing this DSA without armel and sorting out the issue for the point > > release sounds like the best option to me. > > FWIW, following Ben's aproach for unstable, here is my proposed change > for bookworm in the near-term: > > https://salsa.debian.org/kernel-team/linux/-/merge_requests/844 > > I have verified by cross-building that the image size goes down to > > Image size 2644124/2729712, using 96.86%. Image fits. Continuing. > > which would be sufficient so far. > > So we can at least include the above for the point release and > releasing the DSA earlier without the armel builds. Looks good to me. Thanks! > > Thank you! > > Regards, > Salvatore > -- rsa4096: 3B10 0CA1 8674 ACBA B4FE FCD2 CE5B CF17 9960 DE13 ed25519: FFB4 0CC3 7F2E 091D F7DA 356E CC79 2832 ED38 CB05 signature.asc Description: PGP signature
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sat, 2023-09-09 at 11:13 +0200, Paul Gevers wrote: > Hi Salvatore, > > On 09-09-2023 10:15, Salvatore Bonaccorso wrote: > > but should have been support for armel been > > dropped earlier and should we do it for trixie > > The kernel for armel went over some hardware limits before (I was > affected with my NAS, where I couldn't upgrade the kernel to bullseye as > documented in the release notes [1]). Is the current situation reaching > the limit for all armel devices, or "just" for some and are the others > probably fine for some years to come? The issue exists with many devices supported by the "marvell" flavour. We also have an "rpi" flavour for armel that supports Raspberry Pi models with Arm v6 CPUs, and that doesn't have any size limits that we've needed to worry about yet. > If we're now reaching the final limit and if it was foreseeable that we > would reach that limit, then yes it would have made sense to drop armel > *before* the bookworm release, but alas. If the kernel team can't > support the kernel on armel, than armel shouldn't be a release > architecture for trixie. If it's only some devices, than we "just" need > to communicate that clearly. [...] I would be quite happy to drop the "marvell" flavour for trixie. The size limits are a recurring problem, and the supported SoCs and devices seem to have been out of production for a long time. In contrast, the Raspberry Pi models with v6 CPUs are still in production. (I do have my doubts as to whether it will be possible to continue supporting a useful user-space for armel, but I certainly can't speak authoritatively about that.) Ben. -- Ben Hutchings - Debian developer, member of kernel, installer and LTS teams signature.asc Description: This is a digitally signed message part
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sat, 2023-09-09 at 11:42 +0200, Bastian Blank wrote: > The first one is the one with included size limitations, because those > load the kernel from a pre-defined flash partition, whose size can't be > easily changed by the user. This one is now overflowing for the second > to last documented one in the kernel package config. Seems like this would be solvable by writing a bootloader to the flash partition that would be able to load Linux from a normal filesystem? -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sat, Sep 09, 2023 at 11:42:45AM +0200, Bastian Blank wrote: > Hi > > On Sat, Sep 09, 2023 at 11:13:56AM +0200, Paul Gevers wrote: > > If we're now reaching the final limit and if it was foreseeable that we > > would reach that limit, then yes it would have made sense to drop armel > > *before* the bookworm release, but alas. If the kernel team can't support > > the kernel on armel, than armel shouldn't be a release architecture for > > trixie. If it's only some devices, than we "just" need to communicate that > > clearly. > > We have two armel kernel currently: > - "rpi", for Raspberry Pi 1 and related devices. > > The second one is for the original Raspberry Pi 1 type. There we don't > have any size limits, as the kernel is loaded from a file system. > However those systems contain a ARMv6 CPU. So our armel port is only > partially usable anyway, as is is built for ARMv4. There exists with > Raspbian a better suited forked distribution with ARMv6 as target. > No - Raspbian contains commercial software now by default. We have to use Raspberry Pi firmware but Raspberry pi is *not* purely ARM v6 it's ARM v6 with hardware floating point - and therefore incompatible with Debian and every other ARM v6 version. Although it is less than optimal, it is still perfectly supportable by all the packages in armel. > So yes there is a small number of devices we can still support with the > armel port, but where we are a bad choice. > See above: note that the Raspberry Pi Zero (original version) is still very much in production and is 32 bit and ARM v6 hardware floating point. It is still a prime target for armel - and the number of devices produced is not small. > Everything newer is ARMv7, supported by the armhf port, or ARMv8, > supported by the arm64 port. > > Latest popcon for stable is: > > linux-image-marvell: 31 > linux-image-rpi: 7 > > Debian itself does not have any armel hardware. Everything is done on > armhf or arm64. Sadly the armhf supporting systems are already in the > progress of drying up. Even some ARMv8 vendors do not longer include > 32bit support. > This is, unfortunately, the case. I'm not sure where our sd card images are built but the Raspberry Pi iamges are built from Gunnar Wolf's scripts primarily for everything prior to RPi4. All best, as ever, Andy [amaca...@debian.org] > Bastian > > -- > Each kiss is as the first. > -- Miramanee, Kirk's wife, "The Paradise Syndrome", > stardate 4842.6 >
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
Hi, On Sat, Sep 09, 2023 at 11:49:11AM +0300, Adrian Bunk wrote: > On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote: > >... > > - Relese the DSA without armel builds. This is not optimal and for the > > point release > > we need to have to have all builds, but this gives people who still are > > interested > > in this architecture to step up and propose a fix for the problem, > > otherwise then > > disable the image size check, and then effectively dropping some support. > >... > > armel people, can you have ideally look at it ASAP on the comments > > please, I would not like to delay the DSA for linux on > > bookworm-security too much. > > Releasing this DSA without armel and sorting out the issue for the point > release sounds like the best option to me. FWIW, following Ben's aproach for unstable, here is my proposed change for bookworm in the near-term: https://salsa.debian.org/kernel-team/linux/-/merge_requests/844 I have verified by cross-building that the image size goes down to Image size 2644124/2729712, using 96.86%. Image fits. Continuing. which would be sufficient so far. So we can at least include the above for the point release and releasing the DSA earlier without the armel builds. Thank you! Regards, Salvatore
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
Hi On Sat, Sep 09, 2023 at 11:13:56AM +0200, Paul Gevers wrote: > If we're now reaching the final limit and if it was foreseeable that we > would reach that limit, then yes it would have made sense to drop armel > *before* the bookworm release, but alas. If the kernel team can't support > the kernel on armel, than armel shouldn't be a release architecture for > trixie. If it's only some devices, than we "just" need to communicate that > clearly. We have two armel kernel currently: - "marvell", for some CPU from Marvell, and - "rpi", for Raspberry Pi 1 and related devices. The first one is the one with included size limitations, because those load the kernel from a pre-defined flash partition, whose size can't be easily changed by the user. This one is now overflowing for the second to last documented one in the kernel package config. The second one is for the original Raspberry Pi 1 type. There we don't have any size limits, as the kernel is loaded from a file system. However those systems contain a ARMv6 CPU. So our armel port is only partially usable anyway, as is is built for ARMv4. There exists with Raspbian a better suited forked distribution with ARMv6 as target. So yes there is a small number of devices we can still support with the armel port, but where we are a bad choice. Everything newer is ARMv7, supported by the armhf port, or ARMv8, supported by the arm64 port. Latest popcon for stable is: linux-image-marvell: 31 linux-image-rpi: 7 Debian itself does not have any armel hardware. Everything is done on armhf or arm64. Sadly the armhf supporting systems are already in the progress of drying up. Even some ARMv8 vendors do not longer include 32bit support. Bastian -- Each kiss is as the first. -- Miramanee, Kirk's wife, "The Paradise Syndrome", stardate 4842.6
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
Hi Salvatore, On 09-09-2023 10:15, Salvatore Bonaccorso wrote: but should have been support for armel been dropped earlier and should we do it for trixie The kernel for armel went over some hardware limits before (I was affected with my NAS, where I couldn't upgrade the kernel to bullseye as documented in the release notes [1]). Is the current situation reaching the limit for all armel devices, or "just" for some and are the others probably fine for some years to come? If we're now reaching the final limit and if it was foreseeable that we would reach that limit, then yes it would have made sense to drop armel *before* the bookworm release, but alas. If the kernel team can't support the kernel on armel, than armel shouldn't be a release architecture for trixie. If it's only some devices, than we "just" need to communicate that clearly. I don't have a clear advice for the current situation in security and the next point release, let's hope you can stretch the situation a bit longer. I recall that the kernel package has safety checks in place and refuses to *try* to install the kernel if it doesn't fit on the hardware. That means that you don't cripple the hardware of affected people, but "merely" can't give them security support? I guess it would be possible (as long as support lasts; no LTS support) for effected systems to run the security supported bullseye kernel. Paul [1] https://www.debian.org/releases/bullseye/armel/release-notes/ch-information.en.html#no-longer-supported-hardware OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote: >... > - Relese the DSA without armel builds. This is not optimal and for the point > release > we need to have to have all builds, but this gives people who still are > interested > in this architecture to step up and propose a fix for the problem, > otherwise then > disable the image size check, and then effectively dropping some support. >... > armel people, can you have ideally look at it ASAP on the comments > please, I would not like to delay the DSA for linux on > bookworm-security too much. Releasing this DSA without armel and sorting out the issue for the point release sounds like the best option to me. > Thanks for having a look, > > Regards, > Salvatore cu Adrian
Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems
Hi all,
We have problem with the image size of armel builds in bookworm. There
is a pending bookworm-security linux update pending which is currently
blocked due to armel FTBFS due to the image size increase:
https://people.debian.org/~carnil/buildd-logs/linux/linux_6.1.52-1_armel-2023-09-07T08:53:41Z.gz
debian/bin/buildcheck.py debian/build/build_armel_none_marvell armel none
marvell
Can't read ABI reference. ABI not checked!
Image size 2753652/2729712, using 100.88%. Too large. Refusing to continue.
make[2]: *** [debian/rules.real:169: debian/stamps/build_armel_none_marvell]
Error 1
make[2]: Leaving directory '/<>'
make[1]: *** [debian/rules.gen:1615: build-arch_armel_none_marvell_real_image]
Error 2
make[1]: Leaving directory '/<>'
make: *** [debian/rules:39: build-arch] Error 2
dpkg-buildpackage: error: debian/rules binary-arch subprocess returned exit
status 2
In fact we are already too narrow to 100% in any case, but there was a
bump between 6.1.41 and 6.1.42 upstream AFAICS:
6.1.52-1 Image size 2751596/2729712, using 100.80%. Too large. Refusing to
continue.
6.1.51-1 Image size 2752212/2729712, using 100.82%. Too large. Refusing to
continue.
6.1.47-1 Image size 2752676/2729712, using 100.84%. Too large. Refusing to
continue.
6.1.45-1 Image size 2751292/2729712, using 100.79%. Too large. Refusing to
continue.
6.1.43-1 Image size 2751348/2729712, using 100.79%. Too large. Refusing to
continue.
6.1.42-1 Image size 2752924/2729712, using 100.85%. Too large. Refusing to
continue.
6.1.41-1 Image size 2701348/2729712, using 98.96%. Image fits. Continuing.
6.1.40-1 Image size 2703956/2729712, using 99.06%. Under 1% space in
UNRELEASED. Continuing.
6.1.38-1 Image size 2703076/2729712, using 99.02%. Under 1% space in bookworm.
Continuing.
I doupt anybody is sensibly using armel nowdays under bookworm, so my proposed
course of action for unblock the bookworm-security update is:
Either
- ignore the image size and implicitly drop support for devices which would
break
due to size constraints, the current upper limit is adjusted for the
following:
# Buffalo Linkstation LS-WSXL/WXL/WVL (from stock kernel): 2729776 - 64 =
2729712
or:
- Relese the DSA without armel builds. This is not optimal and for the point
release
we need to have to have all builds, but this gives people who still are
interested
in this architecture to step up and propose a fix for the problem, otherwise
then
disable the image size check, and then effectively dropping some support.
Attached is the result of bloat-o-meter script between 6.1.41 and 6.1.42.
I might put me in a bad spot, but should have been support for armel been
dropped earlier and should we do it for trixie following the same done for
mipsel?
Note that the last time the problem arised already earlier in
experimental and Ben workarounded it there with
https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
the performance of kallsyms_lookup_name()") was in fact backported to
6.1.42. So this is next I would try and disable MPTCP and
FUNCTION_TRACER. But the problem with armel will remain.
armel people, can you have ideally look at it ASAP on the comments
please, I would not like to delay the DSA for linux on
bookworm-security too much.
Thanks for having a look,
Regards,
Salvatore
add/remove: 7/6 grow/shrink: 50/14 up/down: 3772/-2456 (1316)
Function old new delta
check_max_stack_depth_subprog - 720+720
psi_rtpoll_worker - 648+648
update_triggers- 504+504
kallsyms_lookup_names.constprop- 264+264
do_check_common 9892 10068+176
__mark_chain_precision 20082148+140
psi_trigger_create 564 684+120
dquot_writeback_dquots 428 548+120
psi_trigger_destroy 344 448+104
psi_schedule_rtpoll_work - 88 +88
__check_func_call880 968 +88
collect_percpu_times 368 452 +84
is_callback_calling_function - 64 +64
list_add22082256 +48
__inet_hash 436 484 +48
request_key_and_link14041448 +44
kvmalloc_array - 40 +40
bpf_lru_pop_free 708 748 +40
list_add_tail 22682304 +36
ip_send_unicast_reply784 820 +36
psi_avgs_work180 212 +32
bpf_check 10812 10844 +32
