Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Fri, Apr 14, 2023 at 08:58:55PM -0700, Steve Langasek wrote: > Sorry for the long radio silence on this; I had identified some issues with > my prior report and wanted to rerun it with some fixes, and that analysis > took rather longer than expected (mainly due to infrastructure slowness). > Refreshed output for Ubuntu is here: > https://people.canonical.com/~vorlon/armhf-time_t/ > So comparing with Wookey's results, I see: > Total -dev packages tested: 4603 > Time_t changes ABI: 349 > LFS changes ABI: 71 > ABI unchanged: 2085 > Compilation failed: 2098 > So certainly this appears to be the same order of magnitude as Wookey's > separate analysis. A little more digging here into the failures; I've prioritized them by number of reverse-build-depends: while read devpkg; do count=$(grep-dctrl -w -FBuild-Depends $devpkg -n -sPackage \ /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources \ | sort -u | wc -l) echo "$count $devpkg"; done < failed-compilation \ | sort -n -r > sorted-revdep-count One nice finding is that 807 of the libraries that have failed analysis have no reverse-build-dependencies at all, so can be more or less ignored for the transition (we should still follow through on the analysis and rename packages, but this can happen any time during the trixie release cycle). One *unfortunate* finding, having fixed up the compilation of the top 5 failures, is that each of libssl-dev, libgtk-3-dev, and libgtk2.0-dev breaks ABI with 64-bit time_t. > The other question is what impact this has on the archive as a whole, with > triggering of rebuild transitions. Looking at the > reverse-build-dependencies of those 349 packages gives: > > $ while read pkg; do grep-dctrl -n -sPackage -w -FBuild-Depends $pkg > /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources ; done < > abi-breaks | sort -u | wc -l > 3626 > $ But on the bright side again, this only increases the number of revdeps that are part of the transition from 3626 to 4004, due to overlap with other libraries. From my perspective, this is entirely achievable and we should move forward with discussing this more broadly on debian-devel. Note that while armhf is the priority wrt Ubuntu, this would also address 2038 compatibility for armel and mipsel as Debian release archs. (I do not think we should change i386 to 64-bit time_t as the primary value of this arch is for binary compatibility with legacy applications that can't be recompiled.) > NB this is a bit of an undercount as a result of packages that ship headers > but aren't the thing being directly build-depended on. A more accurate > count is probably to look at binary packages, from the same source as the > -dev package, that the -dev package depends on and look at > reverse-dependencies of those. But for a first pass, here's where we are. > There are also roughly 2100 packages for which compilation failed. To know > which packages actually have ABI changes and require transitions, we would > need to work through this list and get the count down to 0. > But as a first-order approximation, this looks doable to me as a transition > we could manage. > What do other folks think? Are you on board with proposing this to > debian-devel? Anyone want to help add the necessary quirks to drive down > the list of compilation failures? > On Tue, Mar 21, 2023 at 03:00:41AM +, Wookey wrote: > > On 2023-02-15 17:08 +, Wookey wrote: > > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > > > > > Yes I think we should proceed with this analysis on debian to get a > > > better handle on just how many libraries we are looking at. > > > > OK. We have over 10,000 *-dev package in debian so this took a while, > > but I've now finished an initial pass. > > The results are: > > Total -dev packages: 10323 > > Time_t changes ABI: 329 > > LFS changes ABI: 580.6% > > ABI unchanged: 18405.6% > > Compilation failed: 1637 > > No headers in pkg: 3852 > > No library in pkg: 5086 > > total analysed: 10249 > > (note that the no-headers and no-libs categories are not exclusive > > - the other classifications are) > > > > Not quite sure where we lost 74 packages down the back of the sofa but at > > 0.7% it's noise. > > > > So we have about 5000 packages which can basically be ignored for this > > purpose: > > golang* (1943) > > librust* (1955) - source-only, no dynamic linking, no .so's in any package > > and libghc* (1065) - changes ABI at drop of hat (every new version) anyway. > > > > Of the remaining 5360, 5237 have .so files to check. Of those 329 > > change ABI and another 58 are not built with LFS currently > > so also change ABI due to that. That's 387 (7%). > > > > 34% are fine. And an annoyingly large 1637 (30%) did not build under > > Abi-compliance-checker to determine one way or the other. Assuming 7% > > of those are a problem that's another 114 packages. > > > > I've yet to determine how many of
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Wed, Feb 15, 2023 at 05:08:40PM +, Wookey wrote:
> On 2023-02-04 21:42 -0800, Steve Langasek wrote:
> > On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
> > > This is good. One thing that I think has been missing from the discussion
> > > about armhf rebootstrap is the fact that we do have experience in Debian
> > > doing cross-cutting ABI transitions without having to change an
> > > architecture name. We've done this at least three times in Debian
> > > history:
> > > the libc5->libc6 transition ('g' suffix - which still remains today in
> > > libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
> > > double' transition ('ldbl' suffix, and an example of doing this for an ABI
> > > transition that didn't affect all architectures).
> I feel like I should already know this, but after a bit of thought,
> and reading the transitions wiki page:
> https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
> understand how the suffixes are used in these transitions.
> That wiki page does not appear to mention adding suffixes to package
> names. It appears to only document the process where the version is
> just bumped, everything involved is rebuilt in experimental and then
> re-uploaded to unstable once we are ready to make the change.
> I presume that the suffix is used for transitions involving a large
> number of packages, in order to keep both the old-ABI and new-ABI
> version of the packages around in parallel (and conflicting with each
> other?) for a while which is necessary for larger transitions because
> doing it all in one go would take too long, or get entangled with
> uploads from the unware that might cause breakage, or something?
When we have an ABI change in the distro that affects shared libraries, but
is not the result of sourceful changes to the upstream source, we don't want
to change the soname of the library; that's upstream's domain.
But we do need at the packaging level to distinguish between two packages of
the library that provide different ABIs, so that packages depending on one
ABI don't incorrectly get the package with the other ABI, resulting in
broken runtime behavior.
When we transition, the un-annotated package will remain present in older
releases, but will be entirely superseded in unstable by the package
providing the new ABI.
BTW, I'll suggest "t64" as the library name suffix here.
> Just rebuilding and not renaming sounds like a much smoother process,
> because as soon as we rename then there is a load of renaming in
> reverse dependencies too, and there is presumably a set of rules about
> when conflicts are added and other details that need to be got right.
No, that breaks partial upgrades and makes a right mess of unstable *and* of
upgrades between stable releases. "Just rebuilding and not renaming" is the
"rebootstrap" approach which I am hoping to avoid.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
Sorry for the long radio silence on this; I had identified some issues with my prior report and wanted to rerun it with some fixes, and that analysis took rather longer than expected (mainly due to infrastructure slowness). Refreshed output for Ubuntu is here: https://people.canonical.com/~vorlon/armhf-time_t/ So comparing with Wookey's results, I see: Total -dev packages tested: 4603 Time_t changes ABI: 349 LFS changes ABI: 71 ABI unchanged: 2085 Compilation failed: 2098 So certainly this appears to be the same order of magnitude as Wookey's separate analysis. The other question is what impact this has on the archive as a whole, with triggering of rebuild transitions. Looking at the reverse-build-dependencies of those 349 packages gives: $ while read pkg; do grep-dctrl -n -sPackage -w -FBuild-Depends $pkg /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources ; done < abi-breaks | sort -u | wc -l 3626 $ NB this is a bit of an undercount as a result of packages that ship headers but aren't the thing being directly build-depended on. A more accurate count is probably to look at binary packages, from the same source as the -dev package, that the -dev package depends on and look at reverse-dependencies of those. But for a first pass, here's where we are. There are also roughly 2100 packages for which compilation failed. To know which packages actually have ABI changes and require transitions, we would need to work through this list and get the count down to 0. But as a first-order approximation, this looks doable to me as a transition we could manage. What do other folks think? Are you on board with proposing this to debian-devel? Anyone want to help add the necessary quirks to drive down the list of compilation failures? On Tue, Mar 21, 2023 at 03:00:41AM +, Wookey wrote: > On 2023-02-15 17:08 +, Wookey wrote: > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > > > Yes I think we should proceed with this analysis on debian to get a > > better handle on just how many libraries we are looking at. > > OK. We have over 10,000 *-dev package in debian so this took a while, > but I've now finished an initial pass. > The results are: > Total -dev packages: 10323 > Time_t changes ABI: 329 > LFS changes ABI: 580.6% > ABI unchanged: 18405.6% > Compilation failed: 1637 > No headers in pkg: 3852 > No library in pkg: 5086 > total analysed: 10249 > (note that the no-headers and no-libs categories are not exclusive > - the other classifications are) > > Not quite sure where we lost 74 packages down the back of the sofa but at > 0.7% it's noise. > > So we have about 5000 packages which can basically be ignored for this > purpose: > golang* (1943) > librust* (1955) - source-only, no dynamic linking, no .so's in any package > and libghc* (1065) - changes ABI at drop of hat (every new version) anyway. > > Of the remaining 5360, 5237 have .so files to check. Of those 329 > change ABI and another 58 are not built with LFS currently > so also change ABI due to that. That's 387 (7%). > > 34% are fine. And an annoyingly large 1637 (30%) did not build under > Abi-compliance-checker to determine one way or the other. Assuming 7% > of those are a problem that's another 114 packages. > > I've yet to determine how many of the 'no-libs/no-headers' packages > actually expose an ABI we need to worry about. There will be some > more, from stuff like boost and Qt. > > So the scale of the transition task in Debian is quite a lot bigger than in > Ubuntu: Probably 400-500 packages. > > I've linked my lists on the wiki page. > https://wiki.debian.org/ReleaseGoals/64bit-time > > I believe Steve L (who has done most of the heavy lifting on the > script) is running a check too and will have some results soon. > > Wookey > -- > Principal hats: Debian, Wookware, ARM > http://wookware.org/ -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On 2023-02-15 17:08 +, Wookey wrote: > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > Yes I think we should proceed with this analysis on debian to get a > better handle on just how many libraries we are looking at. OK. We have over 10,000 *-dev package in debian so this took a while, but I've now finished an initial pass. The results are: Total -dev packages: 10323 Time_t changes ABI: 329 LFS changes ABI: 580.6% ABI unchanged: 18405.6% Compilation failed: 1637 No headers in pkg: 3852 No library in pkg: 5086 total analysed: 10249 (note that the no-headers and no-libs categories are not exclusive - the other classifications are) Not quite sure where we lost 74 packages down the back of the sofa but at 0.7% it's noise. So we have about 5000 packages which can basically be ignored for this purpose: golang* (1943) librust* (1955) - source-only, no dynamic linking, no .so's in any package and libghc* (1065) - changes ABI at drop of hat (every new version) anyway. Of the remaining 5360, 5237 have .so files to check. Of those 329 change ABI and another 58 are not built with LFS currently so also change ABI due to that. That's 387 (7%). 34% are fine. And an annoyingly large 1637 (30%) did not build under Abi-compliance-checker to determine one way or the other. Assuming 7% of those are a problem that's another 114 packages. I've yet to determine how many of the 'no-libs/no-headers' packages actually expose an ABI we need to worry about. There will be some more, from stuff like boost and Qt. So the scale of the transition task in Debian is quite a lot bigger than in Ubuntu: Probably 400-500 packages. I've linked my lists on the wiki page. https://wiki.debian.org/ReleaseGoals/64bit-time I believe Steve L (who has done most of the heavy lifting on the script) is running a check too and will have some results soon. Wookey -- Principal hats: Debian, Wookware, ARM http://wookware.org/ signature.asc Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Fri, 2023-03-03 at 04:07 +, Wookey wrote: > Is there a place one can grep all buildlogs? Debian members can access them on the buildd.debian.org server for eg: /srv/buildd.debian.org/db/0/0ad/0.0.11-1/i386_1347540026_log.bz2 Please keep an eye on system load while searching them though. If you only need build-dep versions, probably a better option is the archive of buildinfo files and the database for searching it. https://buildinfos.debian.net/ftp-master.debian.org/ https://salsa.debian.org/bremner/builtin-pho/ -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On 2023-02-20 17:53 +0100, Diederik de Haas wrote: > On Monday, 20 February 2023 16:24:37 CET Arnd Bergmann wrote: > > On Wed, Feb 15, 2023, at 18:08, Wookey wrote: > > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > >> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote: > > >> > > >> Does doing an ABI transition of ~113 libraries seem tractable to folks? > > > > > > It certainly provides evidence for the idea that this is not > > > completely intractable, which I think many people (including me) > > > worried was the case initially. > > > > When I did a similar analysis a few years ago using just pattern-matching > > on header files, the result was that more than half the total packages > > in Debian depended on at least one other package that needed an ABI > > transition, which in my mind made it unrealistic. > If you do it in the early stage of Trixie's dev cycle, would it still be > unrealistic? It may be a bumpy ride and take a while, but I don't see an > immediate issue with that. Sid may finally become Unstable again ;-P > > But *when* you do it, is quite relevant. If you/we are only a few months away > from the Trixie Freeze, then it's probably not a good idea. > But if we're 1-1.5 years before that, there's plenty of time to fix things. > > Or is that too simplistic on my part? No. I think that's reasonable. One thing we do have to worry about right now is that anything which has enabled LFS anytime in the last 15 years that rebuilds against a current gnulib will automatically get 64-bit time_t unless they explicitly turn it off. We may well have packages in the archive that have had this done to them without noticing, and clearly we don't want to do this in bookworm. I'm not quite sure what the best way to check for this is. Is there a place one can grep all buildlogs? AIUI gnulib is statically included in the build a-la autoconf so it probably only happens when a new upstream is pulled in with updated gnulib. BICBW. gnulib doesn't seem to do releases (last tag 9 years ago 'v0.1') so I'm not quite sure when this changed. Ah July 2021: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=history;f=m4/year2038.m4;h=2e4427e6fac10550c99748abebf31b61e6afda2b;hb=b9bf95fd0a6ab666b484b1e224321664f051f7fa So the module has existed since 2017, but only defaults on since mid 2021. I'm not sure what one should be looking for in build logs to set off the alarms. Wookey -- Principal hats: Debian, Wookware, ARM http://wookware.org/ signature.asc Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Monday, 20 February 2023 16:24:37 CET Arnd Bergmann wrote: > On Wed, Feb 15, 2023, at 18:08, Wookey wrote: > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > >> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote: > >> > >> Does doing an ABI transition of ~113 libraries seem tractable to folks? > > > > It certainly provides evidence for the idea that this is not > > completely intractable, which I think many people (including me) > > worried was the case initially. > > When I did a similar analysis a few years ago using just pattern-matching > on header files, the result was that more than half the total packages > in Debian depended on at least one other package that needed an ABI > transition, which in my mind made it unrealistic. If you do it in the early stage of Trixie's dev cycle, would it still be unrealistic? It may be a bumpy ride and take a while, but I don't see an immediate issue with that. Sid may finally become Unstable again ;-P But *when* you do it, is quite relevant. If you/we are only a few months away from the Trixie Freeze, then it's probably not a good idea. But if we're 1-1.5 years before that, there's plenty of time to fix things. Or is that too simplistic on my part? signature.asc Description: This is a digitally signed message part.
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Wed, Feb 15, 2023, at 18:08, Wookey wrote: > On 2023-02-04 21:42 -0800, Steve Langasek wrote: >> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote: > >> Does doing an ABI transition of ~113 libraries seem tractable to folks? > > It certainly provides evidence for the idea that this is not > completely intractable, which I think many people (including me) > worried was the case initially. When I did a similar analysis a few years ago using just pattern-matching on header files, the result was that more than half the total packages in Debian depended on at least one other package that needed an ABI transition, which in my mind made it unrealistic. It's possible that there were a lot of false-positive results in my analysis then, but I would still suggest looking at what portion of the archive depends on the libraries that you identified this time. Arnd
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
> > This is good. One thing that I think has been missing from the discussion
> > about armhf rebootstrap is the fact that we do have experience in Debian
> > doing cross-cutting ABI transitions without having to change an
> > architecture name. We've done this at least three times in Debian history:
> > the libc5->libc6 transition ('g' suffix - which still remains today in
> > libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
> > double' transition ('ldbl' suffix, and an example of doing this for an ABI
> > transition that didn't affect all architectures).
I feel like I should already know this, but after a bit of thought,
and reading the transitions wiki page:
https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
understand how the suffixes are used in these transitions.
That wiki page does not appear to mention adding suffixes to package
names. It appears to only document the process where the version is
just bumped, everything involved is rebuilt in experimental and then
re-uploaded to unstable once we are ready to make the change.
I presume that the suffix is used for transitions involving a large
number of packages, in order to keep both the old-ABI and new-ABI
version of the packages around in parallel (and conflicting with each
other?) for a while which is necessary for larger transitions because
doing it all in one go would take too long, or get entangled with
uploads from the unware that might cause breakage, or something?
Can someone point at an explanation of the process, or add it to the above page?
Just rebuilding and not renaming sounds like a much smoother process,
because as soon as we rename then there is a load of renaming in
reverse dependencies too, and there is presumably a set of rules about
when conflicts are added and other details that need to be got right.
I presume there is a good reason for this distinction between 'minor'
transitions and 'mega' transitions. I'd like to understand it properly.
> > Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
> > the ABIs of C libraries to accurately identify what needs to change for
> > time64_t. I think it would be interesting to know from this how many shared
> > libraries expose time_t size in their ABIs.
>
> Well, I didn't see anybody else expressing interest in this, so I had a go.
>
> I've done a first-pass analysis of Ubuntu lunar/armhf using
> abi-compliance-checker. The results of my investigation, including scripts
> and detailed reports/logs, are here:
Thanks very much for this. Really helpful. I tried to use abi-dumper
and abi-compliance-checker but wasn't having much success in getting
useful reports out of them. I found abigail-tools to be quite a lot
simpler to use. Probably we should run both and investigat if they
disagree about differences.
I found out about your effort just in time to include it in my FOSDEM talk,
which was good.
https://fosdem.org/2023/schedule/event/fixing_2038/ for anyone who is
unaware. (The Q has some dropouts and didn't add all that much
useful info so you can stop at about 17:20 without missing much).
> - abi-compliance-checker was used to analyze headers of 767 library
> packages in the archive. Of those, 82 of 558 were successfully analyzed
> and identified as library packages that would need patched for the
> switch to 64-bit time_t to mark them as ABI-incompatible with the
> previous version on armhf.
>
> - an additional 209 packages could not be analyzed, because a-c-c failed
> to compile the headers using its invocation of gcc. Assuming a roughly
> equal distribution of ABI-changing vs non-ABI-changing libraries among
> these that have failed to compile, we'd be looking at around 113 of 767
> libraries that need changed.
>
> - if we decide the level of effort to patch this many libraries is
> worthwhile, to get an actual actionable list of libraries that need
> patched for this transition would require getting the compile failures
> down to zero. I did dig into the failures alphabetically from "a" to
> "libe"; most of these failures were resolved by one of three methods.
> - letting a-c-c know that this is a C library and to not try to compile
> it with gcc's C++ frontend;
> - excluding various headers that can't be included directly, either
> because they are internal headers that are included via other headers
> in the package, or because they have dependencies on other headers not
> in the archive and can't be compiled (and therefore can't affect the
> library ABI);
> - pulling in additional undeclared package dependencies.
> Even when applying these fixes, I still had 8 library packages whose
> headers I wasn't able to get to compile. So there's still quite a bit
> of work to do here.
>
>
abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
> This is good. One thing that I think has been missing from the discussion
> about armhf rebootstrap is the fact that we do have experience in Debian
> doing cross-cutting ABI transitions without having to change an
> architecture name. We've done this at least three times in Debian history:
> the libc5->libc6 transition ('g' suffix - which still remains today in
> libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
> double' transition ('ldbl' suffix, and an example of doing this for an ABI
> transition that didn't affect all architectures).
> The first of these didn't require analysis, we knew all shared libraries
> were affected (and Debian was much smaller then).
> The second required analysis but it was easy because the C++ ABI is very
> visible in ELF symbols.
> The third is the best analogue to the time_t situation, because it involved
> changing the size of a type that could appear not just in ABIs of C++
> libraries, but also C libraries.
> And we did manage to survive that intact!
> https://lists.debian.org/debian-devel/2007/05/msg01173.html
> Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
> the ABIs of C libraries to accurately identify what needs to change for
> time64_t. I think it would be interesting to know from this how many shared
> libraries expose time_t size in their ABIs.
Well, I didn't see anybody else expressing interest in this, so I had a go.
I've done a first-pass analysis of Ubuntu lunar/armhf using
abi-compliance-checker. The results of my investigation, including scripts
and detailed reports/logs, are here:
https://people.canonical.com/~vorlon/armhf-time_t/
Notes:
- abi-compliance-checker was used to analyze headers of 767 library
packages in the archive. Of those, 82 of 558 were successfully analyzed
and identified as library packages that would need patched for the
switch to 64-bit time_t to mark them as ABI-incompatible with the
previous version on armhf.
- an additional 209 packages could not be analyzed, because a-c-c failed
to compile the headers using its invocation of gcc. Assuming a roughly
equal distribution of ABI-changing vs non-ABI-changing libraries among
these that have failed to compile, we'd be looking at around 113 of 767
libraries that need changed.
- if we decide the level of effort to patch this many libraries is
worthwhile, to get an actual actionable list of libraries that need
patched for this transition would require getting the compile failures
down to zero. I did dig into the failures alphabetically from "a" to
"libe"; most of these failures were resolved by one of three methods.
- letting a-c-c know that this is a C library and to not try to compile
it with gcc's C++ frontend;
- excluding various headers that can't be included directly, either
because they are internal headers that are included via other headers
in the package, or because they have dependencies on other headers not
in the archive and can't be compiled (and therefore can't affect the
library ABI);
- pulling in additional undeclared package dependencies.
Even when applying these fixes, I still had 8 library packages whose
headers I wasn't able to get to compile. So there's still quite a bit
of work to do here.
- On this preliminary pass, I only included library packages that shipped
both headers and .so's, as this supposedly gives the best quality
results from a-c-c. To get a full list of ABI-breaking changes,
however, we need to include packages that ship only headers and not
.so's, to catch those libraries for which the .so is in a different
binary package (boost), and to catch plugin systems for executables
(apache).
While I'm currently working on this using Ubuntu, the results are largely
applicable to Debian, and the script itself certainly is. Note however that
there are a couple of bugs in abi-compliance-checker that I've patched in
Ubuntu, bug #951076 and bug #1030540, which you would want to have fixed
before trying to run this analysis in Debian. Cc:ing the a-c-c maintainer
for awareness of this.
Does doing an ABI transition of ~113 libraries seem tractable to folks? If
there's interest in moving forward on this, I'd say the next step should be
to take it to debian-devel for broader discussion/signoff. I would also
move my scripts into a git repository that folks can contribute to -- adding
the necessary overrides for a-c-c for each library, so we can get an
authoritative list of ABI breakages, is very parallelizable.
Cheers,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slanga...@ubuntu.com
Re: another attempt at Y2038
Hi, On 2022-10-18 12:48, Helmut Grohne wrote: > Attempt #2: reproducible > > My other favourite QA tool is reproducible builds. So how about building > and comparing binary packages? > > Unfortunately, build flags participate in build ids. That directly > implies a high risk of false positives and/or manual work. > > Without further ado, here we go. > > .debs unchanged: > * base-files > * db-defaults > * coreutils coreutils 9.1 enables y2038 support by default if available, so this is actually expected. Regards Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net
Re: another attempt at Y2038
Hi Helmut, all,
On Tue, Oct 18, 2022 at 12:48:25PM +0200, Helmut Grohne wrote:
> Then the libprelude build failed with symbol issues. It happens to build
> a C++ library and its symbols are dependent on time types. Unsurprising.
> This is a case where we do break ABI.
> Then it occured to me that not failing does not imply not being affected
> right? You can break ABI without failing the build. So as a detection
> method this has a significant risk of false negatives.
> For just bootstrapping an incompatible time64 armhf, this method should
> mostly just work. Would someone like to have the resulting .debs? What
> would do with them?
This is good. One thing that I think has been missing from the discussion
about armhf rebootstrap is the fact that we do have experience in Debian
doing cross-cutting ABI transitions without having to change an
architecture name. We've done this at least three times in Debian history:
the libc5->libc6 transition ('g' suffix - which still remains today in
libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
double' transition ('ldbl' suffix, and an example of doing this for an ABI
transition that didn't affect all architectures).
The first of these didn't require analysis, we knew all shared libraries
were affected (and Debian was much smaller then).
The second required analysis but it was easy because the C++ ABI is very
visible in ELF symbols.
The third is the best analogue to the time_t situation, because it involved
changing the size of a type that could appear not just in ABIs of C++
libraries, but also C libraries.
And we did manage to survive that intact!
https://lists.debian.org/debian-devel/2007/05/msg01173.html
Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
the ABIs of C libraries to accurately identify what needs to change for
time64_t. I think it would be interesting to know from this how many shared
libraries expose time_t size in their ABIs.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: PGP signature
Re: another attempt at Y2038
Hi, Arnd Bergmann wrote: > This would be the correct list for the cdrom driver patches, > my list above would be for the isofs time64 patch. Good to know. I'll keep both lists in my cheat sheet. (I guess that y2...@lists.linaro.org is not appropriate for non-Y2038 patches.) > It's generally ok to just use the latest -rc1 (right now 6.1-rc1) Understood. (Two minutes after i sent my latest mail i got yours.) I wrote: > > If you want to do us both a favor then bring the changes from > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38 > > into the kernel. > Done now. Thank you. Also thanks to the bystanders for their tolerance. At least my hijacking of this thread was a successful attempt at Y2038. Have a nice day :) Thomas
Re: another attempt at Y2038
Hi, Arnd Bergmann wrote: > > Can you rebase the patch on top of v6.1-rc1 I wrote: > What kernel branches should i choose for sr and for isofs ? Ok, that question was probably stupid. May i understand your advise as "on top of the newest vX.Y-rcZ in github.com/torvalds/linux" ? (Given that my rebase attempts will not happen quite now.) Have a nice day :) Thomas
Re: another attempt at Y2038
On Thu, Oct 20, 2022, at 16:45, Thomas Schmitt wrote:
>> I think the hard part here is knowing who to send the patches to.
>> Unmaintained file systems are particularly tricky, in this case I
>> would have used
>>
>> To: Alexander Viro
>> To: Jan Kara
>> To: Andrew Morton
>> Cc: Arnd Bergmann
>> Cc: Deepa Dinamani
>> Cc: linux-fsde...@vger.kernel.org
>> Cc: linux-ker...@vger.kernel.org
>> Cc: y2...@lists.linaro.org
>
> Well, that's quite disjoint from the list which i figured out:
> linux-ker...@vger.kernel.org
> linux-s...@vger.kernel.org
> ax...@kernel.dk
> j...@linux.ibm.com
> martin.peter...@oracle.com
>
This would be the correct list for the cdrom driver patches,
my list above would be for the isofs time64 patch.
>> Can you rebase the patch on top of v6.1-rc1 and send it to
>> this list of people?
>
> I know the word "rebase" but cannot promise that i can fill it with
> substance soon ...
>
> What kernel branches should i choose for sr and for isofs ?
It's generally ok to just use the latest -rc1 (right now 6.1-rc1)
as the base, unless a maintainer asks you to use their tree
as a base.
>> I mainly care about the y2038 issue here,
>
> If you want to do us both a favor then bring the changes from
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38
> into the kernel.
> Feel free to take my reasoning and demonstration text. The code change is
> trivial.
Done now. I've left you as author and first Signed-off-by
though. While the change itself is trivial, the important bit
is identifying the problem, and you did that.
> It might be worth to verify my claims:
>
> The only callers of iso_date() are in isofs/inode.c and isofs/rock.c
> and put the result into struct inode.i_{a,c,m}time.tv_sec which is
> of type time64_t.
> The time value of iso_date() essentially stems from mktime64().
>
> and to exercise the demonstration by a xorriso-made ISO.
I could immediately tell that your patch is correct when I saw it.
Arnd
Re: another attempt at Y2038
Hi,
i wrote:
> > https://lore.kernel.org/linux-scsi/20201120140633.1673-1-scdbac...@gmx.net/T/
> > https://lore.kernel.org/linux-scsi/20201006094026.1730-1-scdbac...@gmx.net/T/
Arnd Bergmann wrote:
> These look like you did everything right, and they should have been
> picked up by the scsi maintainers.
I thought the same, but also that insisting, screaming, or trampling
won't help.
> If you ever re-send them, I would
> suggest putting the maintainers in 'To:' for the email and the mailing
> list in Cc,
I remember to have looked into the MAINTAINERS file and the git log for
names.
(In the case of the powerpc64 Oops, it looks like i added you in Cc:.
If i only could remember why ...)
> I think the hard part here is knowing who to send the patches to.
> Unmaintained file systems are particularly tricky, in this case I
> would have used
>
> To: Alexander Viro
> To: Jan Kara
> To: Andrew Morton
> Cc: Arnd Bergmann
> Cc: Deepa Dinamani
> Cc: linux-fsde...@vger.kernel.org
> Cc: linux-ker...@vger.kernel.org
> Cc: y2...@lists.linaro.org
Well, that's quite disjoint from the list which i figured out:
linux-ker...@vger.kernel.org
linux-s...@vger.kernel.org
ax...@kernel.dk
j...@linux.ibm.com
martin.peter...@oracle.com
> Can you rebase the patch on top of v6.1-rc1 and send it to
> this list of people?
I know the word "rebase" but cannot promise that i can fill it with
substance soon ...
What kernel branches should i choose for sr and for isofs ?
> I mainly care about the y2038 issue here,
If you want to do us both a favor then bring the changes from
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38
into the kernel.
Feel free to take my reasoning and demonstration text. The code change is
trivial. So if at all, i would be fully recognized by "Suggested-by:".
It might be worth to verify my claims:
The only callers of iso_date() are in isofs/inode.c and isofs/rock.c
and put the result into struct inode.i_{a,c,m}time.tv_sec which is
of type time64_t.
The time value of iso_date() essentially stems from mktime64().
and to exercise the demonstration by a xorriso-made ISO.
Have a nice day :)
Thomas
Re: another attempt at Y2038
On Thu, Oct 20, 2022, at 12:06, Thomas Schmitt wrote: > Arnd Bergmann wrote: > > > https://lore.kernel.org/linux-scsi/20201120140633.1673-1-scdbac...@gmx.net/T/ > [PATCH] isofs: fix Oops with zisofs and large PAGE_SIZE > > > https://lore.kernel.org/linux-scsi/20201006094026.1730-1-scdbac...@gmx.net/T/ > [PATCH v2 0/2] Fix automatic CD tray loading for data reading via sr > [PATCH v2 1/2] cdrom: delegate automatic CD tray loading to callers of > cdrom_open() > [PATCH v2 2/2] sr: fix automatic tray loading for data reading > > I made more bug fixes and a wishlist patch two years ago. > But keeping them up to date with the agile kernel development is quite a > big task for me. (As said: userlander.) These look like you did everything right, and they should have been picked up by the scsi maintainers. If you ever re-send them, I would suggest putting the maintainers in 'To:' for the email and the mailing list in Cc, and asking them specifically to pick up the patches. > Especially fs/isofs has no maintainer, so i could only submit to linux-scsi > because of the proximity to cdrom and sr. I had hoped that above two > patches would be considered as modest self-introduction, but obviously my > social skills are not sufficient. I think the hard part here is knowing who to send the patches to. Unmaintained file systems are particularly tricky, in this case I would have used To: Alexander Viro To: Jan Kara To: Andrew Morton Cc: Arnd Bergmann Cc: Deepa Dinamani Cc: linux-fsde...@vger.kernel.org Cc: linux-ker...@vger.kernel.org Cc: y2...@lists.linaro.org Can you rebase the patch on top of v6.1-rc1 and send it to this list of people? I'll reply with a 'Reviewed-by' tag, and one of the above should be able to pick it up. > This is what i think needed mending in kernel 5.10 two years ago: > > [PATCH 0/3] Fix the old CD read-ahead bug for media with a single TAO track > (Most drives report the 2 unreadable TAO run-out blocks as >part of the medium capacity.) > > [PATCH 0/1] sr: verify that last_written block is readable before deriving > size from it > (Size assessment of optical media in Linux is quite a mess >and can overshoot beyond the TAO run-out problem.) > > [PATCH 0/4] Attribute size 0 to sr device if no readable medium is loaded > (Linux reports the size of the last loaded medium, or 2048 >bytes if a blank medium is inserted.) > > [PATCH 0/4] Make mount -t iso9660 -o session=N work on DVD and BD media up > to N=168 > (While -o sbsector= works for all media types, session= works >only with CD media.) > > [PATCH 0/1] isofs: truncate oversized Rock Ridge names to 255 bytes > (Truncation currently happens if name length is >= 254 and >then cuts off much more of the name than needed.) > > When those bugs would be fixed (or mitigated in case of TAO), i hoped to > get a favor for my own hobby: > > [PATCH 0/3] Introduce a new ioctl CDROM_SIMUL_CHANGE for burn programs > (Currently Linux knows about new content created via ioctl >SG_IO only after eject and reload of the Medium.) > > The housekeeping aspects of kernel development are really hard for me to > master. I don't strive to become a regular contributor. But seeing those > bugs since years causes me to mention them when there is hope to meet > kernel developers. > > So: > Thanks a lot for replying. Is there a chance to get you interested in the > other bugs above and maybe even my wish for ioctl CDROM_SIMUL_CHANGE ? I mainly care about the y2038 issue here, haven't done anything interesting on cdrom drivers in 20 years ;-) Arnd
Re: another attempt at Y2038
Hi, Arnd Bergmann wrote: > That's just a trivially fixed bug, right? Yes. Very simple. Just s/int/time64_t/ . > I don't recall ever seeing a bug report or a patch for it in the past. I did not submit it to LKML or linux-scsi, because i got no reply to earlier attempts to bring in my userland knowledge about ISO 9660 and optical drives: https://lore.kernel.org/linux-scsi/20201120140633.1673-1-scdbac...@gmx.net/T/ [PATCH] isofs: fix Oops with zisofs and large PAGE_SIZE https://lore.kernel.org/linux-scsi/20201006094026.1730-1-scdbac...@gmx.net/T/ [PATCH v2 0/2] Fix automatic CD tray loading for data reading via sr [PATCH v2 1/2] cdrom: delegate automatic CD tray loading to callers of cdrom_open() [PATCH v2 2/2] sr: fix automatic tray loading for data reading I made more bug fixes and a wishlist patch two years ago. But keeping them up to date with the agile kernel development is quite a big task for me. (As said: userlander.) Especially fs/isofs has no maintainer, so i could only submit to linux-scsi because of the proximity to cdrom and sr. I had hoped that above two patches would be considered as modest self-introduction, but obviously my social skills are not sufficient. kernelnewbies could not help me either. I am still subscribed there in the hope to find allies. At one occasion i saw a chance when Adverg Ebashinskii asked for low hanging fruit in fs: https://www.mail-archive.com/search?q=Kernel+bug+tracker=kernelnewbies%40kernelnewbies.org I proposed the Y2038 patch: https://www.mail-archive.com/kernelnewbies@kernelnewbies.org/msg21629.html with the offer to reduce my "Signed-off-by:" to "Suggested-by:". This is what i think needed mending in kernel 5.10 two years ago: [PATCH 0/3] Fix the old CD read-ahead bug for media with a single TAO track (Most drives report the 2 unreadable TAO run-out blocks as part of the medium capacity.) [PATCH 0/1] sr: verify that last_written block is readable before deriving size from it (Size assessment of optical media in Linux is quite a mess and can overshoot beyond the TAO run-out problem.) [PATCH 0/4] Attribute size 0 to sr device if no readable medium is loaded (Linux reports the size of the last loaded medium, or 2048 bytes if a blank medium is inserted.) [PATCH 0/4] Make mount -t iso9660 -o session=N work on DVD and BD media up to N=168 (While -o sbsector= works for all media types, session= works only with CD media.) [PATCH 0/1] isofs: truncate oversized Rock Ridge names to 255 bytes (Truncation currently happens if name length is >= 254 and then cuts off much more of the name than needed.) When those bugs would be fixed (or mitigated in case of TAO), i hoped to get a favor for my own hobby: [PATCH 0/3] Introduce a new ioctl CDROM_SIMUL_CHANGE for burn programs (Currently Linux knows about new content created via ioctl SG_IO only after eject and reload of the Medium.) The housekeeping aspects of kernel development are really hard for me to master. I don't strive to become a regular contributor. But seeing those bugs since years causes me to mention them when there is hope to meet kernel developers. So: Thanks a lot for replying. Is there a chance to get you interested in the other bugs above and maybe even my wish for ioctl CDROM_SIMUL_CHANGE ? Have a nice day :) Thomas
Re: another attempt at Y2038
On Thu, Oct 20, 2022, at 10:57, Arnd Bergmann wrote: > On Tue, Oct 18, 2022, at 16:41, Thomas Schmitt wrote: >> >> Currently it's still "int iso_date()" in: >> https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h line 109 >> https://github.com/torvalds/linux/blob/master/fs/isofs/util.c line 19 >> >> >> Have a nice day :) > > That's just a trivially fixed bug, right? I don't recall > ever seeing a bug report or a patch for it in the past. Would > you like to send the patch, or should I? I see you actually put the patch into the debian bug tracker, though I can't find it on lore.kernel.org. Where did you submit it? Arnd
Re: another attempt at Y2038
On Tue, Oct 18, 2022, at 16:41, Thomas Schmitt wrote: > Hi, > > Steve McIntyre wrote: >> likely to still be in routine use beyond 2038 > > Sidenote towards ISO 9660 image producers: > > Don't forget to check from time to time whether Linux removed the > int bottleneck in fs/isofs/. > See: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38 > > The bug report was originally about a Y2028 rollover caused by 8-bit > signedness. It was closed after > https://github.com/torvalds/linux/commit/34be4dbf87fc > which left the int problem for future adventures. > > Currently it's still "int iso_date()" in: > https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h line 109 > https://github.com/torvalds/linux/blob/master/fs/isofs/util.c line 19 > > > Have a nice day :) That's just a trivially fixed bug, right? I don't recall ever seeing a bug report or a patch for it in the past. Would you like to send the patch, or should I? Arnd
Re: another attempt at Y2038
On 2022-10-18, Arnd Bergmann wrote: > Just for my understanding: what would be the correct way to handle this, > under the assumption that the new symbol names that dpkg-shlibdeps > found are the ones we want? Is there a way to flag both the time32 > and the tim64 set of symbols as correct, or would we have to have a > per-architecture list of expected symbols for the new (still to be > decided) architecture name? This is pretty normal for c++ libraries to have either symbol files depending on the architecture or symbol files or symbol somehow templated and filled out at build time (e.g. pkgkde-symbolshelper) /Sune
Re: another attempt at Y2038
Hi, Steve McIntyre wrote: > likely to still be in routine use beyond 2038 Sidenote towards ISO 9660 image producers: Don't forget to check from time to time whether Linux removed the int bottleneck in fs/isofs/. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38 The bug report was originally about a Y2028 rollover caused by 8-bit signedness. It was closed after https://github.com/torvalds/linux/commit/34be4dbf87fc which left the int problem for future adventures. Currently it's still "int iso_date()" in: https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h line 109 https://github.com/torvalds/linux/blob/master/fs/isofs/util.c line 19 Have a nice day :) Thomas
Re: another attempt at Y2038
On 10/18/22 15:47, Steve McIntyre wrote: I don't want to be rude, but I really don't see how m68k fits here. Of course, feel free to do a rebootstrap if you like, but I genuinely don't see any great need for it. I don't see the need for starting another "Why does Debian Ports exist?" discussion again. If you want to discuss about what's needed, probably, a lot of open source projects would just have to go away. ¯\_(ツ)_/¯ People just enjoy hacking code. There doesn't always have to have a commercial interest and a bigger reason behind it and that's perfectly fine. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: another attempt at Y2038
On Tue, Oct 18, 2022 at 03:33:17PM +0200, John Paul Adrian Glaubitz wrote: >Hi Helmut! > >On 10/18/22 12:48, Helmut Grohne wrote: >> I was also wondering about this Y2038 thingy and did some experiments. >> I'm reporting what I found to document it, but don't see much actionable >> stuff here. Many thanks to Arnd Bergmann for his input. >> >> Attempt #1: rebootstrap >> >> Given that I develop rebootstrap, attempting to use it for a time64 >> bootstrap seemed quite natural. I've been talking to this with Steve >> multiple times including DC22. The question was how to plug it in. In >> the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to >> modify dpkg-buildflags. Not every package uses these flags, but a >> majority do. For a survey, this is probably good enough. > >I would love to do that for m68k as well. We could use this opportunity to >rebuild the m68k port with 32-bit alignment which would solve quite a number >of problems since many projects like LLVM and Qt assume a minimum alignment >of 32 bits while m68k still defaults to 16 bits. > >Since the time64 rebootstrap would break the ABI anyway, we could use to fix >alignment issue on m68k once and for all :-). So the reason that we're talking about doing a replacement armhf port is for two reasons: * it's probably one of the few 32-bit arches that is likely to still be in routine use beyond 2038 (*many* embedded users depending on Debian stuff) * it's feasible to rebootstrap and break ABI as there isn't a large corpus of older-ABI binaries that people will care about supporting in the future. (This rules out i386 - the main reason to keep it around is just for the older binaries AFAICS.) I don't want to be rude, but I really don't see how m68k fits here. Of course, feel free to do a rebootstrap if you like, but I genuinely don't see any great need for it. -- Steve McIntyre, Cambridge, UK.st...@einval.com “Changing random stuff until your program works is bad coding practice, but if you do it fast enough it’s Machine Learning.” -- https://twitter.com/manisha72617183
Re: another attempt at Y2038
Hi Helmut! On 10/18/22 12:48, Helmut Grohne wrote: I was also wondering about this Y2038 thingy and did some experiments. I'm reporting what I found to document it, but don't see much actionable stuff here. Many thanks to Arnd Bergmann for his input. Attempt #1: rebootstrap Given that I develop rebootstrap, attempting to use it for a time64 bootstrap seemed quite natural. I've been talking to this with Steve multiple times including DC22. The question was how to plug it in. In the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to modify dpkg-buildflags. Not every package uses these flags, but a majority do. For a survey, this is probably good enough. I would love to do that for m68k as well. We could use this opportunity to rebuild the m68k port with 32-bit alignment which would solve quite a number of problems since many projects like LLVM and Qt assume a minimum alignment of 32 bits while m68k still defaults to 16 bits. Since the time64 rebootstrap would break the ABI anyway, we could use to fix alignment issue on m68k once and for all :-). Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: another attempt at Y2038
On Tue, Oct 18, 2022, at 12:48 PM, Helmut Grohne wrote: > Hi, > > I was also wondering about this Y2038 thingy and did some experiments. > I'm reporting what I found to document it, but don't see much actionable > stuff here. Many thanks to Arnd Bergmann for his input. Thanks a lot for taking a look here, this is very promising, compared to my attempt from 2020. > Attempt #1: rebootstrap > > Given that I develop rebootstrap, attempting to use it for a time64 > bootstrap seemed quite natural. I've been talking to this with Steve > multiple times including DC22. The question was how to plug it in. In > the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to > modify dpkg-buildflags. Not every package uses these flags, but a > majority do. For a survey, this is probably good enough. > > Things went somewhat far. The first issue was zlib. There isn't much to > say about it: https://github.com/madler/zlib/issues/447 > > Then the libprelude build failed with symbol issues. It happens to build > a C++ library and its symbols are dependent on time types. Unsurprising. > This is a case where we do break ABI. Just for my understanding: what would be the correct way to handle this, under the assumption that the new symbol names that dpkg-shlibdeps found are the ones we want? Is there a way to flag both the time32 and the tim64 set of symbols as correct, or would we have to have a per-architecture list of expected symbols for the new (still to be decided) architecture name? > Then it occured to me that not failing does not imply not being affected > right? You can break ABI without failing the build. So as a detection > method this has a significant risk of false negatives. > > For just bootstrapping an incompatible time64 armhf, this method should > mostly just work. Would someone like to have the resulting .debs? What > would do with them? I think Wookey wants to do a build of a much larger set of packages as a feasibility study to show that a rebuild actually survives running with time set to 2038+, before a decision is made about how and when this becomes an official release target. > Attempt #2: reproducible > > My other favourite QA tool is reproducible builds. So how about building > and comparing binary packages? > > Unfortunately, build flags participate in build ids. That directly > implies a high risk of false positives and/or manual work. > > Without further ado, here we go. > > .debs unchanged: > * base-files > * db-defaults > * coreutils > * diffutils > * findutils > * libonig > * libidn > * libidn2 > * libpipeline > * libpthread-stubs > * libsm > * gzip > > Question: How many of these ignore dpkg-buildflags? coreutils definitely should have changed, given that 'ls' prints timestamps. > After this sample, it became clear that this method is also not very > helpful. While it shows that basically everything is affected, it > doesn't give a good handle on ABI breakage either. I think it's safe to say that most packages will be affected in some way. Packages that only use glibc and no other libraries could be converted separately, but this is not helpful in the end since it does not get any closer to converting libraries that export modified interfaces. As far as I remember, I could show (using pattern matching on header files) that about half the libraries in Debian will require have an ABI change, but a much higher number of packages depend on at least one library aside from glibc that changes like this. To actually get this done, everything must be rebuilt, and in order to coexist with existing binaries, this requires a new architecture name and target triplet. Arnd
another attempt at Y2038
Hi, I was also wondering about this Y2038 thingy and did some experiments. I'm reporting what I found to document it, but don't see much actionable stuff here. Many thanks to Arnd Bergmann for his input. Attempt #1: rebootstrap Given that I develop rebootstrap, attempting to use it for a time64 bootstrap seemed quite natural. I've been talking to this with Steve multiple times including DC22. The question was how to plug it in. In the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to modify dpkg-buildflags. Not every package uses these flags, but a majority do. For a survey, this is probably good enough. Things went somewhat far. The first issue was zlib. There isn't much to say about it: https://github.com/madler/zlib/issues/447 Then the libprelude build failed with symbol issues. It happens to build a C++ library and its symbols are dependent on time types. Unsurprising. This is a case where we do break ABI. Then it occured to me that not failing does not imply not being affected right? You can break ABI without failing the build. So as a detection method this has a significant risk of false negatives. For just bootstrapping an incompatible time64 armhf, this method should mostly just work. Would someone like to have the resulting .debs? What would do with them? Attempt #2: reproducible My other favourite QA tool is reproducible builds. So how about building and comparing binary packages? Unfortunately, build flags participate in build ids. That directly implies a high risk of false positives and/or manual work. Without further ado, here we go. .debs unchanged: * base-files * db-defaults * coreutils * diffutils * findutils * libonig * libidn * libidn2 * libpipeline * libpthread-stubs * libsm * gzip Question: How many of these ignore dpkg-buildflags? minor changes: * gmp: build-id and #define __GMP_CFLAGS * guile-3.0: build-id and recorded build path unclear: * fribidi: only libfribidi-bin changes, probably harmless ftbfs: * zlib: see earlier starts using symbols such as __stat64_time64, __ioctl_time64: * acl * attr * base-passwd * bash * blt * bsdmainutils * bzip2 * c-ares * curl * dash * debianutils * dpkg * e2fsprogs * expat * file * flex * fontconfig * freetype * fuse * glib2.0 * gnupg2 * gpm * groff * hostname * icu * jansson * jemalloc * keyutils * kmod * krb5 * libassuan * libatomic-ops * libbsd * libcap2 * libdebian-installer * libev * libevent * libgc * libgcrypt20 * libice * libksba * libmd * libnsl * libpng1.6 * libsepol starts using large file symbols, but no time64 symbols: * libffi starts using symbols such as above, but also changes C++ symbols: * apt unreproducible for other non-time reasons: * binutils After this sample, it became clear that this method is also not very helpful. While it shows that basically everything is affected, it doesn't give a good handle on ABI breakage either. Please Cc me in replies as I am not subscribed. Helmut
