Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Richard W.M. Jones]

On Mon, Jan 30, 2023 at 02:09:18PM +0100, Petr Pisar wrote:
> V Sat, Jan 28, 2023 at 06:12:11PM -0800, Gordon Messmer napsal(a):
> > 
> > I don't think convincing hundreds or thousands of developers to add symbol
> > versioning to their libraries is a viable solution. I'd love to see it
> > happen, but rpm/dnf should be more reliable in the meantime.
> > 
> Is symbol versioning portable across toolchains? I.e. is the linker script
> with symbol aliases understandible by other tools than (llvm|gcc)+binutils?
> I speak about upstreams who do not want to impede building e.g. on FreeBSD,
> Windows, or Solaris.

It only applies to ELF platforms.  It also works fine with Clang.

FreeBSD uses ELF but their linker has issues with wildcards in the
linker script so we had to disable it there:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=190851

macOS doesn't use ELF and doesn't support linker scripts, so obviously
disabled there too.

Windows (PE) linking has entirely its own set of complexities and
strangeness, but I don't think there's anything quite like symbol
versioning.

Symbol versioning makes it tricky to backport an upstream patch to
downstream (eg. to RHEL) if the patch adds a new API.  You have to
decide whether or not to add a new version to the linker script or put
the symbol into one of the old sections, and I seem to recollect there
is no always good answer.

Nevertheless I think symbol versioning is a worthwhile thing to do.

Some examples of projects using linker scripts and/or symbol
versioning:

https://gitlab.com/nbdkit/nbdkit
 - linker script only, to hide symbols

https://gitlab.com/nbdkit/libnbd
 - linker script + versioning, everything is generated from a
   machine-readable description of the API

https://gitlab.com/libvirt/libvirt
 - extensive use of linker scripts + symbol versioning, see the src/*.syms
   files

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Florian Weimer]

* Petr Pisar:

> V Sat, Jan 28, 2023 at 06:12:11PM -0800, Gordon Messmer napsal(a):
>> 
>> I don't think convincing hundreds or thousands of developers to add symbol
>> versioning to their libraries is a viable solution. I'd love to see it
>> happen, but rpm/dnf should be more reliable in the meantime.

> Is symbol versioning portable across toolchains? I.e. is the linker
> script with symbol aliases understandible by other tools than
> (llvm|gcc)+binutils?  I speak about upstreams who do not want to
> impede building e.g. on FreeBSD, Windows, or Solaris.

The GNU variant of symbol versioning is GNU-specific.  Solaris symbol
versioning is quite different, but maybe sufficient in this context for
dependency generation.  Windows doesn't have anything like it, as far as
I know.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Neal Gompa]

On Mon, Jan 30, 2023 at 1:27 PM Richard W.M. Jones  wrote:
>
> On Sat, Jan 28, 2023 at 02:14:58AM -0600, Bruno Wolff III wrote:
> > On Fri, Jan 27, 2023 at 18:43:56 -0800,
> >  Gordon Messmer  wrote:
> > >
> > >Second, I'd like to suggest that in the future, at least in
> > >Fedora, for any "install" or "update" operation that dnf performs,
> > >dnf's default behavior should be checking all of the direct and
> > >indirect dependencies of the packages being installed (or updated)
> > >and updating any dependencies which have updates available.
> > >
> > >Does anyone else have any opinions on the subject?  Should I
> > >simply file a bug against dnf proposing this behavior?
> >
> > If there is a problem with not uodating dependencies when you do an
> > install or an update on selected packages, the packages dependencies
> > are not properly defined.
>
> You're correct, but ...
>
> It's quite often the case that updates fail because of a missing
> symbol in some library that wasn't updated.  I think this even happens
> if you're using symbol versioning, but in any case many libraries
> don't use symbol versioning[1].
>
> Gordon's suggestion is a pretty good one for fixing this common
> problem without making developers add full ">= version" dependencies
> to every BuildRequires line in every package they maintain.
>

This breaks down as soon as you talk about swapping library
implementations. The most recent example of this was SDL ->
sdl12-compat. Initially, the latter's version scheme didn't match the
former at all. If we had this, then replacing the library would have
been more painful to do.



-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Petr Pisar]

V Sat, Jan 28, 2023 at 06:12:11PM -0800, Gordon Messmer napsal(a):
> 
> I don't think convincing hundreds or thousands of developers to add symbol
> versioning to their libraries is a viable solution. I'd love to see it
> happen, but rpm/dnf should be more reliable in the meantime.
> 
Is symbol versioning portable across toolchains? I.e. is the linker script
with symbol aliases understandible by other tools than (llvm|gcc)+binutils?
I speak about upstreams who do not want to impede building e.g. on FreeBSD,
Windows, or Solaris.

-- Petr


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Richard W.M. Jones]

On Sat, Jan 28, 2023 at 02:14:58AM -0600, Bruno Wolff III wrote:
> On Fri, Jan 27, 2023 at 18:43:56 -0800,
>  Gordon Messmer  wrote:
> >
> >Second, I'd like to suggest that in the future, at least in
> >Fedora, for any "install" or "update" operation that dnf performs,
> >dnf's default behavior should be checking all of the direct and
> >indirect dependencies of the packages being installed (or updated)
> >and updating any dependencies which have updates available.
> >
> >Does anyone else have any opinions on the subject?  Should I
> >simply file a bug against dnf proposing this behavior?
> 
> If there is a problem with not uodating dependencies when you do an
> install or an update on selected packages, the packages dependencies
> are not properly defined.

You're correct, but ...

It's quite often the case that updates fail because of a missing
symbol in some library that wasn't updated.  I think this even happens
if you're using symbol versioning, but in any case many libraries
don't use symbol versioning[1].

Gordon's suggestion is a pretty good one for fixing this common
problem without making developers add full ">= version" dependencies
to every BuildRequires line in every package they maintain.

> I think the case where you don't want to keep the full system up to
> date, but a selective update or install causes problems as well is
> pretty rare. I think it might be reasonable to have an option that
> requests doing a recursive update. I would consider this to be a low
> priority feature request that has to compete with all of the other
> work being done on dnf, rather than a bug. I don't work on dnf and
> the people that do might feel differently.

Not really that rare.  I do this all the time.  Especially useful when
keeping a system "mainly" on a Fedora release, but selectively testing
Rawhide packages.

Rich.

[1] https://www.berrange.com/posts/2011/01/13/versioning-in-the-libvirt-library/

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Florian Weimer]

* Gordon Messmer:

> On 2023-01-28 13:03, Zbigniew Jędrzejewski-Szmek wrote:
>>> ...or "(libfoo.so.2()(64bit) with foo-libs >= x.y.z)", where x.y.z is the
>>> version of the package that provides libfoo.so.2 in the build root, which is
>>> an idea that's growing on me.
>> This is indeed a shortcoming in the rpm symbol dependency generation scheme.
>> But there's a problem with the proposed approach: versioning as 
>> major.minor.micro
>> is just a convention, and not all upstream follow it.
>
>
> The version doesn't have to be major.minor.micro, though.  The
> dependency generator only needs to get the version of the package that
> provides the .so, and use that version.  As long as changes within a
> so version are always backward compatible, and the so version is
> bumped when breaking changes are made (which are already constraints
> on the existing system), then the proposed solution is reliable.  It's
> sometimes too strict -- it might require 1.4.3 when 1.4.0 is
> compatible -- but it's reliable.

We already do this for glibc in rawhide, for the symbol version under
development.

# Auto-generating dependencies for glibc development snapshots.
#
# A glibc development snapshot (say version 2.33.9000) may define
# symbols in its under-development symbol version (GLIBC_2.34).  RPM
# automatically derives RPM dependencies such as
# libc.so.6(GLIBC_2.34)(64bit) from that.  While the GLIBC_2.34
# version is under development, these dependencies may be inaccurate
# and could be satisfied by glibc RPM package versions that lack the
# symbols because they were created from an earlier development
# snapshot that had some other GLIBC_2.34 symbols.  Therefore, if the
# latest, under-development ELF symbol version is detected, this
# dependency generator adds an explicit RPM dependencies on the glibc
# packaging version against which an RPM package is built.



This is an example build:

  util-linux-2.38.1-1.fc37
  

(util-linux-core in particular.)

Right now, this isn't used much, but it when it is, it causes issues for
developers who use the rawhide compose in chroots and newer builds leak
into the chroot from the buildroot without a full chroot update.
Manually ELN tagging can cause similar issues, so we no longer embed the
dist tag in the auto-generated versions.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Gordon Messmer]

On 2023-01-29 00:00, Zbigniew Jędrzejewski-Szmek wrote:

The version doesn't have to be major.minor.micro, though.  The dependency
generator only needs to get the version of the package that provides the
.so, and use that version.  As long as changes within a so version are
always backward compatible, and the so version is bumped when breaking
changes are made (which are already constraints on the existing system),
then the proposed solution is reliable.  It's sometimes too strict -- it
might require 1.4.3 when 1.4.0 is compatible -- but it's reliable.

OK, it sounds like something worth doing.



OK.  For discussion, I opened 
https://github.com/rpm-software-management/rpm/pull/2372


I tested it by rebuilding libsoup3 in a mock build root that included a 
version of rpm with (basically) that patch, and then installing the 
resulting package.  It looks like it functions correctly, so this seems 
like a sensible place to start.


$ podman run --rm -it fedora:37
[root@ae5c590fa03a /]# rpm -qp --requires 
/var/tmp/libsoup3-3.2.2-3.fc37.x86_64.rpm

(libnghttp2.so.14()(64bit) with libnghttp2 >= 1.51.0)
...
[root@ae5c590fa03a /]# dnf install 
/var/tmp/libsoup3-3.2.2-3.fc37.x86_64.rpm

Dependencies resolved.
===
 Package Architecture Version 
Repository  Size

===
Installing:
 libsoup3 x86_64 3.2.2-3.fc37 
@commandline   373 k

Upgrading:
 libnghttp2 x86_64 1.51.0-1.fc37 
updates 74 k

...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Zbigniew Jędrzejewski-Szmek]

On Sat, Jan 28, 2023 at 06:12:11PM -0800, Gordon Messmer wrote:
> On 2023-01-28 13:03, Zbigniew Jędrzejewski-Szmek wrote:
> > > ...or "(libfoo.so.2()(64bit) with foo-libs >= x.y.z)", where x.y.z is the
> > > version of the package that provides libfoo.so.2 in the build root, which 
> > > is
> > > an idea that's growing on me.
> > This is indeed a shortcoming in the rpm symbol dependency generation scheme.
> > But there's a problem with the proposed approach: versioning as 
> > major.minor.micro
> > is just a convention, and not all upstream follow it.
> 
> 
> The version doesn't have to be major.minor.micro, though.  The dependency
> generator only needs to get the version of the package that provides the
> .so, and use that version.  As long as changes within a so version are
> always backward compatible, and the so version is bumped when breaking
> changes are made (which are already constraints on the existing system),
> then the proposed solution is reliable.  It's sometimes too strict -- it
> might require 1.4.3 when 1.4.0 is compatible -- but it's reliable.

OK, it sounds like something worth doing.

(One issue that I expect that there'll be some libraries which do "strange
things" and may need different handling, so some opt-out mechanism would need
to be provided.)

> > There is an alternative scheme that is supported by our rpm tooling already:
> > symbol versioning.
> 
> 
> Yes, the enhanced rpm dependencies would only be necessary for libraries
> that don't provide versioned symbols (which seems to be the vast majority of
> them).
> 
> I don't think convincing hundreds or thousands of developers to add symbol
> versioning to their libraries is a viable solution. I'd love to see it
> happen, but rpm/dnf should be more reliable in the meantime.

Ack.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Zbigniew Jędrzejewski-Szmek]

On Sun, Jan 29, 2023 at 12:23:49AM +0100, Fabio Valentini wrote:
> On Sun, Jan 29, 2023 at 12:03 AM Zbigniew Jędrzejewski-Szmek
>  wrote:
> >
> > On Sat, Jan 28, 2023 at 01:49:17PM -0800, Kevin Fenzi wrote:
> > > On Sat, Jan 28, 2023 at 09:03:35PM +, Zbigniew Jędrzejewski-Szmek 
> > > wrote:
> > > >
> > > > This is indeed a shortcoming in the rpm symbol dependency generation 
> > > > scheme.
> > >
> > > Is it though? I'm probibly reading this too quickly and missing
> > > something, but isn't the underlying problem here that nghttp2 changed
> > > abi and didn't change soname? If they had, soup would have kept the
> > > older version around, or asked to upgrade both libsoup and libnghttp2?
> >
> > No. libnghttp2 added one symbol, and (correctly, i.e. following the usual
> > practice), it changed the suffix from .so.14.23.0 to .so.14.24.1.  (The last
> > number is bumped when the implementation of existing symbols changes, the 
> > middle
> > one when when new symbols are added, and the first one when an incompatible
> > change happens, i.e. symbols are removed or changed incompatibly. Lower 
> > numbers
> > are reset to zero whenever a higher one changes.)  The SOVERSION 
> > corresponds to
> > the major number and does not change in the case under discussion.
> >
> > As Gordon wrote, our rpm dependency generation logic does not care about the
> > minor or patch numbers.
> 
> If I understand things correctly, this is not entirely true. RPM
> generates a dependency for the soname / soversion, and some projects
> include not only X, but all of X.Y.Z in that, which RPM will happily
> generate Provides / Requires for (whether that's a nice thing to do
> for an upstream project or whether that's a bug in their build systems
> I leave as an exercise to the reader).

That is all true. But it's probably not a good idea to for the upstream
projects to do that, because the SONAME is used to load the library at runtime. 
(*)
If the SONAME changes, all programs and libraries linking to that library
need to be recompiled/relinked. This would essentially defeat the shared library
system.

(*) Or more precisely speaking, when a library is installed in the file system,
a symlink is created to the real name (e.g. libcurl.so.4.8.0) from the embedded
SONAME. Subsequently, a program that wants a given SONAME is run, the dynamic
linker looks for the SONAME in the file system and finds the symlink
(libcurl.so.4 in this case) and uses that. IIUC, the SONAME isn't actually
checked again. So SONAME is _not_ used at runtime directly, but it's still used
to set things up so that the library is found.

Earlier, when a program is being compiled, a similar indirection happens. The
compiler/linker is given a library name (e.g. -lcurl) and uses that to locate
the library as %{_libdir}/libcurl.so, reads the SONAME (libcurl.so.4) from that
file and saves the SONAME in the linked program.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Gordon Messmer]

On 2023-01-28 15:23, Fabio Valentini wrote:

If I understand things correctly, this is not entirely true. RPM
generates a dependency for the soname / soversion, and some projects
include not only X, but all of X.Y.Z in that, which RPM will happily
generate Provides / Requires for



$ ls -l /lib64/libnghttp2.so.14*
lrwxrwxrwx. 1 root root 21 Nov 21 05:45 /lib64/libnghttp2.so.14 -> 
libnghttp2.so.14.24.1*

-rwxr-xr-x. 1 root root 173696 Nov 21 05:45 /lib64/libnghttp2.so.14.24.1*

$ ldd /lib64/libsoup-3.0.so.0 | grep libnghttp
    libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x7f08fb366000)

$ rpm -q libsoup3 --requires | grep libnghttp2
libnghttp2.so.14()(64bit)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Gordon Messmer]

On 2023-01-28 13:03, Zbigniew Jędrzejewski-Szmek wrote:

...or "(libfoo.so.2()(64bit) with foo-libs >= x.y.z)", where x.y.z is the
version of the package that provides libfoo.so.2 in the build root, which is
an idea that's growing on me.

This is indeed a shortcoming in the rpm symbol dependency generation scheme.
But there's a problem with the proposed approach: versioning as 
major.minor.micro
is just a convention, and not all upstream follow it.



The version doesn't have to be major.minor.micro, though.  The 
dependency generator only needs to get the version of the package that 
provides the .so, and use that version.  As long as changes within a so 
version are always backward compatible, and the so version is bumped 
when breaking changes are made (which are already constraints on the 
existing system), then the proposed solution is reliable.  It's 
sometimes too strict -- it might require 1.4.3 when 1.4.0 is compatible 
-- but it's reliable.




There is an alternative scheme that is supported by our rpm tooling already:
symbol versioning.



Yes, the enhanced rpm dependencies would only be necessary for libraries 
that don't provide versioned symbols (which seems to be the vast 
majority of them).


I don't think convincing hundreds or thousands of developers to add 
symbol versioning to their libraries is a viable solution. I'd love to 
see it happen, but rpm/dnf should be more reliable in the meantime.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Fabio Valentini]

On Sun, Jan 29, 2023 at 12:03 AM Zbigniew Jędrzejewski-Szmek
 wrote:
>
> On Sat, Jan 28, 2023 at 01:49:17PM -0800, Kevin Fenzi wrote:
> > On Sat, Jan 28, 2023 at 09:03:35PM +, Zbigniew Jędrzejewski-Szmek wrote:
> > >
> > > This is indeed a shortcoming in the rpm symbol dependency generation 
> > > scheme.
> >
> > Is it though? I'm probibly reading this too quickly and missing
> > something, but isn't the underlying problem here that nghttp2 changed
> > abi and didn't change soname? If they had, soup would have kept the
> > older version around, or asked to upgrade both libsoup and libnghttp2?
>
> No. libnghttp2 added one symbol, and (correctly, i.e. following the usual
> practice), it changed the suffix from .so.14.23.0 to .so.14.24.1.  (The last
> number is bumped when the implementation of existing symbols changes, the 
> middle
> one when when new symbols are added, and the first one when an incompatible
> change happens, i.e. symbols are removed or changed incompatibly. Lower 
> numbers
> are reset to zero whenever a higher one changes.)  The SOVERSION corresponds 
> to
> the major number and does not change in the case under discussion.
>
> As Gordon wrote, our rpm dependency generation logic does not care about the
> minor or patch numbers.

If I understand things correctly, this is not entirely true. RPM
generates a dependency for the soname / soversion, and some projects
include not only X, but all of X.Y.Z in that, which RPM will happily
generate Provides / Requires for (whether that's a nice thing to do
for an upstream project or whether that's a bug in their build systems
I leave as an exercise to the reader).

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Zbigniew Jędrzejewski-Szmek]

On Sat, Jan 28, 2023 at 01:49:17PM -0800, Kevin Fenzi wrote:
> On Sat, Jan 28, 2023 at 09:03:35PM +, Zbigniew Jędrzejewski-Szmek wrote:
> > 
> > This is indeed a shortcoming in the rpm symbol dependency generation scheme.
> 
> Is it though? I'm probibly reading this too quickly and missing
> something, but isn't the underlying problem here that nghttp2 changed
> abi and didn't change soname? If they had, soup would have kept the
> older version around, or asked to upgrade both libsoup and libnghttp2?

No. libnghttp2 added one symbol, and (correctly, i.e. following the usual
practice), it changed the suffix from .so.14.23.0 to .so.14.24.1.  (The last
number is bumped when the implementation of existing symbols changes, the middle
one when when new symbols are added, and the first one when an incompatible
change happens, i.e. symbols are removed or changed incompatibly. Lower numbers
are reset to zero whenever a higher one changes.)  The SOVERSION corresponds to
the major number and does not change in the case under discussion.

As Gordon wrote, our rpm dependency generation logic does not care about the
minor or patch numbers.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Nico Kadel-Garcia]

On Sat, Jan 28, 2023 at 1:23 PM Gordon Messmer  wrote:
>
> On 2023-01-28 00:14, Bruno Wolff III wrote:
> > If there is a problem with not uodating dependencies when you do an
> > install or an update on selected packages, the packages dependencies
> > are not properly defined.
>
>
> By definition, yes.  But rpm auto-detects dependencies, and rpm doesn't
> do symbol-level dependency detection, so it doesn't capture
> minor-version dependency creep.  In order for rpm to do this, you'd
> probably have to throw out the current implementation of dependency
> resolution that provides "libfoo.so.2()(64bit)" and instead provide a
> dependency like "(foo-libs >= 2.4 with foo-libs < 3)", at least for the
> cases where libraries do not provide versioned symbols -- which I
> believe includes the vast majority of them.  (You'd also need to forbid
> restructuring packages within a stable release.)

The RPM build process tries to do so, but it can't possibly outsmart
the morass of dependencies that "modularity" tends to create. It's why
so many have given up on modularity since its introduction in Fedora,
and one of the reasons I have trouble getting approval to do RHEL 8
and RHEL 9 updates. The various sets of "modularity" components are
not tested against all the other versions of related packages: I'm
running into this headlong with ansible collections and the morass of
dependency chains weaving into and out of modularity deployed
coponents, myself. The "modular" channels are almost invariably
incomplete and require additional RPM builds to satisfy "test" or
"doc" building steps from the same base software package.

Modularity and its inevitable build-time and deployment
inconsistencies have consistently hindered dependency management.

> > I think the case where you don't want to keep the full system up to
> > date, but a selective update or install causes problems as well is
> > pretty rare. I think it might be reasonable to have an option that
> > requests doing a recursive update. I would consider this to be a low
> > priority feature request that has to compete with all of the other
> > work being done on dnf, rather than a bug. I don't work on dnf and the
> > people that do might feel differently.
>
>
> Yeah, I agree, it's not super high priority.  But it's also not really
> well understood among the community that partial updates (such as `dnf
> update --security`) and package installation without a contemporaneous
> update are unreliable.

Yeah. It's hard to predict exactly when it will break, but it *does*
break. Unfortunately, so do fork list upgrades, such as using a base
OS image that is three years old and then doing "dnf -y update" on it.
It's very difficult to predict which update will break what, but
updating 700 packages at once as I just saw on a base OS image of RHEL
8. It's one reason Fedora's frequent releases are useful: they provide
a much fresher, smoke tested baseline image to apply changes to,
rather than relying on upgrades in places.

> I can work on some of those changes to documentation and to rpm or dnf,
> but I'd like input from the developer community before starting.  (And
> at some point I think that Fedora, the org, should probably consider
> whether `dnf update --security` is broken and unreliable.)

It's popular. I'm remembering when the "dnf update -y --security"
broke curl, because either "curl-libs" or "crypto-policies" was not
listed in those updates, and hilarity ensued. It's why I try to
schedule regular "dnf -y update" operations, and updating my base OS
images to match. It can get a bit awkward when the base OS images at
your favorite cloud vendor fall increasingly out of date over time.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Kevin Fenzi]

On Sat, Jan 28, 2023 at 09:03:35PM +, Zbigniew Jędrzejewski-Szmek wrote:
> 
> This is indeed a shortcoming in the rpm symbol dependency generation scheme.

Is it though? I'm probibly reading this too quickly and missing
something, but isn't the underlying problem here that nghttp2 changed
abi and didn't change soname? If they had, soup would have kept the
older version around, or asked to upgrade both libsoup and libnghttp2?

Or did I miss some part of this?

...snip symbol versioning... 

symbol versioning does indeed sound like a better way to do things. :)

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Zbigniew Jędrzejewski-Szmek]

On Sat, Jan 28, 2023 at 12:12:40PM -0800, Gordon Messmer wrote:
> On  2023-01-28 10:22, Gordon Messmer wrote:
> > In order for rpm to do this, you'd probably have to throw out the
> > current implementation of dependency resolution that provides
> > "libfoo.so.2()(64bit)" and instead provide a dependency like "(foo-libs
> > >= 2.4 with foo-libs < 3)", at least for the cases where libraries do
> > not provide versioned symbols
> 
> ...or "(libfoo.so.2()(64bit) with foo-libs >= x.y.z)", where x.y.z is the
> version of the package that provides libfoo.so.2 in the build root, which is
> an idea that's growing on me.

This is indeed a shortcoming in the rpm symbol dependency generation scheme.
But there's a problem with the proposed approach: versioning as 
major.minor.micro
is just a convention, and not all upstream follow it.

There is an alternative scheme that is supported by our rpm tooling already:
symbol versioning. This is ultimately more powerful, because it allows depending
on specific versions of functions, and not just one monotonic version that
tries to describe anything. (glibc makes extensive use of this to provide
backwards comaptiblity…)

Since symbol versioning is more powerful and pretty well handled by rpm,
I think it's better to try to convince upstream to switch to that if they
change symbols a lot.

(Or even better, in my personal opinion, is to never change SOVERSION, i.e.
to never remove symbols. It is surprisingly easy to provide backwards compat,
for most project much less hassle than to deal with SOVERSION changes and
forced rebuilds.)

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Gordon Messmer]

On  2023-01-28 10:22, Gordon Messmer wrote:
In order for rpm to do this, you'd probably have to throw out the 
current implementation of dependency resolution that provides 
"libfoo.so.2()(64bit)" and instead provide a dependency like "(foo-libs 
 >= 2.4 with foo-libs < 3)", at least for the cases where libraries do 
not provide versioned symbols


...or "(libfoo.so.2()(64bit) with foo-libs >= x.y.z)", where x.y.z is 
the version of the package that provides libfoo.so.2 in the build root, 
which is an idea that's growing on me.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Gordon Messmer]

On 2023-01-28 00:14, Bruno Wolff III wrote:
If there is a problem with not uodating dependencies when you do an 
install or an update on selected packages, the packages dependencies 
are not properly defined.



By definition, yes.  But rpm auto-detects dependencies, and rpm doesn't 
do symbol-level dependency detection, so it doesn't capture 
minor-version dependency creep.  In order for rpm to do this, you'd 
probably have to throw out the current implementation of dependency 
resolution that provides "libfoo.so.2()(64bit)" and instead provide a 
dependency like "(foo-libs >= 2.4 with foo-libs < 3)", at least for the 
cases where libraries do not provide versioned symbols -- which I 
believe includes the vast majority of them.  (You'd also need to forbid 
restructuring packages within a stable release.)



I think the case where you don't want to keep the full system up to 
date, but a selective update or install causes problems as well is 
pretty rare. I think it might be reasonable to have an option that 
requests doing a recursive update. I would consider this to be a low 
priority feature request that has to compete with all of the other 
work being done on dnf, rather than a bug. I don't work on dnf and the 
people that do might feel differently. 



Yeah, I agree, it's not super high priority.  But it's also not really 
well understood among the community that partial updates (such as `dnf 
update --security`) and package installation without a contemporaneous 
update are unreliable.


I can work on some of those changes to documentation and to rpm or dnf, 
but I'd like input from the developer community before starting.  (And 
at some point I think that Fedora, the org, should probably consider 
whether `dnf update --security` is broken and unreliable.)

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Michael J Gruber]

libsoup3 depends on libnghttp2.so.14

Apparantly, either libsoup3 should depend on the minor version (in addition to 
the major version), or libnghhttp2 should have bumped major, depending on the 
"history" of that symbol. Probably the former (API addition in a minor bump to 
the same major).
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Bruno Wolff III]

On Fri, Jan 27, 2023 at 18:43:56 -0800,
 Gordon Messmer  wrote:


Second, I'd like to suggest that in the future, at least in Fedora, 
for any "install" or "update" operation that dnf performs, dnf's 
default behavior should be checking all of the direct and indirect 
dependencies of the packages being installed (or updated) and updating 
any dependencies which have updates available.


Does anyone else have any opinions on the subject?  Should I simply 
file a bug against dnf proposing this behavior?


If there is a problem with not uodating dependencies when you do an install 
or an update on selected packages, the packages dependencies are not 
properly defined.


I think the case where you don't want to keep the full system up to date, 
but a selective update or install causes problems as well is pretty rare. 
I think it might be reasonable to have an option that requests doing 
a recursive update. I would consider this to be a low priority feature 
request that has to compete with all of the other work being done on 
dnf, rather than a bug. I don't work on dnf and the people that do might 
feel differently.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Proposal: dnf should offer to update all of the dependencies of any package installed or updated

[Gordon Messmer]

I recently helped another user repair their Fedora workation, after an 
update broke gnome-shell.  In their case, I believe that the problem 
occurred because they had the nodejs:14 module enabled, which contained 
an outdated libnghttp2 [1], but in principle, the problem can affect any 
system that selectively applies updates, or any system on which packages 
are installed without also applying updates [2].


I expected that if I checked, I could find some mention in the Fedora 
documentation to the effect that the only supportable configuration for 
a major-version stable distribution is fully patched.  The only thing I 
could actually find, though, was an article about applying only security 
or bugfix updates [3], which is basically the opposite.


(As an aside, I *also* think that "dnf update --security" is broken on 
Fedora because not only can it result in broken dependencies, but it can 
be misleading.  In the case that a security update is published and 
subsequently obsoleted by another update, systems with an affected, 
older package would not be informed that a security update was required. 
 The second, subsequent update does include security fixes for older 
versions, but that information is lost.  Maybe Fedora should retain any 
package which was marked as a security fix?)


So, first, I will open a PR with some changes to Fedora's DNF 
documentation [4] encouraging users to apply all updates before 
installing packages.  I may follow up on the Fedora Magazine article, as 
well.  It doesn't feel right to propose an article on Fedora Magazine 
advising users not to use "dnf update --security", but it also doesn't 
feel right to publish articles describing that feature without also 
describing any caveats.


Second, I'd like to suggest that in the future, at least in Fedora, for 
any "install" or "update" operation that dnf performs, dnf's default 
behavior should be checking all of the direct and indirect dependencies 
of the packages being installed (or updated) and updating any 
dependencies which have updates available.


Does anyone else have any opinions on the subject?  Should I simply file 
a bug against dnf proposing this behavior?



1: https://bugzilla.redhat.com/show_bug.cgi?id=2164944

2: For example:
$ podman run --rm -it fedora:37
[root@30a0f8c3f6a8 /]# rpm -q libnghttp2
libnghttp2-1.49.0-1.fc37.x86_64
[root@30a0f8c3f6a8 /]# dnf install chezdav
...
[root@30a0f8c3f6a8 /]# chezdav
chezdav: symbol lookup error: /lib64/libsoup-3.0.so.0: undefined symbol: 
nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation

[root@30a0f8c3f6a8 /]# rpm -q libsoup3 libnghttp2
libsoup3-3.2.2-2.fc37.x86_64
libnghttp2-1.49.0-1.fc37.x86_64

3: 
https://fedoramagazine.org/how-to-install-only-security-and-bugfixes-updates-with-dnf/


4: 
https://docs.fedoraproject.org/en-US/fedora/latest/system-administrators-guide/package-management/DNF/

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue