Re: Eliminating automatic provides on private libs
Hi, On 04/29/2012 05:13 PM, Panu Matilainen wrote: On 04/27/2012 07:36 PM, Toshio Kuratomi wrote: On Fri, Apr 27, 2012 at 11:13:47AM +0300, Panu Matilainen wrote: I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. Will the toggle to turn it on and off work for those libtool generated modules (as long as there's not public libraries in the same package)? Is it possible to turn this on and off per subpackage or is that along the lines of asking for pretty pink ponies with wings? For now, I'm only talking about a switch that stops faking a soname provide when an ELF DSO does not provide one, nothing more. Also that switch is going to be a global macro to avoid spec syntax compatibility issues, at least for now. As for the other issues like libtool generated modules, the question is: why does libtool insist on creating a soname for them, and could we get libtool changed not to do that? Also it might be possible to come up with heuristics to detect (some of) such modules based on the data in elf headers. If we go the heuristics way, why not do heuristics based on the location of the library? If the .so sits directly in a standard libdir it is likely something other packages can depend on / link against. If it sits in a subdir it is likely not. You can extend this heuristic by adding dirs found in $RPM_BUILD_ROOT/etc/ld.so.conf.d/*.conf to the standard libdirs before running this heuristic. This should give a pretty good score both wrt filtering out unwanted provides as well as not having any false positives. Regards, Hans -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
On 04/29/2012 06:25 PM, Panu Matilainen wrote: On 04/29/2012 06:13 PM, Panu Matilainen wrote: On 04/27/2012 07:36 PM, Toshio Kuratomi wrote: On Fri, Apr 27, 2012 at 11:13:47AM +0300, Panu Matilainen wrote: I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. Will the toggle to turn it on and off work for those libtool generated modules (as long as there's not public libraries in the same package)? Is it possible to turn this on and off per subpackage or is that along the lines of asking for pretty pink ponies with wings? For now, I'm only talking about a switch that stops faking a soname provide when an ELF DSO does not provide one, nothing more. Also that switch is going to be a global macro to avoid spec syntax compatibility issues, at least for now. As for the other issues like libtool generated modules, the question is: why does libtool insist on creating a soname for them, and could we get libtool changed not to do that? Also it might be possible to come up with heuristics to detect (some of) such modules based on the data in elf headers. Actually... one painfully simple and obvious heuristic is the DSO basename: if it doesn't start with lib, it's almost certainly not a regular library. ...but then there's ld-*.so. Which of course is not your average regular library either, but requiring spec tweaks to get the necessary provides into place for such a fundamental piece of the system doesn't seem right either. One possibility that looks like it should weed out nearly all the private libs (plugins etc) whether created with libtool or not, and still handle ld-linux.so correctly by default: Only create soname provides if the DSO specificies specifies a SONAME, and the soname either starts with lib or is versioned. It's just that it's getting a bit thick in the heuristics department. Sigh... it would all be so simple if presence of SONAME could be used as a mostly reliable indicator for the DSO's public/private status :( - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
On 04/27/2012 07:36 PM, Toshio Kuratomi wrote: On Fri, Apr 27, 2012 at 11:13:47AM +0300, Panu Matilainen wrote: I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. Will the toggle to turn it on and off work for those libtool generated modules (as long as there's not public libraries in the same package)? Is it possible to turn this on and off per subpackage or is that along the lines of asking for pretty pink ponies with wings? For now, I'm only talking about a switch that stops faking a soname provide when an ELF DSO does not provide one, nothing more. Also that switch is going to be a global macro to avoid spec syntax compatibility issues, at least for now. As for the other issues like libtool generated modules, the question is: why does libtool insist on creating a soname for them, and could we get libtool changed not to do that? Also it might be possible to come up with heuristics to detect (some of) such modules based on the data in elf headers. Overall seems like a positive change to me. Let us know the details when it's changed and we'll add those to the Packaging Guidelines. Ok. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
On 04/29/2012 06:13 PM, Panu Matilainen wrote: On 04/27/2012 07:36 PM, Toshio Kuratomi wrote: On Fri, Apr 27, 2012 at 11:13:47AM +0300, Panu Matilainen wrote: I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. Will the toggle to turn it on and off work for those libtool generated modules (as long as there's not public libraries in the same package)? Is it possible to turn this on and off per subpackage or is that along the lines of asking for pretty pink ponies with wings? For now, I'm only talking about a switch that stops faking a soname provide when an ELF DSO does not provide one, nothing more. Also that switch is going to be a global macro to avoid spec syntax compatibility issues, at least for now. As for the other issues like libtool generated modules, the question is: why does libtool insist on creating a soname for them, and could we get libtool changed not to do that? Also it might be possible to come up with heuristics to detect (some of) such modules based on the data in elf headers. Actually... one painfully simple and obvious heuristic is the DSO basename: if it doesn't start with lib, it's almost certainly not a regular library. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Eliminating automatic provides on private libs
I started once again looking at ways to eliminate the unwanted provides on private libs such as dlopen()'ed modules with minimal fuss and breakage. Been down this route more than once but I suspect the last time was before the major dependency generator changes in rpm-4.9.x, which made tweaking these things a good deal easier, so time for another round: The short background is that for libraries which dont have a SONAME, rpmbuild fakes one based on the file name. The rationale for this has been that since the linker permits linking to libraries without sonames, rpm must permit this too. The downside is that for the 1% corner cases that need this, we end up polluting the provides namespace with thousands of unnecessary and unwanted names, names that have not been picked up with a global namespace in mind. To get a rough idea just how much of this unwanted junk exists, using a slightly modified elfdeps generator on my system: This is how we currently generate the provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only --fake-soname|wc -l 3875 And this is without faked soname provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only |wc -l 2816 So... about a third of the soname provides are faked by using library basename for its provide name, and almost all of them are for no good reason. The three general cases that something actually depends on are 1) libreoffice sub-packages depending on its internal libraries from libreoffice-core 2) a handful of packages depending on llvm libraries 3) bigger pile of packages depending on libperl.so Of these at least 2) does look like a bug really, and in fact has been fixed upstream: http://llvm.org/bugs/show_bug.cgi?id=12334. Dunno about 3), perl is... what it is. 1) might well classify as an inevitable special case (there will always be those), dunno. Obviously this is just a fairly small sample of the entire Fedora package set, there are bound to be many more special cases, buggy libraries which do not have SONAME when they should and stuff that falls into gray areas. I can try to provide more comprehensive stats based on current rawhide if needed. I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
On 04/27/2012 09:13 AM, Panu Matilainen wrote: I started once again looking at ways to eliminate the unwanted provides on private libs such as dlopen()'ed modules with minimal fuss and breakage. Been down this route more than once but I suspect the last time was before the major dependency generator changes in rpm-4.9.x, which made tweaking these things a good deal easier, so time for another round: The short background is that for libraries which dont have a SONAME, rpmbuild fakes one based on the file name. The rationale for this has been that since the linker permits linking to libraries without sonames, rpm must permit this too. The downside is that for the 1% corner cases that need this, we end up polluting the provides namespace with thousands of unnecessary and unwanted names, names that have not been picked up with a global namespace in mind. To get a rough idea just how much of this unwanted junk exists, using a slightly modified elfdeps generator on my system: This is how we currently generate the provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only --fake-soname|wc -l 3875 And this is without faked soname provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only |wc -l 2816 So... about a third of the soname provides are faked by using library basename for its provide name, and almost all of them are for no good reason. The three general cases that something actually depends on are 1) libreoffice sub-packages depending on its internal libraries from libreoffice-core 2) a handful of packages depending on llvm libraries 3) bigger pile of packages depending on libperl.so Of these at least 2) does look like a bug really, and in fact has been fixed upstream: http://llvm.org/bugs/show_bug.cgi?id=12334. Dunno about 3), perl is... what it is. 1) might well classify as an inevitable special case (there will always be those), dunno. Obviously this is just a fairly small sample of the entire Fedora package set, there are bound to be many more special cases, buggy libraries which do not have SONAME when they should and stuff that falls into gray areas. I can try to provide more comprehensive stats based on current rawhide if needed. I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. Regarding perl, of the 48 packages that require libperl.so in Rawhide, only 13 of them don't also require perl(:MODULE_COMPAT_5.14.2), which will also pull in the perl-libs package, so I don't think perl is a big issue. Paul. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
On 04/27/2012 11:25 AM, Paul Howarth wrote: On 04/27/2012 09:13 AM, Panu Matilainen wrote: I started once again looking at ways to eliminate the unwanted provides on private libs such as dlopen()'ed modules with minimal fuss and breakage. Been down this route more than once but I suspect the last time was before the major dependency generator changes in rpm-4.9.x, which made tweaking these things a good deal easier, so time for another round: The short background is that for libraries which dont have a SONAME, rpmbuild fakes one based on the file name. The rationale for this has been that since the linker permits linking to libraries without sonames, rpm must permit this too. The downside is that for the 1% corner cases that need this, we end up polluting the provides namespace with thousands of unnecessary and unwanted names, names that have not been picked up with a global namespace in mind. To get a rough idea just how much of this unwanted junk exists, using a slightly modified elfdeps generator on my system: This is how we currently generate the provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only --fake-soname|wc -l 3875 And this is without faked soname provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only |wc -l 2816 So... about a third of the soname provides are faked by using library basename for its provide name, and almost all of them are for no good reason. The three general cases that something actually depends on are 1) libreoffice sub-packages depending on its internal libraries from libreoffice-core 2) a handful of packages depending on llvm libraries 3) bigger pile of packages depending on libperl.so Of these at least 2) does look like a bug really, and in fact has been fixed upstream: http://llvm.org/bugs/show_bug.cgi?id=12334. Dunno about 3), perl is... what it is. 1) might well classify as an inevitable special case (there will always be those), dunno. Obviously this is just a fairly small sample of the entire Fedora package set, there are bound to be many more special cases, buggy libraries which do not have SONAME when they should and stuff that falls into gray areas. I can try to provide more comprehensive stats based on current rawhide if needed. I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. Regarding perl, of the 48 packages that require libperl.so in Rawhide, only 13 of them don't also require perl(:MODULE_COMPAT_5.14.2), which will also pull in the perl-libs package, so I don't think perl is a big issue. In these cases, it's not a question of what other requires they might have. If something links against a library (in this case libperl.so), causing a require to be generated, something needs to provide that dependency. So in the case of libperl.so, the choices would basically be either: a) Add an actual soname to libperl.so (since it's something that other packages do link against, this would seem reasonable but then I dont know the perl world pecularities at all...) b) Enable the switch (whatever it will be) to generate these fake soname provides in the main perl package (or add the provide manually in the spec, but that gets ugly real fast) - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
Hi, On 04/27/2012 10:13 AM, Panu Matilainen wrote: I started once again looking at ways to eliminate the unwanted provides on private libs such as dlopen()'ed modules with minimal fuss and breakage. Been down this route more than once but I suspect the last time was before the major dependency generator changes in rpm-4.9.x, which made tweaking these things a good deal easier, so time for another round: The short background is that for libraries which dont have a SONAME, rpmbuild fakes one based on the file name. The rationale for this has been that since the linker permits linking to libraries without sonames, rpm must permit this too. The downside is that for the 1% corner cases that need this, we end up polluting the provides namespace with thousands of unnecessary and unwanted names, names that have not been picked up with a global namespace in mind. To get a rough idea just how much of this unwanted junk exists, using a slightly modified elfdeps generator on my system: This is how we currently generate the provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only --fake-soname|wc -l 3875 And this is without faked soname provides: [root@localhost rpm]# find /lib64/ /usr/lib64/ -type f |./elfdeps -P --soname-only |wc -l 2816 So... about a third of the soname provides are faked by using library basename for its provide name, and almost all of them are for no good reason. The three general cases that something actually depends on are 1) libreoffice sub-packages depending on its internal libraries from libreoffice-core 2) a handful of packages depending on llvm libraries 3) bigger pile of packages depending on libperl.so Of these at least 2) does look like a bug really, and in fact has been fixed upstream: http://llvm.org/bugs/show_bug.cgi?id=12334. Dunno about 3), perl is... what it is. 1) might well classify as an inevitable special case (there will always be those), dunno. Obviously this is just a fairly small sample of the entire Fedora package set, there are bound to be many more special cases, buggy libraries which do not have SONAME when they should and stuff that falls into gray areas. I can try to provide more comprehensive stats based on current rawhide if needed. I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Great! Can we disable these bogus provides starting tomorrow please? :) Regards, Hans -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
On Fri, Apr 27, 2012 at 11:13:47AM +0300, Panu Matilainen wrote: I'm going to add a switch to allow packages to control the behavior anyway. Whether rpm upstream defaults to the traditional behavior for compatibility reasons or not is another question, but Fedora is obviously free to override that. Given the packaging guideline mandate to eliminate private soname provides from packages it seems like a no-brainer to disable faked sonames by default, and require the handful of special cases to work around it (adding enable-switches to specs) instead of adding disable-switches to possibly hundreds of packages to cater for the special cases needs. Thoughts? This would obviously only change in rawhide. Oh and of course eliminating the faked soname provides does not eliminate all unwanted provides, eg libtool adds a soname even when -module is used. But ... you need to start someplace. Will the toggle to turn it on and off work for those libtool generated modules (as long as there's not public libraries in the same package)? Is it possible to turn this on and off per subpackage or is that along the lines of asking for pretty pink ponies with wings? Overall seems like a positive change to me. Let us know the details when it's changed and we'll add those to the Packaging Guidelines. -Toshio pgpnQWRm9AJ5B.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Eliminating automatic provides on private libs
[resent another time because the list automatically rejected my mail]
On 27/04/12 10:13, Panu Matilainen wrote:
[...]
The short background is that for libraries which dont have a SONAME,
rpmbuild fakes one based on the file name. The rationale for this has
been that since the linker permits linking to libraries without sonames,
rpm must permit this too. The downside is that for the 1% corner cases
that need this, we end up polluting the provides namespace with
thousands of unnecessary and unwanted names, names that have not been
picked up with a global namespace in mind.
[...]
So... about a third of the soname provides are faked by using library
basename for its provide name, and almost all of them are for no good
reason. The three general cases that something actually depends on are
1) libreoffice sub-packages depending on its internal libraries from
libreoffice-core
while i'm still quite unfamiliar with RPM or the libreoffice.spec file,
i'm very familiar with LibreOffice upstream; generally with the
exception of the -ure sub-package there are no stable ABIs provided by
the various libraries, so the sub-packages need to depend on the exact
same version of the core package anyway (and no code outside of the
libreoffice source package should have dependencies) which is probably
why the spec file contains this:
Requires: %{name}-core = %{epoch}:%{version}-%{release}
now the URE (UNO runtime environment) subpackage actually provides a
stable ABI and libraries with SONAMEs (in addition to some private
libraries without SONAME for dlopen), all installed in a sub-directory
of /usr/lib/libreoffice, but other than the other libreoffice
sub-packages, which again depend on it with exact version
Requires: %{name}-ure = %{epoch}:%{version}-%{release}
using repoquery i could find only 3 non-libreoffice packages with a
dependency on ure or its libraries in Fedora 16, all of which
LibreOffice extensions (actually 2 of the 3 are the same, once with the
old openoffice.org name), and depending only on the public libraries
with SONAME:
repoquery --requires openoffice.org-voikko-0:3.1.2-5.fc16.x86_64
openoffice.org-diafilter-0:1.7.0-6.fc15.x86_64
libreoffice-voikko-0:3.2-4.fc16.x86_64
[...]
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
