Re: -Wl,--as-needed considered possibly harmful
On Fri, 2008-01-04 at 17:52 -0800, Steve Langasek wrote: On Wed, Jan 02, 2008 at 11:53:16AM +, Neil Williams wrote: That's upstream covered, it appears I also need debian/libqof1.symbols from http://qa.debian.org/cgi-bin/mole/seedsymbols ? If I had done symbol versioning correctly upstream, shouldn't dpkg-shlibdeps be able to create the necessary data itself? I don't provide a .shlib file of my own at this stage. The symbols files are orthogonal to symbol versioning. Symbols files are basically per-function shlibs, which are applicable even to libs that don't use symbol versioning; their main benefit is for libraries which make backwards-compatible ABI changes. In which case I should have been supporting symbols files since v0.6.0 because that is exactly what I've been doing in this package for the last two years or more - adding new functions, deprecating the old ones, collecting all the changes into one transition to the next SONAME. Not sure why the sample symbols file showed the same version for all functions - quite a lot were simply not present in 0.6.0. Ne'er mind. It's a bit late now (I'm preparing for a SONAME bump where all those backwards-compatible layers are removed) but I'll implement it in the new version to allow tracking of future backwards-compatible ABI changes. Thanks. -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ signature.asc Description: This is a digitally signed message part
Re: -Wl,--as-needed considered possibly harmful
On Wed, Jan 02, 2008 at 11:53:16AM +, Neil Williams wrote: Because *all* libraries which have other libs as reverse-dependencies should implement symbol versioning, precisely as the shortest path for reliably and permanently addressing the various issues we're discussing. It *is* libbaz's fault for having an ABI-breaking change. And based on history, it is only reasonable to assume all such libraries are guilty, and demand that they use symbol versioning in anticipation of such future ABI changes. OK, I'm doing this upstream for one of my packages - following threads from: http://lists.debian.org/debian-mentors/2007/05/msg00387.html http://www.gnu.org/software/binutils/manual/ld-2.9.1/html_node/ld_25.html That gave me a basic versioning addition to the library, a diff of objdump -p shows: +Version definitions: +1 0x01 0x06077991 libqof.so.1 +2 0x00 0x0f307023 LIBQOF_0.7.3 + (I'm not putting the history of the symbols into the file at this stage - no new symbols are added in the new version so the libqof.ver file is simply marking all as global. I could add historical data - is there any benefit in that?) No, I don't think there's any benefit in adding the historical data after the fact. That's upstream covered, it appears I also need debian/libqof1.symbols from http://qa.debian.org/cgi-bin/mole/seedsymbols ? If I had done symbol versioning correctly upstream, shouldn't dpkg-shlibdeps be able to create the necessary data itself? I don't provide a .shlib file of my own at this stage. The symbols files are orthogonal to symbol versioning. Symbols files are basically per-function shlibs, which are applicable even to libs that don't use symbol versioning; their main benefit is for libraries which make backwards-compatible ABI changes. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Tue, 1 Jan 2008 23:17:50 -0800 Steve Langasek [EMAIL PROTECTED] wrote: On Wed, Dec 26, 2007 at 01:11:40PM +, Neil Williams wrote: I think I'm doing this already - if libfoo1 implements and exports types from libbar2 and libbar2 moves to libbar3, I would expect to have to port libfoo to libbar3 and this would usually cause a SONAME bump in libfoo. Have I got these possible scenarios correct: 1. libfoo depends on libbar2 without exporting libbar symbols. libfoo can migrate to libbar3 with internal code changes and if these do not require changing any libfoo interfaces, libfoo needs no SONAME bump. No, exporting libbar symbols is not the right line to draw. The right line is the one I already identified, namely, libfoo uses some of libbar's types in its own exported ABI. That makes more sense, thank you. Does libfoo need to use versioned symbols to cope with this No, libbar does, as the library which has other libraries for reverse-dependencies. (codehelp has a light bulb moment of clarity). :-) 2. libfoo depends on libbar2 and exports some libbar symbols in libfoo-dev. libfoo can migrate to libbar3 with only internal code changes because it only exports a portion of the libbar API that has not actually been changed in the libbar2-libbar3 transition. libfoo needs to export libbar in the libfoo --libs data. Is a SONAME bump needed in libfoo even though *none* of the libfoo interfaces have changed? As upstream for libfoo, I would not normally make a SONAME bump in this situation. If libfoo's exported ABI hasn't changed, then any package name or soname change is gratuitous and should be avoided. Agreed. 3. libfoo depends on libbar2 and needs to make changes to its own interfaces to cope with the libbar transition or needs to export symbols from the new libbar API and therefore makes a SONAME bump. No problem here except ensuring that applications transition to both API's at the same time. Can be enforced by making libbar-dev conflict with the old libfoo-dev or making the new libfoo-dev depend on the new libbar-dev version. Requires disruptive name changes to libbar-dev which affect all packages that build-depend on it, and is therefore again inferior to implementing symbol versions up front. OK. Because *all* libraries which have other libs as reverse-dependencies should implement symbol versioning, precisely as the shortest path for reliably and permanently addressing the various issues we're discussing. It *is* libbaz's fault for having an ABI-breaking change. And based on history, it is only reasonable to assume all such libraries are guilty, and demand that they use symbol versioning in anticipation of such future ABI changes. OK, I'm doing this upstream for one of my packages - following threads from: http://lists.debian.org/debian-mentors/2007/05/msg00387.html http://www.gnu.org/software/binutils/manual/ld-2.9.1/html_node/ld_25.html That gave me a basic versioning addition to the library, a diff of objdump -p shows: +Version definitions: +1 0x01 0x06077991 libqof.so.1 +2 0x00 0x0f307023 LIBQOF_0.7.3 + (I'm not putting the history of the symbols into the file at this stage - no new symbols are added in the new version so the libqof.ver file is simply marking all as global. I could add historical data - is there any benefit in that?) That's upstream covered, it appears I also need debian/libqof1.symbols from http://qa.debian.org/cgi-bin/mole/seedsymbols ? If I had done symbol versioning correctly upstream, shouldn't dpkg-shlibdeps be able to create the necessary data itself? I don't provide a .shlib file of my own at this stage. (A SONAME bump is due in this library fairly soon so I want to get symbol versioning into libqof1 prior to the ABI change.) No. You seem to agree that each soname change of a library near the bottom of the stack should not propagate its way up the stack to affect other libraries that themselves are not affected by the ABI change, but you again overlook (or disdain) the impact that this has on partial upgrades if we don't require those base libraries to use symbol versioning. It was an oversight. Sorry. -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ pgpCd2KIOnk2h.pgp Description: PGP signature
Re: -Wl,--as-needed considered possibly harmful
On Wed, Dec 26, 2007 at 01:11:40PM +, Neil Williams wrote: ... until both the application and libfoo are rebuilt. So the issue here is triggering rebuilds of reverse dependencies of libbar? No. That doesn't cause previously released binaries to blink out of existence. What's required is to ensure that there's a package rename each time libfoo's ABI changes, including when the libfoo ABI change is caused by type changes in the underlying lib. This is essentially the general case of the c102, c2, c2a, and ldbl transitions that we've been through for compiler changes over the past few years; in theory we should be able to accomplish these transitions for libraries with fewer reverse-dependencies than libstdc++ with significantly less pain, as long as library maintainers are forewarned. But once the package name change is handled, the rebuild of reverse-deps follows naturally. That sounds a lot better than the original idea of incorporating the entire dependency chain into the package name. I think I'm doing this already - if libfoo1 implements and exports types from libbar2 and libbar2 moves to libbar3, I would expect to have to port libfoo to libbar3 and this would usually cause a SONAME bump in libfoo. So could I ask, from an upstream perspective, what kind of changes in the underlying lib might *not* cause such a port and therefore end up with libfoo1 still being buildable against libbar3 yet *still* require a SONAME bump to accommodate the transition? Succinctly addressed by Robert. Have I got these possible scenarios correct: 1. libfoo depends on libbar2 without exporting libbar symbols. libfoo can migrate to libbar3 with internal code changes and if these do not require changing any libfoo interfaces, libfoo needs no SONAME bump. No, exporting libbar symbols is not the right line to draw. The right line is the one I already identified, namely, libfoo uses some of libbar's types in its own exported ABI. Does libfoo need to use versioned symbols to cope with this No, libbar does, as the library which has other libraries for reverse-dependencies. There needs to be some check on the application that compares the dependency chain as well as the dependency list in the package itself. lintian cannot do that. Not if libbar uses symbol versioning. 2. libfoo depends on libbar2 and exports some libbar symbols in libfoo-dev. libfoo can migrate to libbar3 with only internal code changes because it only exports a portion of the libbar API that has not actually been changed in the libbar2-libbar3 transition. libfoo needs to export libbar in the libfoo --libs data. Is a SONAME bump needed in libfoo even though *none* of the libfoo interfaces have changed? As upstream for libfoo, I would not normally make a SONAME bump in this situation. Would it be better to rename the Debian package without bumping the SONAME, e.g. libfoo1-0-libfoo1-1 ? Wouldn't this cause unnecessary rebuilds of applications that only depend on libfoo and not libbar? If libfoo's exported ABI hasn't changed, then any package name or soname change is gratuitous and should be avoided. 3. libfoo depends on libbar2 and needs to make changes to its own interfaces to cope with the libbar transition or needs to export symbols from the new libbar API and therefore makes a SONAME bump. No problem here except ensuring that applications transition to both API's at the same time. Can be enforced by making libbar-dev conflict with the old libfoo-dev or making the new libfoo-dev depend on the new libbar-dev version. Requires disruptive name changes to libbar-dev which affect all packages that build-depend on it, and is therefore again inferior to implementing symbol versions up front. Maybe I've got this wrong but there would appear to be a few methods to fix this: 1. Incorporating the libbar SONAME into the libfoo package SONAME - as Simon Richter recommended - this could quickly end up with libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that this would actually help anyone if, for example, libbaz0 migrates to libbaz1 in a way that completely breaks libbar2. Then either libbar2 becomes libbar3 at the same time, or libbar becomes unsupported (and the stack above it has to be ported or else it's not releasable), or libbaz needs to support symbol versioning to permit coexistence of the two versions in the same address space. Even if the actual fix for libbar2 is only internal? Then that would eliminate the first option, leaving only the other two. Yet the problem isn't in libbaz either, so why force libbaz to implement versioned symbols through (apparently) no fault of its own? Because *all* libraries which have other libs as reverse-dependencies should implement symbol versioning, precisely as the shortest path for reliably and permanently addressing the various issues we're discussing. It *is* libbaz's fault for
Re: -Wl,--as-needed considered possibly harmful
On Wed, 2007-12-26 at 13:11 +, Neil Williams wrote:
So could I ask, from an upstream perspective, what kind of changes in
the underlying lib might *not* cause such a port and therefore end up
with libfoo1 still being buildable against libbar3 yet *still* require
a
SONAME bump to accommodate the transition?
bar.h:
typedef unsigned int bar;
foo.h:
typedef struct {
bar a_bar;
} exported_foo_type;
Then this patch to bar.h:
@@ -0,1 +0,1 @@
-typedef unsigned int bar;
+typedef size_t bar;
is enough to cause an ABI break in libfoo, without needing 'porting'
work (though it may well cause print formatting failures on 64
environments and other side effects).
-Rob
--
GPG key available at: http://www.robertcollins.net/keys.txt.
signature.asc
Description: This is a digitally signed message part
Re: -Wl,--as-needed considered possibly harmful
On Tue, 2007-12-25 at 20:12 -0800, Steve Langasek wrote: On Fri, Dec 21, 2007 at 10:55:32AM +, Neil Williams wrote: On Thu, 20 Dec 2007 21:44:19 -0800 Steve Langasek [EMAIL PROTECTED] wrote: But *nothing* here guarantees that the version of libbar the application is linked against has the same ABI as the one libfoo itself linked against! ... until both the application and libfoo are rebuilt. So the issue here is triggering rebuilds of reverse dependencies of libbar? No. That doesn't cause previously released binaries to blink out of existence. What's required is to ensure that there's a package rename each time libfoo's ABI changes, including when the libfoo ABI change is caused by type changes in the underlying lib. This is essentially the general case of the c102, c2, c2a, and ldbl transitions that we've been through for compiler changes over the past few years; in theory we should be able to accomplish these transitions for libraries with fewer reverse-dependencies than libstdc++ with significantly less pain, as long as library maintainers are forewarned. But once the package name change is handled, the rebuild of reverse-deps follows naturally. That sounds a lot better than the original idea of incorporating the entire dependency chain into the package name. I think I'm doing this already - if libfoo1 implements and exports types from libbar2 and libbar2 moves to libbar3, I would expect to have to port libfoo to libbar3 and this would usually cause a SONAME bump in libfoo. So could I ask, from an upstream perspective, what kind of changes in the underlying lib might *not* cause such a port and therefore end up with libfoo1 still being buildable against libbar3 yet *still* require a SONAME bump to accommodate the transition? Have I got these possible scenarios correct: 1. libfoo depends on libbar2 without exporting libbar symbols. libfoo can migrate to libbar3 with internal code changes and if these do not require changing any libfoo interfaces, libfoo needs no SONAME bump. pkgconfig does not export libbar in the libfoo --libs data. If an application uses libbar and libfoo (for whatever reason), the ported version of that application that uses libbar3 simply needs to depend on the version of libfoo that depends on libbar3. In this case, an automated rebuild of the application could miss this step, causing breakage, yet forcing a change in the libfoo package name seems unnecessary (especially if there are other applications that depend on libfoo but not directly on libbar). Another problem is if the application ports to libbar3 before libfoo - currently, there is no way of reliably picking this up except the application maintainer being particularly careful with the pbuilder logs and noticing that libbar2 is still being installed during the build as well as libbar3 due to the indirect dependency on libfoo. Maybe pbuilder could highlight such incidences? (or even fail the build if it happens)? Maybe sbuild too? However, this would again raise the spectre of being unable to fix critical bugs in the application should libfoo be slow to port to libbar3. Again, it's a balance of risk - problems are *possible* when the app is linked against libbar3 and indirectly against libbar2 but in the absence of real bugs, is it worth causing more delays to other bug fixes? Does libfoo need to use versioned symbols to cope with this or is it simply down to those applications that use libfoo and libbar to specify the right version of libfoo once the application itself has ported to libbar3? (And wait until libfoo has also migrated?) What methods are needed to ensure that the application does this? (e.g. if left without changes, an application directly dependent upon libbar3 and libfoo could be built for unstable and presumably migrate into testing before the updated libfoo). There needs to be some check on the application that compares the dependency chain as well as the dependency list in the package itself. lintian cannot do that. 2. libfoo depends on libbar2 and exports some libbar symbols in libfoo-dev. libfoo can migrate to libbar3 with only internal code changes because it only exports a portion of the libbar API that has not actually been changed in the libbar2-libbar3 transition. libfoo needs to export libbar in the libfoo --libs data. Is a SONAME bump needed in libfoo even though *none* of the libfoo interfaces have changed? As upstream for libfoo, I would not normally make a SONAME bump in this situation. Would it be better to rename the Debian package without bumping the SONAME, e.g. libfoo1-0-libfoo1-1 ? Wouldn't this cause unnecessary rebuilds of applications that only depend on libfoo and not libbar? 3. libfoo depends on libbar2 and needs to make changes to its own interfaces to cope with the libbar transition or needs to export symbols from the new libbar API and therefore makes a SONAME bump. No problem here except ensuring that applications
Re: -Wl,--as-needed considered possibly harmful
On Fri, Dec 21, 2007 at 10:55:32AM +, Neil Williams wrote: On Thu, 20 Dec 2007 21:44:19 -0800 Steve Langasek [EMAIL PROTECTED] wrote: Consider two libraries, libfoo and libbar. libfoo depends on libbar, references functions from it and uses some of libbar's types in its own exported API. We assume the Debian-style libbar-dev, which ensures that the libbar headers match the version of libbar.so on the system. The pkg-config technique guarantee that, when an application links against libfoo, it is also linked against libbar. Because of the preceding, this guarantees that the application is linked against the version of libbar whose ABI matches that of the headers used when building the application. But *nothing* here guarantees that the version of libbar the application is linked against has the same ABI as the one libfoo itself linked against! ... until both the application and libfoo are rebuilt. So the issue here is triggering rebuilds of reverse dependencies of libbar? No. That doesn't cause previously released binaries to blink out of existence. What's required is to ensure that there's a package rename each time libfoo's ABI changes, including when the libfoo ABI change is caused by type changes in the underlying lib. This is essentially the general case of the c102, c2, c2a, and ldbl transitions that we've been through for compiler changes over the past few years; in theory we should be able to accomplish these transitions for libraries with fewer reverse-dependencies than libstdc++ with significantly less pain, as long as library maintainers are forewarned. But once the package name change is handled, the rebuild of reverse-deps follows naturally. Maybe I've got this wrong but there would appear to be a few methods to fix this: 1. Incorporating the libbar SONAME into the libfoo package SONAME - as Simon Richter recommended - this could quickly end up with libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that this would actually help anyone if, for example, libbaz0 migrates to libbaz1 in a way that completely breaks libbar2. Then either libbar2 becomes libbar3 at the same time, or libbar becomes unsupported (and the stack above it has to be ported or else it's not releasable), or libbaz needs to support symbol versioning to permit coexistence of the two versions in the same address space. 2. A rebuild trigger mechanism that is separate from the library SONAME and the pkg-config files - an automated version of what happens now for requesting binNMU's ? Not as easy as it may sound - there's no guarantee that libfoo *can* build against libbar2. Wrong for the reasons stated above. 3. A build-time hook that checks the entire dependency chain for duplicates and fails if a freshly built binary ends up linked to two versions of one source? (i.e. requiring a bug in whichever package is using libbar1 to upgrade to libbar2). That could lead to being unable to upload packages to fix other bugs. Doesn't sound like it actually solves the problem that needs solving for the same reasons as option 2. above, but could be usable if fleshed out appropriately. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Thu, 20 Dec 2007 21:44:19 -0800 Steve Langasek [EMAIL PROTECTED] wrote: Consider two libraries, libfoo and libbar. libfoo depends on libbar, references functions from it and uses some of libbar's types in its own exported API. We assume the Debian-style libbar-dev, which ensures that the libbar headers match the version of libbar.so on the system. The pkg-config technique guarantee that, when an application links against libfoo, it is also linked against libbar. Because of the preceding, this guarantees that the application is linked against the version of libbar whose ABI matches that of the headers used when building the application. But *nothing* here guarantees that the version of libbar the application is linked against has the same ABI as the one libfoo itself linked against! ... until both the application and libfoo are rebuilt. So the issue here is triggering rebuilds of reverse dependencies of libbar? I can see this getting to be a big problem if the application in this example is actually another library. If libfoo actually doesn't build against libbar2 because of the API changes, the whole thing grinds to a halt. After all, libbar1-libbar2 could involve any level of changes to the API. If the application uses 5% of the libbar functions and the library uses 50%, it's easy to see that the library has a higher risk of needing intrusive source changes to build against the new libbar. The Gtk1-Gtk2 transition is still far from complete, KDE3-4 is current. Though the combination of libbar1 and libbar2 raises the chances of problems, forcing rebuilds whether the problem appears or not would seem to increase the problems of blocked transitions and hinder releases even further. If libfoo linked against libbar1, the pkg-config approach only ensures that when your application is built against the libbar2 version of libbar-dev, the resulting binary will depend on both libbar1 and libbar2 (despite not using any symbols from libbar2). All this does is increase the chances of segfaults or bad runtime behavior as a result of symbol clobbering. So the usage recommended by pkg-config upstream does nothing at all to solve the problem it's supposed to, and instead causes other linkage problems. Maybe I've got this wrong but there would appear to be a few methods to fix this: 1. Incorporating the libbar SONAME into the libfoo package SONAME - as Simon Richter recommended - this could quickly end up with libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that this would actually help anyone if, for example, libbaz0 migrates to libbaz1 in a way that completely breaks libbar2. 2. A rebuild trigger mechanism that is separate from the library SONAME and the pkg-config files - an automated version of what happens now for requesting binNMU's ? Not as easy as it may sound - there's no guarantee that libfoo *can* build against libbar2. 3. A build-time hook that checks the entire dependency chain for duplicates and fails if a freshly built binary ends up linked to two versions of one source? (i.e. requiring a bug in whichever package is using libbar1 to upgrade to libbar2). That could lead to being unable to upload packages to fix other bugs. What is the correct solution? Is there a way that the current pkg-config *can* provide what upstream believe it already does? Even if pkg-config upstream are thinking of gentoo-type distributions, you'll still end up with a prebuilt libfoo depending on libbar1 and a freshly built application depending on libbar2, even using libbar1-dev and libbar2-dev (although at least that is more obvious). Maybe debtree could be useful here - or maybe edos-debcheck could be extended to look for problems like this. It should be apparent from the apt cache that the application uses libbar2 and libfoo but that libfoo depends on libbar1. Maybe there is a role here for deborphan or a similar tool as well - to scan an installation for not just orphaned libraries but duplicates as well. Getting back to the original thread, if applications and libraries each take more care not to link to spurious dependencies, it should be easier to identify the real problems instead of chasing red herrings where the application is linked to libbar2 and libfoo but actually only uses symbols from libfoo. I've just uploaded libqof1 that used the information from the new dpkg-shlibdeps to trim the dependencies of libqof1, excluding some libraries that were previously only brought in because of bugs in the libgda pkg-config data. It meant changing the QOF upstream to check for the particular functions that it needs instead of trusting the pkg-config data from libgda so that it could identify the correct dependency data for a public library provided by libgda but not exposed in the pkg-config data. (libgdasql). -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ pgp9rOUAD4yYe.pgp Description:
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 05:45:27PM +0100, Josselin Mouette wrote: Le dimanche 09 décembre 2007 à 15:44 -0800, Steve Langasek a écrit : For example, pkg-config --libs gtk+-x11-2.0 will return, among others, -lglib-2.0 and -lm. And this is perfectly intentional. Just because it's intentional doesn't mean it isn't absurd and wrong. It may be absurd, but I don’t think it is wrong. No, what can be done is to fix upstream's broken declaration that 'you can assume glib functions are available when doing #include gtk/gtk.h'. It doesn't follow that just because this works in practice, it should be a supported usage. When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. Well, that part is fairly reasonable, I admit. What isn't reasonable is to go from including gtk.h lets you use glib types to calling pkg-config --libs gtk+-2.0 lets you invoke glib functions. Yes, including gtk.h is always going to be sufficient, in practice, to get the glib types; but a) includes != pkg-config, b) since it isn't implied there really is no obligation to support such an expectation wrt pkg-config use, aside from upstream's apparent commitment to compatibility with foolishly-assembled build systems. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 09:47:36PM +0100, Simon Richter wrote: No, what can be done is to fix upstream's broken declaration that 'you can assume glib functions are available when doing #include gtk/gtk.h'. It doesn't follow that just because this works in practice, it should be a supported usage. When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. If it is really expected that type declarations are to be shared between a program and a library, then the program becomes dependent on the library's ABI without this dependency being formally expressed in any usable form, which is broken in and by itself. GTK needs to provide its own definitions of types and not expose interfaces it does not control. Fine in theory, in practice this can be a significant burden to the library maintainer. FWIW, I don't see any reason why a library can't use externally-defined types directly, /as long as/ they have some mechanism for ensuring ABI consistency between the two libraries (i.e., an ABI change in the underlying library is always accompanied by an soname change in the dependent library). (BTW, pkg-config upstream wrongly claims that exporting dependent libs in the pkg-config --libs output provides this protection. :/) -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Le jeudi 20 décembre 2007 à 00:41 -0800, Steve Langasek a écrit : When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. Well, that part is fairly reasonable, I admit. What isn't reasonable is to go from including gtk.h lets you use glib types to calling pkg-config --libs gtk+-2.0 lets you invoke glib functions. Most GTK+ data types are inherited from Glib, and you need at least to free them with g_free, g_object_unref, g_slist_free, and so on. I’ve tried to sum up in bug #456335 the garbage that remains in GTK’s .pc files, so that in we end we can keep only the legitimate ones, but I don’t think we can manage to drop things like glib and gobject. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: -Wl,--as-needed considered possibly harmful
On Thu, 2007-12-20 at 00:51 -0800, Steve Langasek wrote: On Mon, Dec 10, 2007 at 09:47:36PM +0100, Simon Richter wrote: No, what can be done is to fix upstream's broken declaration that 'you can assume glib functions are available when doing #include gtk/gtk.h'. It doesn't follow that just because this works in practice, it should be a supported usage. When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. If it is really expected that type declarations are to be shared between a program and a library, then the program becomes dependent on the library's ABI without this dependency being formally expressed in any usable form, which is broken in and by itself. GTK needs to provide its own definitions of types and not expose interfaces it does not control. Fine in theory, in practice this can be a significant burden to the library maintainer. FWIW, I don't see any reason why a library can't use externally-defined types directly, /as long as/ they have some mechanism for ensuring ABI consistency between the two libraries (i.e., an ABI change in the underlying library is always accompanied by an soname change in the dependent library). I'd make a stronger statement and say that reusing types is a positive thing. If you are reusing a glib list in your pet library, creating your own type makes your library incompatible with other libraries also using glib lists and providing code to manipulate them... and if it doesn't make it incompatible then its not going to achieve the ABI considerations that are under discussion here. So really I think each dependant library of a library that is dynamically linked to falls into one of two categories: - internal detail - implementation detail resulting in exposed ABI So if I link program X against libfoo which uses libbar, if libbar is an internal detail of libfoo then: libfoo - [EMAIL PROTECTED] ABI X - [EMAIL PROTECTED]'s ABI only. And a rebuild of libfoo to link a new libbar ABI won't break X, nor will it require a rebuild of X. The conditions for internal detail only are very stringent: compiler alignment, structure sizes etc etc for all non opaque (where opaque is opaque to the /compiler/ not to the user's code!) exported types of libfoo must not be dependent on libbar at all. I doubt that many libraries will meet this requirement for all their dependent libraries (and consider libc as just an extremely common case of this). For all other cases, when a used ABI gets exposed, while we may not want X to depend (at the package level) on libbar; at an ABI level it does. We need to decide how we represent that. We also have at least two dimensions here: runtime linker behaviour package management behaviour From the runtime linker we want: * an error/warning/something when two instances of the same API with different ABI's are loaded. This commonly occurs when libbar.1 and libbar.2 are both loaded, but it can occur more generally when libbar is renamed to libquux and then libbar.1 and libquux.2 are loaded (by X and libfoo respectively). * To be able to interrogate how something was linked sufficiently well to generate packaging metadata automatically. From the package manager we want: * To know when we have to rebuild X because libbar has changed. * To allow upgrades to libbar on users systems without changing X when possible. The case where libbar is an internal detail is trivial (detection isn't: see abicheck and icheck - libraries really need icheck in the package build rules IMNSHO); I'm going to ignore it here. For the case where libbar is not an internal detail. Starting with packaging as thats easiest: * we should depend on a minimum version of the libbar we need (as we already bump the package name when the ABI changes incompatibly in a library this is straight forward) and gives the opportunity to upgrade libbar within an ABI safely. * we can do an rdepends on e.g. libbar2 when libbar's API changes from 2 to 3 to find packages we need to rebuild. One defect here is when symbols are shared from within a non-library - e.g. the case with some programs plugin's. Here, a plugin may use a symbol from within the program X, which is ABI dependent all the way back, so treating programs as libraries - that is having an ABI - might make sense. I haven't considered this in detail... The runtime loader is more tricky: * If we link X to libfoo only: - libfoo's soname has to be manually changed when libbar's changes or hilarity results - we don't have a robust way that I know of to interrogate X to find out that we need a package dependency on libbar + we don't need to have libbar present when it is only optionally used (that is by calling some libfoo feature that needs libbar) * If we link X to libbar directly: - because we can't tell how much of recent changes to libbar are needed, any
Re: -Wl,--as-needed considered possibly harmful
On Fri, Dec 21, 2007 at 04:08:28AM +1100, Robert Collins wrote: (BTW, pkg-config upstream wrongly claims that exporting dependent libs in the pkg-config --libs output provides this protection. :/) It doesn't? (I must be missing something as I thought it did, just over aggressively). Nope. Consider two libraries, libfoo and libbar. libfoo depends on libbar, references functions from it and uses some of libbar's types in its own exported API. We assume the Debian-style libbar-dev, which ensures that the libbar headers match the version of libbar.so on the system. The pkg-config technique guarantee that, when an application links against libfoo, it is also linked against libbar. Because of the preceding, this guarantees that the application is linked against the version of libbar whose ABI matches that of the headers used when building the application. But *nothing* here guarantees that the version of libbar the application is linked against has the same ABI as the one libfoo itself linked against! If libfoo linked against libbar1, the pkg-config approach only ensures that when your application is built against the libbar2 version of libbar-dev, the resulting binary will depend on both libbar1 and libbar2 (despite not using any symbols from libbar2). All this does is increase the chances of segfaults or bad runtime behavior as a result of symbol clobbering. So the usage recommended by pkg-config upstream does nothing at all to solve the problem it's supposed to, and instead causes other linkage problems. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 18:29:36 +0100, Raphael Hertzog wrote: On Mon, 10 Dec 2007, Sune Vuorela wrote: Isn't the new dpkg-shlipdeps warning about symbols it can't find ? Only if the library has a SONAME. This is intended so that perl/python/apache modules do not generate bad warnings: they rely on the binary dlopening them to provide the symbols (and it's okay that way). So the problem might still be missed on some plugins without SONAME. And for plugins/modules with an SONAME, it will be missed in the noise of warnings about symbols provided by the main binary/library. Cheers, Julien -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 17:45:27 +0100, Josselin Mouette wrote: When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. It sounds wrong not to ask them to use -lglib-2.0 themselves if they call glib functions. Cheers, Julien -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Le lundi 10 décembre 2007 à 18:24 +, Neil Williams a écrit : I don't have a problem with libglib2.0-0 in gtk+2.0.pc - it may well be correct to have that one in the pkgconfig because gtk headers define variables in terms of Glib typedefs. (I have to do the same with libqof1). The actual string is: -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 -lm -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0 And it used to be much longer. We’ve been working on reducing it and it is still work in progress. 1. Glib: gmodule2.0 (and it's variants) shouldn't export -ldl. This is minor because it's a libc6 lib anyway so it doesn't cause an extra dependency but it will be the only unwanted link in the next version of libqof1 now that I've fixed the others. 2. Gtk+2.0 really shouldn't export gmodule. Even if there is a reason for lglib2.0, there is no reason for -lgmodule. It is trivial to omit this linkage from the gtk+2.0.pc file. Both sound right, although this is something that needs some more checking. 3. Ditto for gobject although I might be wrong on that one. Yes, GObject provides the object model and I don’t think we can omit it. 4. Gtk+2: There is no need, as I see it, for atk, m, pangocairo, pango or cairo to be Requires: in gtk+2.0.pc - these should be Requires.private: instead. IIRC most of them are used in either macros or for the types they provide. Maybe some of them could be omitted, but not all of them. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: -Wl,--as-needed considered possibly harmful
Le Monday 10 December 2007 20:03:50 Pierre Habouzit, vous avez écrit :
Perhaps the absurd and wrong part is that pkg-config does not
provide a way to distinguish between use cases, and that the name for
Wrong, please read pkg-config(1) and think again.
$ pkg-config tokyocabinet --libs
-ltokyocabinet
$ pkg-config tokyocabinet --libs --static
-ltokyocabinet -lz -lpthread -lm -lc
$ grep Libs /usr/lib/pkgconfig/tokyocabinet.pc
Libs: -L${libdir} -ltokyocabinet
Libs.private: -lz -lpthread -lm -lc
pkg-config *is* perfectly suited for that matter actually, and
broken pkgconfig upstreams are this trivial to fix. My upstream for
tokyocabinet uses in a tokyocabinet.pc.in:
Libs: -L${libdir} -ltokyocabinet @LIBS@
the fix is just to put the @LIBS@ onto Libs.private and be done with it.
Upstreams using pkg-config are actually the first _easy_ targets to fix
the dependency issues _they_ generate in others binaries. And there is
no reason no to fix those when it's easy to do so.
Right, so now I now how to fix #395185 .. Seems I lacked some knowledge about
pkg-config :-)
Romain
Re: -Wl,--as-needed considered possibly harmful
On Sun, Dec 09, 2007, Simon Richter wrote: If there are broken scripts adding too many libraries then it is time to fix those scripts, not employ an evil hack that makes the symptoms go away. We met actual cases were --as-needed breaks things; however, for pure C program, I think combining it with -z defs should be safe: LDFLAGS += -Wl,-z,defs -Wl,--as-needed Sadly, there are many cases where -z defs will actually fail the build, for example when creating Python bindings: these may not be linked to libpython or to python as they might be loaded from either one, and you don't want to load the other one (if you run within python, you don't want to load libpython and vice-versa). In such cases, -z defs will fail the build because of missing link flags for the Python symbols. I heartily agree with other people that fixing the build rules to not add superfluous link flags is the best thing to do, but this ain't easy in many cases, and --as-needed might allow us to benefit from shorter Depends immediately. If fixing the build is too hard and your build passes with -z defs, then you are probably safe with --as-needed. NB: -z defs is the same as --no-undefined. -- Loïc Minier -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Sun, Dec 09, 2007, Zack Weinberg wrote: I have in the past argued for --as-needed to be made the default in upstream binutils; that's how safe I think it is. (Upstream maintainers, conscious of the above and other (isomorphic) corner cases, wanted a distribution to try it on a large scale first. Thus I am very happy to see Debian experimenting with it.) I'm curious what prompted your message. Did a program you use actually break? What was the failure mode? When this was added to pkg-gnome at large, we were hit by at least toolchain issues on some arches (was it hppa?) -- now fixed -- and also got some borken packages. I think gnome-panel / gnome-applets and gedit were the two packages that were hit when we added --as-needed. I think in both cases it was something like the main binary being linked to many utility libraries (because it was easy to link it to everything which configure found), and then the plugin calling directly into these libs, but not being linked to them. If the main binary didn't actually use the libs, these were stripped out, and then the dlopening of the plugins would fail. These were really issues in the build process, issues which would have been caught with -z defs before --as-needed (which is why I'm advocating usage of -z defs before --as-needed), but slipped silently in the end package and actually hit users. I don't think there's an insanely strong incentive to make --as-needed the default or to not make it; distros such as Debian and Gentoo are using it wildly at the moment; it just can point out some surprize bugs in my experience. HTH, -- Loïc Minier -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On 2007-12-10, Loïc Minier [EMAIL PROTECTED] wrote: I think in both cases it was something like the main binary being linked to many utility libraries (because it was easy to link it to everything which configure found), and then the plugin calling directly into these libs, but not being linked to them. If the main binary didn't actually use the libs, these were stripped out, and then the dlopening of the plugins would fail. These were really issues in the build process, issues which would have been caught with -z defs before --as-needed (which is why I'm advocating usage of -z defs before --as-needed), but slipped silently in the end package and actually hit users. Isn't the new dpkg-shlipdeps warning about symbols it can't find ? /Sune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007, Sune Vuorela wrote: Isn't the new dpkg-shlipdeps warning about symbols it can't find ? Indeed; I was telling a story from the dark ages where .debs were crafted with silex in your bare hands. That said, -z defs is a good idea to check all intermediate objects and to make the build actually fail on the first incorrect intermediate object (dpkg-shlibdeps will only tell you at the end of the build and only tell you about the installed final objects with missing NEEDED). -- Loïc Minier -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Le dimanche 09 décembre 2007 à 15:44 -0800, Steve Langasek a écrit : For example, pkg-config --libs gtk+-x11-2.0 will return, among others, -lglib-2.0 and -lm. And this is perfectly intentional. Just because it's intentional doesn't mean it isn't absurd and wrong. It may be absurd, but I don’t think it is wrong. No, what can be done is to fix upstream's broken declaration that 'you can assume glib functions are available when doing #include gtk/gtk.h'. It doesn't follow that just because this works in practice, it should be a supported usage. When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: -Wl,--as-needed considered possibly harmful
On Mon, 10 Dec 2007, Sune Vuorela wrote: On 2007-12-10, Loïc Minier [EMAIL PROTECTED] wrote: I think in both cases it was something like the main binary being linked to many utility libraries (because it was easy to link it to everything which configure found), and then the plugin calling directly into these libs, but not being linked to them. If the main binary didn't actually use the libs, these were stripped out, and then the dlopening of the plugins would fail. These were really issues in the build process, issues which would have been caught with -z defs before --as-needed (which is why I'm advocating usage of -z defs before --as-needed), but slipped silently in the end package and actually hit users. Isn't the new dpkg-shlipdeps warning about symbols it can't find ? Only if the library has a SONAME. This is intended so that perl/python/apache modules do not generate bad warnings: they rely on the binary dlopening them to provide the symbols (and it's okay that way). So the problem might still be missed on some plugins without SONAME. Cheers, -- Raphaël Hertzog Le best-seller français mis à jour pour Debian Etch : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Josselin Mouette wrote: Le dimanche 09 décembre 2007 à 15:44 -0800, Steve Langasek a écrit : For example, pkg-config --libs gtk+-x11-2.0 will return, among others, I don't have a problem with libglib2.0-0 in gtk+2.0.pc - it may well be correct to have that one in the pkgconfig because gtk headers define variables in terms of Glib typedefs. (I have to do the same with libqof1). The actual string is: -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 -lm -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0 As I see it, there are a few problems here, one of which is in libglib2.0: 1. Glib: gmodule2.0 (and it's variants) shouldn't export -ldl. This is minor because it's a libc6 lib anyway so it doesn't cause an extra dependency but it will be the only unwanted link in the next version of libqof1 now that I've fixed the others. 2. Gtk+2.0 really shouldn't export gmodule. Even if there is a reason for lglib2.0, there is no reason for -lgmodule. It is trivial to omit this linkage from the gtk+2.0.pc file. 3. Ditto for gobject although I might be wrong on that one. 4. Gtk+2: There is no need, as I see it, for atk, m, pangocairo, pango or cairo to be Requires: in gtk+2.0.pc - these should be Requires.private: instead. This is the single largest cause of unwanted linking in GPE. Sadly, it isn't possible to get lintian to check for these inter-package problems but the basic check, IMHO, would be to check the headers of the -dev package against the headers of the package being considered for Requires: If none match, it should be Requires.private: It should be possible to automate that in perl or python - gather the declarations of your package into a hash, read in the declarations of the package being considered for Requires:. No matches - Requires.private or omit altogether. The problem is how many packages already have this assumption built-in and will FTBFS when gtk+2.0.pc changes to drop the extra libraries. Theoretically, other packages should have done things properly and specified their -dev packages in full but I think that isn't going to be the reality. -lglib-2.0 and -lm. And this is perfectly intentional. Just because it's intentional doesn't mean it isn't absurd and wrong. It may be absurd, but I don’t think it is wrong. No, what can be done is to fix upstream's broken declaration that 'you can assume glib functions are available when doing #include gtk/gtk.h'. It doesn't follow that just because this works in practice, it should be a supported usage. When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. Maybe so, but it doesn't excuse the rest. -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ signature.asc Description: OpenPGP digital signature
Re: -Wl,--as-needed considered possibly harmful
Steve Langasek writes: On Sun, Dec 09, 2007 at 07:23:00PM +0100, Josselin Mouette wrote: Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit : Just curing the symptoms instead of the problems will not help to get there any sooner. What if there is no problem? For example, pkg-config --libs gtk+-x11-2.0 will return, among others, -lglib-2.0 and -lm. And this is perfectly intentional. Just because it's intentional doesn't mean it isn't absurd and wrong. What happens for a user who (however absurd or insane he might be to try this with gtk+) tries to link his application statically? Perhaps the absurd and wrong part is that pkg-config does not provide a way to distinguish between use cases, and that the name for the current behavior should also be --static-libs rather than --libs, but there is a good reason to provide the transitive closure of dependencies for a package *somewhere* in pkg-config. Michael Poole
Re: -Wl,--as-needed considered possibly harmful
This one time, at band camp, Michael Poole said: What happens for a user who (however absurd or insane he might be to try this with gtk+) tries to link his application statically? Perhaps the absurd and wrong part is that pkg-config does not provide a way to distinguish between use cases, and that the name for the current behavior should also be --static-libs rather than --libs, but there is a good reason to provide the transitive closure of dependencies for a package *somewhere* in pkg-config. That is supported in pkg-config, but no upstream I've ever met seems to understand it yet. -- - | ,''`.Stephen Gran | | : :' :[EMAIL PROTECTED] | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 06:47:15PM +, Michael Poole wrote:
Steve Langasek writes:
On Sun, Dec 09, 2007 at 07:23:00PM +0100, Josselin Mouette wrote:
Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit :
Just curing the symptoms instead of the problems will not help to get
there any sooner.
What if there is no problem?
For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
-lglib-2.0 and -lm. And this is perfectly intentional.
Just because it's intentional doesn't mean it isn't absurd and wrong.
What happens for a user who (however absurd or insane he might be to
try this with gtk+) tries to link his application statically?
Perhaps the absurd and wrong part is that pkg-config does not
provide a way to distinguish between use cases, and that the name for
Wrong, please read pkg-config(1) and think again.
$ pkg-config tokyocabinet --libs
-ltokyocabinet
$ pkg-config tokyocabinet --libs --static
-ltokyocabinet -lz -lpthread -lm -lc
$ grep Libs /usr/lib/pkgconfig/tokyocabinet.pc
Libs: -L${libdir} -ltokyocabinet
Libs.private: -lz -lpthread -lm -lc
pkg-config *is* perfectly suited for that matter actually, and
broken pkgconfig upstreams are this trivial to fix. My upstream for
tokyocabinet uses in a tokyocabinet.pc.in:
Libs: -L${libdir} -ltokyocabinet @LIBS@
the fix is just to put the @LIBS@ onto Libs.private and be done with it.
Upstreams using pkg-config are actually the first _easy_ targets to fix
the dependency issues _they_ generate in others binaries. And there is
no reason no to fix those when it's easy to do so.
--
·O· Pierre Habouzit
··O[EMAIL PROTECTED]
OOOhttp://www.madism.org
pgpCfl6oQ2bbd.pgp
Description: PGP signature
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 01:47:15PM -0500, Michael Poole wrote: Steve Langasek writes: On Sun, Dec 09, 2007 at 07:23:00PM +0100, Josselin Mouette wrote: Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit : Just curing the symptoms instead of the problems will not help to get there any sooner. What if there is no problem? For example, pkg-config --libs gtk+-x11-2.0 will return, among others, -lglib-2.0 and -lm. And this is perfectly intentional. Just because it's intentional doesn't mean it isn't absurd and wrong. What happens for a user who (however absurd or insane he might be to try this with gtk+) tries to link his application statically? Not that you should link statically with gtk (this yields the classic problem of internal ABI skew when statically linking with a library that uses dlopen), but pkg-config --static does exactly this. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Stephen Gran writes: This one time, at band camp, Michael Poole said: What happens for a user who (however absurd or insane he might be to try this with gtk+) tries to link his application statically? Perhaps the absurd and wrong part is that pkg-config does not provide a way to distinguish between use cases, and that the name for the current behavior should also be --static-libs rather than --libs, but there is a good reason to provide the transitive closure of dependencies for a package *somewhere* in pkg-config. That is supported in pkg-config, but no upstream I've ever met seems to understand it yet. Ah, I see. I overlooked the --static flag to pkg-config because (case in point) pkg-config --libs gtk+-2.0 only misses seven of the libraries that the --static version uses. I would guess this is because Requires.private is not mentioned in the pkg-config man page. At least on my system, Libs.private seems to be used correctly by most or all .pc files, but Requires.private use is spotty. Thank you for highlighting that. Michael Poole -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 01:47:15PM -0500, Michael Poole wrote: Steve Langasek writes: For example, pkg-config --libs gtk+-x11-2.0 will return, among others, -lglib-2.0 and -lm. And this is perfectly intentional. Just because it's intentional doesn't mean it isn't absurd and wrong. What happens for a user who (however absurd or insane he might be to try this with gtk+) tries to link his application statically? Perhaps the absurd and wrong part is that pkg-config does not provide a way to distinguish between use cases, and that the name for the current behavior should also be --static-libs rather than --libs, but there is a good reason to provide the transitive closure of dependencies for a package *somewhere* in pkg-config. pkg-config *does* have a --static flag, which does appear to cause --libs to emit Libs.private and other things that would normally be handled by the dynamic-linking of shared libs. dan -- Daniel Macks [EMAIL PROTECTED] http://www.netspace.org/~dmacks -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Am Montag 10 Dezember 2007 schrieb Michael Poole: Stephen Gran writes: This one time, at band camp, Michael Poole said: What happens for a user who (however absurd or insane he might be to try this with gtk+) tries to link his application statically? Perhaps the absurd and wrong part is that pkg-config does not provide a way to distinguish between use cases, and that the name for the current behavior should also be --static-libs rather than --libs, but there is a good reason to provide the transitive closure of dependencies for a package *somewhere* in pkg-config. That is supported in pkg-config, but no upstream I've ever met seems to understand it yet. Ah, I see. I overlooked the --static flag to pkg-config because (case in point) pkg-config --libs gtk+-2.0 only misses seven of the libraries that the --static version uses. I would guess this is because Requires.private is not mentioned in the pkg-config man page. At least on my system, Libs.private seems to be used correctly by most or all .pc files, but Requires.private use is spotty. The flaw in the documentation is surely the part to blame here. Requires.Private works but is not mentioned anywhere. Neither is the --static option in the SYNOPSIS part of the man page. Not many use undocumented stuff. HS -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 08:16:10PM +, Hendrik Sattler wrote: Am Montag 10 Dezember 2007 schrieb Michael Poole: Stephen Gran writes: This one time, at band camp, Michael Poole said: What happens for a user who (however absurd or insane he might be to try this with gtk+) tries to link his application statically? Perhaps the absurd and wrong part is that pkg-config does not provide a way to distinguish between use cases, and that the name for the current behavior should also be --static-libs rather than --libs, but there is a good reason to provide the transitive closure of dependencies for a package *somewhere* in pkg-config. That is supported in pkg-config, but no upstream I've ever met seems to understand it yet. Ah, I see. I overlooked the --static flag to pkg-config because (case in point) pkg-config --libs gtk+-2.0 only misses seven of the libraries that the --static version uses. I would guess this is because Requires.private is not mentioned in the pkg-config man page. At least on my system, Libs.private seems to be used correctly by most or all .pc files, but Requires.private use is spotty. The flaw in the documentation is surely the part to blame here. Requires.Private works but is not mentioned anywhere. Neither is the --static option in the SYNOPSIS part of the man page. Not many use undocumented stuff. You're kidding ? If someone that produces libraries stop at the SYNOPSIS section while reading documentation, he's a moron. The pkg-config tool has a very concise documentation that says it all, --static is documented in the OPTIONS section, and Requires, Libs, Libs.private in METADATA FILE SYNTAX where due. Calling it undocumented is just either bad faith or well, I can't see what else it can be in fact. -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpbVDohNY3Wz.pgp Description: PGP signature
Re: -Wl,--as-needed considered possibly harmful
Hi, Josselin Mouette wrote: No, what can be done is to fix upstream's broken declaration that 'you can assume glib functions are available when doing #include gtk/gtk.h'. It doesn't follow that just because this works in practice, it should be a supported usage. When many of the types used by GTK+ are those provided by GLib, it sounds wrong to ask developers to include the GLib headers to have these types available. If it is really expected that type declarations are to be shared between a program and a library, then the program becomes dependent on the library's ABI without this dependency being formally expressed in any usable form, which is broken in and by itself. GTK needs to provide its own definitions of types and not expose interfaces it does not control. Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, 10 Dec 2007 22:10:49 +0100 Pierre Habouzit [EMAIL PROTECTED] wrote: That is supported in pkg-config, but no upstream I've ever met seems to understand it yet. Ah, I see. I overlooked the --static flag to pkg-config because (case in point) pkg-config --libs gtk+-2.0 only misses seven of the libraries that the --static version uses. I would guess this is because Requires.private is not mentioned in the pkg-config man page. At least on my system, Libs.private seems to be used correctly by most or all .pc files, but Requires.private use is spotty. The flaw in the documentation is surely the part to blame here. Requires.Private works but is not mentioned anywhere. Neither is the --static option in the SYNOPSIS part of the man page. Not many use undocumented stuff. You're kidding ? If someone that produces libraries stop at the SYNOPSIS section while reading documentation, he's a moron. The pkg-config tool has a very concise documentation that says it all, --static is documented in the OPTIONS section, and Requires, Libs, Libs.private in METADATA FILE SYNTAX where due. Calling it undocumented is just either bad faith or well, I can't see what else it can be in fact. What *is* missing is documentation of Requires.private: -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ pgpOwR2Qu3swy.pgp Description: PGP signature
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 12:35:11AM -0300, Felipe Sateler wrote: Steve Langasek wrote: ... also, -Wl,--as-needed is *not* a complete solution for the problems caused by generating extra -l arguments. Every -lfoo option passed to the linker requires the corresponding -dev package to be installed at build time; while it is a bug if the -dev packages don't declare their dependencies, the impact of such bugs (which do happen fairly often) would be limited to static linking if appropriately-constructed .pc files were available. Note that pkgconfig is not the only problem here. Some extra links arise from the fact that several utilities or plugins for an application are built with the same environment (ie: the same flags), and not all of them require all the linked libraries. Of course, one could argue that fixing the build script is the way to go, but sometimes it is not trivial to do that (badly crafted or unmaintained build scripts tend to be quite ugly). However, most of them are fixable. The only two problems we (ALT) had/have with default --as-needed were mpeg4ip suit and net-snmp (though it's being fixed already). But again, not every upstream author accepts changes to .pc; but i've never seen one not wanting to fix a build system. -- ALT Linux Team, ALT Linux Sisyphus project, http://www.altlinux.com signature.asc Description: Digital signature
-Wl,--as-needed considered possibly harmful
Hi, due to the recent dpkg-shlibdeps changes, people have started adding -Wl,--as-needed into their LDFLAGS. THIS IS NOT A GOOD IDEA. You are essentially telling gcc to pass an option to the linker without understanding what it does, and, more specifically, an option that tells the linker to second-guess the gcc compiler driver. This can introduce really interesting and subtle bugs that will be difficult to find. If there are broken scripts adding too many libraries then it is time to fix those scripts, not employ an evil hack that makes the symptoms go away. Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
due to the recent dpkg-shlibdeps changes, people have started adding -Wl,--as-needed into their LDFLAGS. THIS IS NOT A GOOD IDEA. You are essentially telling gcc to pass an option to the linker without understanding what it does, and, more specifically, an option that tells the linker to second-guess the gcc compiler driver. This can introduce really interesting and subtle bugs that will be difficult to find. To first order, the only scenario I am aware of in which it can cause problems is if someone uses a global variable with a C++ constructor in a shared library, that constructor does critical work for the application the library is linked into, and the application does not reference any symbols whatsoever from the shared library. This is not impossible, but it is so unlikely IMO that the possibliity can be neglected. I have in the past argued for --as-needed to be made the default in upstream binutils; that's how safe I think it is. (Upstream maintainers, conscious of the above and other (isomorphic) corner cases, wanted a distribution to try it on a large scale first. Thus I am very happy to see Debian experimenting with it.) I'm curious what prompted your message. Did a program you use actually break? What was the failure mode? If there are broken scripts adding too many libraries then it is time to fix those scripts, not employ an evil hack that makes the symptoms go away. There are a *lot* of broken scripts. Would you like to mass-file bugs on every package that contains a binary that links to libnsl? (this iscommon, thanks to a buggy example in the autoconf manual, but completely unnecessary under glibc for anything other than a small handful of NIS-related programs, which probably have their own copies of that code anyway) How about programs that link to libm, but every symbol they use from libm happens to have been replaced by inline code? (I have seen this happen in real life.) Libraries that are linked against libpthread as a defensive measure for actual use of threads by their users, but only need the stubs in libc for that? (Can causes severe performance hits for single-threaded users of that library.) zw -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Le Sunday 09 December 2007 09:56:39 Zack Weinberg, vous avez écrit : I'm curious what prompted your message. Did a program you use actually break? What was the failure mode? Yes, I agree too. I've been using this flag recently, and in most cases it just does what's required... I think that before warning in capital letters it would be good to study and advertise the case when it's potentially harmfull.. There's a good reference there: http://www.gentoo.org/proj/en/qa/asneeded.xml Romain
Re: -Wl,--as-needed considered possibly harmful
Le dimanche 09 décembre 2007 à 09:03 +0100, Simon Richter a écrit : Hi, due to the recent dpkg-shlibdeps changes, people have started adding -Wl,--as-needed into their LDFLAGS. THIS IS NOT A GOOD IDEA. Of course it is. This is the most reliable way of dropping unneeded dependencies. All other ways will keep some of them. If there are bugs in --as-needed, we need to fix them. It is much easier than fixing thousands of packages. Or are you willing to drop dh_fixperms and fix thousands of broken packages by hand as well? If there are broken scripts adding too many libraries then it is time to fix those scripts, not employ an evil hack that makes the symptoms go away. Some of these scripts cannot be “fixed”. And for others, well, good luck in getting libtool’s upstream to integrate the required changes. Maybe they will be here in 5 years, and in 15 years all upstreams will have migrated to a recent enough version. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: -Wl,--as-needed considered possibly harmful
* Josselin Mouette [EMAIL PROTECTED] [071209 18:48]: due to the recent dpkg-shlibdeps changes, people have started adding -Wl,--as-needed into their LDFLAGS. THIS IS NOT A GOOD IDEA. If there are broken scripts adding too many libraries then it is time to fix those scripts, not employ an evil hack that makes the symptoms go away. Some of these scripts cannot be ???fixed???. And for others, well, good luck in getting libtool???s upstream to integrate the required changes. Maybe they will be here in 5 years, and in 15 years all upstreams will have migrated to a recent enough version. Just curing the symptoms instead of the problems will not help to get there any sooner. There might be stuff where the stuff at hand is not easily fixed, but that is no reason to not consider it not worth doing. (or to consider using -Wl,--as-needed as anything short of giving in to a broken build system and putting the dirt under the carpet, so people do no longer get dirty feet walking around most of the time). Hochachtungsvoll, Bernhard R. Link -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Hi, Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit : Just curing the symptoms instead of the problems will not help to get there any sooner. What if there is no problem? For example, pkg-config --libs gtk+-x11-2.0 will return, among others, -lglib-2.0 and -lm. And this is perfectly intentional. The former is here because you can assume glib functions are available when doing #include gtk/gtk.h. The latter is here because some macros need math functions. Still, many binaries using it don’t need glib, and most of them don’t need libm. There is nothing that can be done to avoid that, the only way is a check that removes the dependency if it isn’t actually needed. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: -Wl,--as-needed considered possibly harmful
On Sun, Dec 09, 2007 at 07:23:00PM +0100, Josselin Mouette wrote: Hi, Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit : Just curing the symptoms instead of the problems will not help to get there any sooner. What if there is no problem? For example, pkg-config --libs gtk+-x11-2.0 will return, among others, -lglib-2.0 and -lm. And this is perfectly intentional. The former is here because you can assume glib functions are available when doing #include gtk/gtk.h. The latter is here because some macros need math functions. Still, many binaries using it don't need glib, and most of them don't need libm. There is nothing that can be done to avoid that, the only way is a check that removes the dependency if it isn't actually needed. I can't seem to find any example of a macro calling a math functions? I also can't find any header that includes math.h. I do see that libgtk-x11-2.0.so uses symbols from libm, like log10, atan2, sin, cos, tan, pow, ceil. But I can't find anything in the header files that would use them. It's gdk-pixbuf-2.0.pc and cairo.pc that have -lm in them. Anyway, I think it is bad style to use macro's in public header files that call functions from another library. It's also easy to replace them with real functions. If there are such functions being called I suggest you replace them. What might be harder is avoiding the includes from other libraries because you use types from the other library. It would be great if you didn't have to do that, and there are some ways to avoid that. Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Le Sunday 09 December 2007 20:03:27 Kurt Roeckx, vous avez écrit : Anyway, I think it is bad style to use macro's in public header files that call functions from another library. It's also easy to replace them with real functions. If there are such functions being called I suggest you replace them. What might be harder is avoiding the includes from other libraries because you use types from the other library. It would be great if you didn't have to do that, and there are some ways to avoid that. Ok, let's consider another very simple case: libshout allows to perform streaming of speex, vorbis, and theora formats. Hence, when asking for the libs to link with, you got -lspeex and -ltheora since it's needed to cover all build cases. However, most of the applications don't need to link against theora and speex. So, not only do I agree with Josselin, but I think that looking at each project to fullfill any possible use case IS the problem itself. we can't expect any upstream to commit and release every little hack that might solve this or that issue. The solution is, indeed, to link what's needed for each application and nothing else. Romain
Re: -Wl,--as-needed considered possibly harmful
Romain Beauxis [EMAIL PROTECTED] writes: Ok, let's consider another very simple case: libshout allows to perform streaming of speex, vorbis, and theora formats. Hence, when asking for the libs to link with, you got -lspeex and -ltheora since it's needed to cover all build cases. Why? Aren't you calling a function in libshout to stream such files? libshout should then link against libseepx and libtheora, but why should the *application* have to do so if it's using libshout? This sounds broken to me. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Ok, let's consider another very simple case: libshout allows to perform streaming of speex, vorbis, and theora formats. Hence, when asking for the libs to link with, you got -lspeex and -ltheora since it's needed to cover all build cases. However, most of the applications don't need to link against theora and speex. Doesn't libshout reference symbols from libspeex and libtheora? If it does, resulting binary must be linked against both these libraries. Unresolved symbols within binaries are really bad, because they will once be unexpectedly called (e.g. in some init or deinit or error-handling), and binary will crash, leaving user data unsaved. If it does not, then what for -lspeex and -ltheora are in the script output? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
On Mon, Dec 10, 2007 at 12:45:46AM +0300, Nikita V. Youshchenko wrote: Ok, let's consider another very simple case: libshout allows to perform streaming of speex, vorbis, and theora formats. Hence, when asking for the libs to link with, you got -lspeex and -ltheora since it's needed to cover all build cases. However, most of the applications don't need to link against theora and speex. Doesn't libshout reference symbols from libspeex and libtheora? If it does, resulting binary must be linked against both these libraries. Where binary means libshout, yes. The application does not directly use libspeex and libtheora so should not have any undefined symbols that are defined in either libspeex or libtheora. Unresolved symbols within binaries are really bad, because they will once be unexpectedly called (e.g. in some init or deinit or error-handling), and binary will crash, leaving user data unsaved. If it does not, then what for -lspeex and -ltheora are in the script output? That is only useful for linking to a static version of libshout. Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: -Wl,--as-needed considered possibly harmful
Le Sunday 09 December 2007 22:45:46 Nikita V. Youshchenko, vous avez écrit : However, most of the applications don't need to link against theora and speex. Doesn't libshout reference symbols from libspeex and libtheora? Yes If it does, resulting binary must be linked against both these libraries. Unresolved symbols within binaries are really bad, because they will once be unexpectedly called (e.g. in some init or deinit or error-handling), and binary will crash, leaving user data unsaved. I think you are confusing the issue. Yes, libshout uses symbols from libspeex: objdump -x /usr/lib/libshout.so | grep NEEDED | grep speex NEEDED libspeex.so.1 However, not all applications uses those symbols in their current code, so they do not need to be linked against speex themselves. Of course, correct dependencies will be pulled when requesting libshout. Romain
Re: -Wl,--as-needed considered possibly harmful
Steve Langasek wrote: ... also, -Wl,--as-needed is *not* a complete solution for the problems caused by generating extra -l arguments. Every -lfoo option passed to the linker requires the corresponding -dev package to be installed at build time; while it is a bug if the -dev packages don't declare their dependencies, the impact of such bugs (which do happen fairly often) would be limited to static linking if appropriately-constructed .pc files were available. Note that pkgconfig is not the only problem here. Some extra links arise from the fact that several utilities or plugins for an application are built with the same environment (ie: the same flags), and not all of them require all the linked libraries. Of course, one could argue that fixing the build script is the way to go, but sometimes it is not trivial to do that (badly crafted or unmaintained build scripts tend to be quite ugly). -- Felipe Sateler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
