Re: Summary of C++ symbols experience (was: Do symbols make sense for C++)
On Sáb 28 Ene 2012 02:28:25 Russ Allbery escribió: [snip] 2. Build and upload this version of the package. Now wait for all the buildds to fail (because they will, on probably nearly every other architecture than your local one). Sometimes we may expect FTBFSs due to changes in other archs toolchains not catched by the local arch, like missing symbols. This is when this may come handy: override_dh_makeshlibs: dh_makeshlibs -- -c0 If in your local arch you couldn't find missing symbols that create and ABI change, then chances that missing symbols in other archs are safe to ignore. But take into account that I'm writing all this out of pure empiric trials, which are clearly not enough to assert it. Kinds regards, Lisandro. -- Wiki participants are, by nature, a pedantic, ornery, and unreasonable bunch. So there's a camaraderie here we seldom see outside of our professional contacts. http://www.c2.com/cgi/wiki?WhyWikiWorks Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/ signature.asc Description: This is a digitally signed message part.
Re: Do symbols make sense for C++
* Russ Allbery r...@debian.org [120127 22:30]: Symbol files are nice if you have a upstream who doesn't always take binary compatibility that serious (which is probably the case for 75% of upstreamers). Then you have a list that helps you finding out. This is probably the best summary of why a symbols file might be useful. It helps catch cases where ABI compatibility is not maintained without people being aware that this broke. On the other hand a symbols file can also make situations worse if the ABI changes and dpkg-gensymbols does not catch this case, as the dependencies are less strict then. (and then of course there is the usual bits and pieces about why symbol files are useful) I think this is probably the summary of my previous message: a lot of those places don't seem as useful for C++ as for C, because ABI compatibility is a lot more complex (and in some ways a lot harder to maintain), so the equivalent of a shlibs bump is probably more frequently the right thing to do. I think this argument can also mean that symbols files are more useful for C++ than for C: For C there are many ways to break the ABI incompatibly and compatibly without anything visible at symbol level. As C++ is more likely to show differences at symbol level in this cases the danger of cases where dpkg-symbols introduces wrong dependencies might be lower with C++. Bernhard R. Link -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120128124957.ga2...@server.brlink.eu
Re: Summary of C++ symbols experience (was: Do symbols make sense for C++)
On 2012-01-28, Russ Allbery r...@debian.org wrote: 5. It's still not clear that the benefit is worth the amount of effort, since I expect most C++ libraries to require frequent SONAME changes anyway, which means that the long-term binary compatibility angle of Qt has kept binary compatibility since 4.0 was released in 2005. (and 3.0 kept binary compatibility during the 3.0 series) kdelibs has kept binary compatibility since 4.0 was released in 2007 so I wouldn't as such call it 'frequent SONAME changes', but it do require carefulness and commitment from upstream to do it. /Sune -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnji7gl8.p7v.nos...@sshway.ssh.pusling.com
Re: Do symbols make sense for C++
Bernhard R. Link brl...@debian.org writes: * Russ Allbery r...@debian.org [120127 22:30]: This is probably the best summary of why a symbols file might be useful. It helps catch cases where ABI compatibility is not maintained without people being aware that this broke. On the other hand a symbols file can also make situations worse if the ABI changes and dpkg-gensymbols does not catch this case, as the dependencies are less strict then. But this is true regardless; the same is true for shlibs. Either way, if there was an incompatible change that the Debian maintainer didn't catch that doesn't affect the exported symbols, you potentially lose. shlibs does make it easier to just always bump the dependency to require a matching upstream version, but you could in theory do the same thing with symbols. I think this is probably the summary of my previous message: a lot of those places don't seem as useful for C++ as for C, because ABI compatibility is a lot more complex (and in some ways a lot harder to maintain), so the equivalent of a shlibs bump is probably more frequently the right thing to do. I think this argument can also mean that symbols files are more useful for C++ than for C: For C there are many ways to break the ABI incompatibly and compatibly without anything visible at symbol level. As C++ is more likely to show differences at symbol level in this cases the danger of cases where dpkg-symbols introduces wrong dependencies might be lower with C++. It's true that C++ is much more likely to show symbol differences on at least some types of changes (such as changing the function signature, which with C++ is encoded in the symbol). However, C++ also introduces a significant number of new ways to break compatibility that just don't exist with C, some of which don't affect the symbols. But, more to the point for practical concerns, there appears to be a truly staggering amount of noise. What I'm seeing in these experiments is that trying to track symbols for C++ libraries, at least for ones that aren't using symbol export control, is that there are huge numbers of false positives and spurious differences from changes in platform and compiler, to the extent that one is awash in fixing symbol differences that don't represent any interesting property of the library but are just odd artifacts of how C++ is compiled. I think, given this experience, that symbols for C++ libraries, at least for the ones that I'm looking at, are a failed experiment, and I'm going to back out of all of these changes and go back to just using shlibs. That seems particularly indicated in this case, since upstream changes SONAMEs for these libraries regularly and hence it's very rare for it to be even possible to attempt running stable binaries with unstable libraries or vice versa. There has not yet been a case where stable and oldstable had the same SONAME for one of these libraries. That eliminates much of the advantage of symbols; the remaining advantage is detecting ABI breaks, but since upstream changes SONAME regularly, most of the damage from ABI breaks would be limited in this case anyway. Somewhat more controversially, so I'd like to get a debian-devel feeling on this, I think the Lintian info tag for a library using shlibs instead of symbols should be suppressed if the library is a C++ library (probably most easily detected by seeing if it depends on libstdc++). There *are* excellent tools from the KDE team available, but even with those tools, this turned into something I wasn't willing to deal with, and I'm a fairly experienced packager. I have a hard time imagining the average Debian packager is going to be equipped to deal with this properly, and for most C++ libraries just using shlibs and dh_makeshlibs -V to conservatively set the dependencies is probably the best tradeoff of usability against maintainer time. If packagers are willing to do more, more power to them! Some C++ libraries may be amenable to it. But a Lintian tag implies that packagers who are not doing this are doing something (minorly) wrong, and I don't think that's actually the case. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87k44bo169@windlord.stanford.edu
Re: Do symbols make sense for C++
On Thu, Jan 26, 2012 at 11:49:06PM +, Sune Vuorela wrote: On 2012-01-26, Thomas Weber twe...@debian.org wrote: I'm currently creating symbol files for Octave in Debian. And quite frankly, the way symbol files for C++ libraries are handled and (especially) documented is totally frustrating. There's exactly zero precise documentation on how to maintain symbol files there. When the topic is brought up on mailing list, people point at some tools from the KDE packages, but still there's no documentation on the problem itself. The best example is dpkg-gensymbols(1) itself: did you read the link I posted? That's currently the best docs we have for pkgkde-symbolshelper. Better docs is always a nice thing to have. Please help improve it. I have read the link, but that's not the issue. I want to understand the problem and after that I can think about a tool to use. As an example, the policy tells you what a package must look like - it doesn't just document debhelper. In other words, I'm lacking the knowledge for writing better documentation for pkgkde-symbolhelper. Yeah, great. Which instantiations cannot be marked as optional and how do I recognize them? And if it's actually impossible to maintain symbol files for C++ libraries (as Florian Weimer has claimed in this thread), why doesn't the manpage just say so? It is not at all impossible to maintain symbol files for c++ libraries. I surely hope so, I have already spent quite some time on getting them in shape for Octave ;). That doesn't change the fact that I'm less than sure about some of the things I'm doing there. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120127140552.GA14865@t61
Re: Do symbols make sense for C++
Sune Vuorela nos...@vuorela.dk writes: On 2012-01-26, Russ Allbery r...@debian.org wrote: But that feels like a result contrary to what I had previously thought was the intended direction, so I wanted to ask the Debian development community as a whole: am I missing something? Are these symbols files actually useful? Symbol files are nice if you have a upstream who doesn't always take binary compatibility that serious (which is probably the case for 75% of upstreamers). Then you have a list that helps you finding out. This is probably the best summary of why a symbols file might be useful. It helps catch cases where ABI compatibility is not maintained without people being aware that this broke. (and then of course there is the usual bits and pieces about why symbol files are useful) I think this is probably the summary of my previous message: a lot of those places don't seem as useful for C++ as for C, because ABI compatibility is a lot more complex (and in some ways a lot harder to maintain), so the equivalent of a shlibs bump is probably more frequently the right thing to do. You miss pkgkde-symbolshelper written by the fabulous Modestas Vainius which is available in pkg-kde-tools. http://pkg-kde.alioth.debian.org/symbolfiles.html I've gone back and built symbols files for two of my libraries using these tools, and indeed, they are excellent. They handle marking template symbols as optional and decoding some of the symbols with c++ tags, as well as doing all the other tedious work that I ran into with creating the symbols file. I'm going to try maintaining these libraries a while using the tools from pkg-kde-tools and see how it goes. I highly recommend anyone else who wants to look at symbols files for C++ also look at them. (It might be a good idea to eventually move these tools into devscripts or some other place that doesn't sound KDE-specific and would be easier for the general Debian developer to find.) -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87mx98q1dv@windlord.stanford.edu
Summary of C++ symbols experience (was: Do symbols make sense for C++)
Okay, I've spent parts of another couple of days working on this using the pkg-kde-tools infrastructure, and I think I can draw some more conclusions. First of all, for those who haven't explored the pkg-kde-tools infrastructure for this, it looks like the effective process goes something like this: 1. Do a local build of the package and then generate an initial symbols template for that architecture. So, for example, I did: fakeroot debian/rules build install pkgkde-gensymbols -plibxml-security-c16 -v1.6.1 -Osymbols.i386 \ -edebian/libxml-security-c16/usr/lib/*/libxml-security-c.so.*.* pkgkde-symbolshelper create -o debian/libxml-security-c16.symbols \ -v 1.6.1 symbols.i386 This generates an initial symbols file that will work for i386. 2. Build and upload this version of the package. Now wait for all the buildds to fail (because they will, on probably nearly every other architecture than your local one). 3. Download all of the build logs and feed them back into pkgkde-symbolshelper so that it can adjust your symbols template for all of the architecture variations. So, for example, I did: pkgkde-getbuildlogs pkgkde-symbolshelper batchpatch -v 1.6.1 *_unstable_logs/*.build This will generate rather nice annotated symbols files with appropriate arch lists and using subst tags where appropriately (such as when an argument to a function is a size_t). 4. Modify your package to use pkg-kde-tools during the build, because subst tags are a really nice way to handle a lot of C++ symbol patterns but #533916 against dpkg-dev is wontfix, so you need the pkg-kde-tools replacement for dpkg-gensymbols. I added a dependency on pkg-kde-tools and then added the (undocumented) pkgkde_symbolshelper add-on to my dh --with flag. 5. Build and upload the package again. Now it will hopefully build on all architectures. This is, as you might expect, a fair bit of work, but it *does* work, and what comes out the other end is a symbols file that works across the current buildd architectures and detects any mistakes by upstream that change the symbols exported. However, it does have some problems, some obvious, and some more subtle. Here's a list of the issues that I see: 1. It feels like this symbols file is still likely to be fragile. While pkg-kde-tools detects a *lot* of template instances and marks them optional, I was still seeing a lot of leaked symbols from other C++ libraries that my package build-depends on, and which I suspect may also disappear. There's also the problem of optimizers eliminating some things that are inlined; for example, that appears to be the variation that my package sees on arm. And for a substantial C++ library, the symbols file is HUGE. Over 12,000 lines for one of my libraries. It's kind of scary to think of how many fragility landmines are buried under there. 2. By the nature of this process, it's very sensitive to the current supported architectures. I expect any package with a symbols template built this way stands a good chance of FTBFS on a new architecture. For example, my symbols file has a lot of !armel !armhf patterns, which don't appear to be anything unique to arm except that the C++ compiler behaves slightly differently there due to inlining, and those are the only two architectures in the current set with that pattern. 3. The lack of dpkg-gensymbols support for subst is very unfortunate, since it's used by the pkg-kde-tools utilities and it's the Right Thing To Do for symbols that take, for example, size_t as an argument. I would strongly encourage the dpkg maintainers to accept the work in #533916 except maybe for the vt work, since the alternative is to generate regexes that are significantly worse at capturing mistakes (since they're going to accept either int or long, rather than only whatever one corresponds to size_t, for example), or to embed a lot of specific architecture restrictions listing, for instance, all 64-bit architectures and guarantee FTBFS for every new architecture. 4. It's a little hard on the infrastructure, since adding a symbols file requires uploading a package that's guaranteed to fail to build almost everywhere just so that you can get all the build logs to feed into the machinery. I always feel guilty about doing that. 5. It's still not clear that the benefit is worth the amount of effort, since I expect most C++ libraries to require frequent SONAME changes anyway, which means that the long-term binary compatibility angle of symbols is probably futile. Mostly, all of this is to give you a tool to do ABI checking, which will catch some mistakes but not all of them. And I'm guessing it's going to be substantially more work than maintaining symbols for a C library if one actually does it properly and checks every missing symbol
Re: Do symbols make sense for C++
On Thu, Jan 26, 2012 at 08:58:55AM +0100, Raphael Hertzog wrote: Hi, On Wed, 25 Jan 2012, Russ Allbery wrote: 4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file? The main benefit of the symbols file is that new symbols (API extension) are listed with the version where they have been introduced. So that seriously reduces the number of packages which would be affected by the equivalent shlibs bump. Otherwise I agree with you, maintainers of C++ libs are unlikely to be able to (preemptively) detect when a given symbol should have had its version bumped... so you should really trust upstream and hope to catch API extension that preserve ABI in the upstream changelog. 5. The exported symbols of the library contained many symbols that obviously weren't really from that library, but instead were artifacts of the C++ compilation process, things like instantiations of std::vector. Do those go into the symbols file? Do they change from architecture to architecture? If they disappear again, is that actually an ABI break? How do I know? It's all very mysterious, and while shlibs provides the same semantics as just ignoring this, at least I'm not then including in the package data, generated by me, things that I'm just blindly ignoring. It's also very mysterious to me. :) But Modestas Vainius has still managed to maintain symbols files for many KDE libraries. He might be able to tell you more. Chances are KDE libraries are using visibility flags when compiling, which limits the number of exported symbols. Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120126080540.ga23...@glandium.org
Re: Do symbols make sense for C++
On Wed, Jan 25, 2012 at 10:53:04PM -0800, Russ Allbery wrote: I'm currently working on the Policy modification to document (and recommend) use of symbols instead of shlibs, but I'd only personally used symbols with C libraries. Today I decided that I should try adding a symbols file to a C++ library, particularly if I'm going to recommend everyone do it. I tried this exercise with xml-security-c, which is, I think, a reasonably typical C++ library. Not the sort of core C++ library that would sit at the center of the distribution, but a random software package that's in Debian because other things use it. The experience was rather interesting, and I ended up uploading the new version without a symbols file and continuing to just use shlibs. That's for the following reasons: 1. The generated symbols file was HUGE. Hundreds of lines. This is a marked difference from the typical C symbols file, which is of quite manageable size. Some of that is that the library provides a lot of different classes, but some of it is that C++ just generates a lot of exported symbols. There's no way that I could do what I would do with a C library and understand those symbols, why they're there, and whether they are likely to have changed between revisions. 2. Generating a reasonable symbols file was a pain. Generating an unreasonable symbols file that just contains all of the mangled symbols is largely mechanical and uninteresting, but that symbols file doesn't seem to me to convey useful information. So I did some scripting to translate the symbols back with c++filt, and add (c++) tags, and then try to understand what I was looking at and figure out whether I should sort the symbols list because the default sort is by mangled name, which is meaningless. This is a rather unappealing process. It's not particularly difficult, but it's very awkward and feels like it's missing vital tools. 3. The resulting symbols file is incomprehensible to someone without strong knowledge of C++. It's full of opaque entries that don't make sense to the non-C++ programmer, wihch I suspect is a substantial number of people who package C++ libraries for Debian. I know enough C++ from school that I can evaluate security fixes, make simple patches, and review upstream changes, and I think that's all that should be needed to package things for Debian. But I'm deeply uncomfortable producing a symbols file on my own that contains entries for things that I know nothing about and cannot evaluate when they've last changed, like non-virtual thunk to FooClass::~FooClass@Base. 4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file? 5. The exported symbols of the library contained many symbols that obviously weren't really from that library, but instead were artifacts of the C++ compilation process, things like instantiations of std::vector. Do those go into the symbols file? Do they change from architecture to architecture? If they disappear again, is that actually an ABI break? How do I know? It's all very mysterious, and while shlibs provides the same semantics as just ignoring this, at least I'm not then including in the package data, generated by me, things that I'm just blindly ignoring. 6. Different levels of optimizations lead to different exported symbols, depending on whether some things are inlined or not. Different architectures may also have different exported symbols. Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120126075029.ga22...@glandium.org
Re: Do symbols make sense for C++
On 2012-01-26, Russ Allbery r...@debian.org wrote: But that feels like a result contrary to what I had previously thought was the intended direction, so I wanted to ask the Debian development community as a whole: am I missing something? Are these symbols files actually useful? Symbol files are nice if you have a upstream who doesn't always take binary compatibility that serious (which is probably the case for 75% of upstreamers). Then you have a list that helps you finding out. (and then of course there is the usual bits and pieces about why symbol files are useful) You do need a bit of knowledge, especially when some of the symbols you mentioned go missing (like the std::vector ones), which isn't normally a abi break. Am I missing some trick to make them useful? You miss pkgkde-symbolshelper written by the fabulous Modestas Vainius which is available in pkg-kde-tools. http://pkg-kde.alioth.debian.org/symbolfiles.html /Sune -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnji22f7.p7v.nos...@sshway.ssh.pusling.com
Re: Do symbols make sense for C++
Raphael Hertzog hert...@debian.org writes: On Wed, 25 Jan 2012, Russ Allbery wrote: 4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file? The main benefit of the symbols file is that new symbols (API extension) are listed with the version where they have been introduced. So that seriously reduces the number of packages which would be affected by the equivalent shlibs bump. My impression is that it's more difficult (not impossible, but more difficult) to maintain a C++ library in that fashion the way that one would maintain a C library, where one adds new symbols without requiring an SONAME change. Certainly, with the three C++ libraries that I maintain, upstream has a policy of not even attempting to do this. Bug fixes are made in an ABI-compatible way and should not, as I understand it, involve anything that would change the symbols file, and all other releases change the SONAME. I certainly believe there are carefully-maintained C++ libraries that are more conservative and that go to the required effort to make this work properly, and KDE is probably an excellent example of a large C++ project with enough C++ experts involved to do this properly. But I'm wondering how many packages we really have like that. By all means, if the library makes that possible, maintain symbols! But the library I looked at is fairly typical, I think, in that the maintainers are not deeply attuned to maintaining consistency of the export list in anything other than patch releases. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fwf2x4uu@windlord.stanford.edu
Re: Do symbols make sense for C++
On 26/01/2012 09:08, Sune Vuorela wrote: On 2012-01-26, Russ Allbery r...@debian.org wrote: But that feels like a result contrary to what I had previously thought was the intended direction, so I wanted to ask the Debian development community as a whole: am I missing something? Are these symbols files actually useful? Symbol files are nice if you have a upstream who doesn't always take binary compatibility that serious (which is probably the case for 75% of upstreamers). Then you have a list that helps you finding out. (and then of course there is the usual bits and pieces about why symbol files are useful) You do need a bit of knowledge, especially when some of the symbols you mentioned go missing (like the std::vector ones), which isn't normally a abi break. About what i wonder (since i'm not c++ expert) if abi-compliance-checker is serious enough to be used to decide soname bumps ? http://www.upstream-tracker.org/ Regards, Jérémy. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f210e27.5070...@melix.org
Re: Do symbols make sense for C++
On Do, Jan 26, 2012 at 09:26:15 (CET), Jérémy Lal wrote: [...] About what i wonder (since i'm not c++ expert) if abi-compliance-checker is serious enough to be used to decide soname bumps ? http://www.upstream-tracker.org/ Thank you very much for pointing out this service. It surely is very useful for both package maintainers as well as upstreams. I've analyzed the report for libav: http://www.upstream-tracker.org/versions/libav.html and found the results very interesting and enlightning. The results themselves seem very detailed and accurate to me but I think their interpretations of the results is too strong. For libav, most of the incompatible changes are covered by API usage constraints which this automated service cannot possibly be aware of. This means that you have to carefully read all of the report and use the findings with a grain of salt. Cheers, Reinhard -- Gruesse/greetings, Reinhard Tartler, KeyID 945348A4 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87vcny4z7q@faui43f.informatik.uni-erlangen.de
Re: Do symbols make sense for C++
On 26/01/2012 14:53, Russ Allbery wrote: 5. The exported symbols of the library contained many symbols that obviously weren't really from that library, but instead were artifacts of the C++ compilation process, things like instantiations of std::vector. Do those go into the symbols file? Do they change from architecture to architecture? If they disappear again, is that actually an ABI break? How do I know? It's all very mysterious, and while shlibs provides the same semantics as just ignoring this, at least I'm not then including in the package data, generated by me, things that I'm just blindly ignoring. Most upstream libraries don't usually have any std:: symbols that are meant to be exported. In the case of libzen and libmediainfo which I recently packaged, I just added the following to the bottom of my symbols file to ignore them: (optional|c++|regex)^std:: 0.7.52 (optional|c++|regex)^(bool|void|MediaInfoLib::File_Mxf::indextable\*) std:: 0.7.52 What I found most frustrating, though, was the size_t handling, which ends up being expanded into unsigned int on some architectures, and unsigned long on others, resulting in the same function (which takes a size_t parameter) being mangled differently per arch. I used the following workaround for size_t using functions: (c++|regex)MediaInfoLib::MediaInfo::Count_Get\(MediaInfoLib::stream_t, unsigned (int|long)\)@Base 0.7.52 And then there's also long double/__float128 (powerpc), which I worked around in the same manner. -- Kind regards, Loong Jin signature.asc Description: OpenPGP digital signature
Re: Do symbols make sense for C++
On 2012-01-26, Jérémy Lal kapo...@melix.org wrote: About what i wonder (since i'm not c++ expert) if abi-compliance-checker is serious enough to be used to decide soname bumps ? http://www.upstream-tracker.org/ The abi checker that iirc is hosted somewhere underneath linuxfoundation is producing quite good results. I can't find a url for it right now though. Or the url I found was down for maintenance. It's a giant perl script with various helper things. /Sune -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnji25h8.p7v.nos...@sshway.ssh.pusling.com
Re: Do symbols make sense for C++
On 2012-01-26, Russ Allbery r...@debian.org wrote: I certainly believe there are carefully-maintained C++ libraries that are more conservative and that go to the required effort to make this work properly, and KDE is probably an excellent example of a large C++ project with enough C++ experts involved to do this properly. But I'm wondering Ha. it is only libraries from src:kde4libs and src:kdepimlibs that is properly managed in this regard, and not the only source packages producing shared libraries consumed by other source packages. /Sune -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnji25m3.p7v.nos...@sshway.ssh.pusling.com
Re: Do symbols make sense for C++
On 2012-01-26, Chow Loong Jin hyper...@debian.org wrote:
(c++|regex)MediaInfoLib::MediaInfo::Count_Get\(MediaInfoLib::stream_t,
unsigned (int|long)\)@Base 0.7.52
(subst)MediaInfoLib::MediaInfo::Count_Get(MediaInfoLib::stream_t
{size_t})@base 0.7.52
note that you might actually want to use the mangled symbol names in
your symbol file, given that all the C0, C1 and C2 contsructors
demangles to the same symbol.
/
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/slrnji25tb.p7v.nos...@sshway.ssh.pusling.com
Re: Do symbols make sense for C++
On 26/01/2012 17:06, Sune Vuorela wrote:
On 2012-01-26, Chow Loong Jin hyper...@debian.org wrote:
(c++|regex)MediaInfoLib::MediaInfo::Count_Get\(MediaInfoLib::stream_t,
unsigned (int|long)\)@Base 0.7.52
(subst)MediaInfoLib::MediaInfo::Count_Get(MediaInfoLib::stream_t
{size_t})@base 0.7.52
That feature has not been implemented, and its bug[1] has been tagged wontfix.
note that you might actually want to use the mangled symbol names in
your symbol file, given that all the C0, C1 and C2 contsructors
demangles to the same symbol.
C0, C1 and C2 constructors? What are those?
[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=533916
--
Kind regards,
Loong Jin
signature.asc
Description: OpenPGP digital signature
Re: Do symbols make sense for C++
On 2012-01-26, Chow Loong Jin hyper...@debian.org wrote:
(subst)MediaInfoLib::MediaInfo::Count_Get(MediaInfoLib::stream_t
{size_t})@base 0.7.52
That feature has not been implemented, and its bug[1] has been tagged won=
tfix.
it's available in pkg-kde-tools as mentioned at other places in this
thread.
C0, C1 and C2 constructors? What are those?
ctor-dtor-name ::= C1 # complete object constructor
::= C2 # base object constructor
::= C3 # complete object allocating constructor
::= D0 # deleting destructor
::= D1 # complete object destructor
::= D2 # base object destructor
/Sune
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/slrnji29ld.p7v.nos...@sshway.ssh.pusling.com
Re: Do symbols make sense for C++
* Russ Allbery:
I'm currently working on the Policy modification to document (and
recommend) use of symbols instead of shlibs, but I'd only personally used
symbols with C libraries. Today I decided that I should try adding a
symbols file to a C++ library, particularly if I'm going to recommend
everyone do it. I tried this exercise with xml-security-c, which is, I
think, a reasonably typical C++ library.
Symbols just don't work for C++. Large parts of a library ABI are not
expressed in terms of exported symbols.
Let's start with this:
struct Foo {
long a;
virtual void foo();
};
Foo *
f()
{
return new Foo;
}
void
g(Foo *p)
{
p-foo();
}
It is compiled to:
.globl _Z1gP3Foo
.type _Z1gP3Foo, @function
_Z1gP3Foo:
.LFB4:
.cfi_startproc
movq(%rdi), %rax
movq(%rax), %rax
jmp *%rax
.cfi_endproc
.LFE4:
.size _Z1gP3Foo, .-_Z1gP3Foo
.p2align 4,,15
.globl _Z1fv
.type _Z1fv, @function
_Z1fv:
.LFB0:
.cfi_startproc
subq$8, %rsp
.cfi_def_cfa_offset 16
movl$16, %edi
call_Znwm
movq$_ZTV3Foo+16, (%rax)
addq$8, %rsp
ret
.cfi_endproc
.LFE0:
.size _Z1fv, .-_Z1fv
Now we threw new members to struct Foo and use Foo::bar() in g():
struct Foo {
long a, b, c;
virtual void foo();
virtual void bar();
};
Foo *
f()
{
return new Foo;
}
void
g(Foo *p)
{
p-bar();
}
This does not change the symbols at all:
.globl _Z1gP3Foo
.type _Z1gP3Foo, @function
_Z1gP3Foo:
.LFB4:
.cfi_startproc
movq(%rdi), %rax
movq8(%rax), %rax
jmp *%rax
.cfi_endproc
.LFE4:
.size _Z1gP3Foo, .-_Z1gP3Foo
.p2align 4,,15
.globl _Z1fv
.type _Z1fv, @function
_Z1fv:
.LFB0:
.cfi_startproc
subq$8, %rsp
.cfi_def_cfa_offset 16
movl$32, %edi
call_Znwm
movq$_ZTV3Foo+16, (%rax)
addq$8, %rsp
ret
.cfi_endproc
.LFE0:
.size _Z1fv, .-_Z1fv
Two constants change in an incompatible way, but the set of symbols
remains the same. Of course, this is similar to C code where the size
of structs is not covered by symbol versioning, either. For the Foo
struct itself, the size dependency can be hidden by only handing out
pointers to application code. For Foo's vtable (which grows in size
to accommodate the Foo::bar() method), this is more difficult, and
downright impossible if application code needs to derive from Foo and
actually needs the method to be virtual.
In some isolated cases, symbols may still make sense for C++ (e.g., if
the library ABI is really stable and all you need to detect is use of
newly added classes to bump dependencies), but I don't think they work
in general.
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762fy3cin@mid.deneb.enyo.de
Re: Do symbols make sense for C++
On Jue 26 Ene 2012 05:08:07 Sune Vuorela escribió: [snip] Am I missing some trick to make them useful? You miss pkgkde-symbolshelper written by the fabulous Modestas Vainius which is available in pkg-kde-tools. http://pkg-kde.alioth.debian.org/symbolfiles.html +1. As Sune said in other mail, only :kde4libs and src:kdepimlibs are normally properly managed. But there are lots of third-party libs we have to maintain and pkg-kde-tools have been very helpfull in this regard. Kinds regards, Lisandro. -- So long, and thanks for all the fish! The Hitchhickers guide to the Galaxy Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/ signature.asc Description: This is a digitally signed message part.
Re: Do symbols make sense for C++
* From: Russ Allbery, Wed, 25 Jan 2012 22:53:04 -0800 4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file? How many upstream are there that try to achieve ABI equivalence? Usually people only try to achieve ABI compability, i.e. programs compiled against the old version will work with the new one, but there is no garantee that programs compiled against the new version also work with the new one. (And in that case the maintainer has to manually increase the versions in the symbols file, just as they needed to increment the shlibs file before (unless they used auto-updating by using dh_makeshlibs -V without any version anyway)). Bernhard R. Link -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120126161155.ga22...@server.brlink.eu
Re: Do symbols make sense for C++
Bernhard R. Link brl...@debian.org writes: How many upstream are there that try to achieve ABI equivalence? Usually people only try to achieve ABI compability, i.e. programs compiled against the old version will work with the new one, but there is no garantee that programs compiled against the new version also work with the new one. I'm not sure. I'm not even sure that the libraries I'm looking at really do try to do this, although that's the impression that I've gotten. Usually a bug-fix-only policy will do this (although not necessarily). (And in that case the maintainer has to manually increase the versions in the symbols file, just as they needed to increment the shlibs file before (unless they used auto-updating by using dh_makeshlibs -V without any version anyway)). Right, that's the conclusion I reached. And it's easier to do that with shlibs than symbols. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zkdagw8s@windlord.stanford.edu
Re: Do symbols make sense for C++
On Thu, Jan 26, 2012 at 08:08:07AM +, Sune Vuorela wrote: Am I missing some trick to make them useful? You miss pkgkde-symbolshelper written by the fabulous Modestas Vainius which is available in pkg-kde-tools. http://pkg-kde.alioth.debian.org/symbolfiles.html I'm currently creating symbol files for Octave in Debian. And quite frankly, the way symbol files for C++ libraries are handled and (especially) documented is totally frustrating. There's exactly zero precise documentation on how to maintain symbol files there. When the topic is brought up on mailing list, people point at some tools from the KDE packages, but still there's no documentation on the problem itself. The best example is dpkg-gensymbols(1) itself: [topic is the 'optional' tag]: For example, most of C++ template instantiations fall into this category. Yeah, great. Which instantiations cannot be marked as optional and how do I recognize them? And if it's actually impossible to maintain symbol files for C++ libraries (as Florian Weimer has claimed in this thread), why doesn't the manpage just say so? Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120126230100.GA1677@t61
Re: Do symbols make sense for C++
On 2012-01-26, Thomas Weber twe...@debian.org wrote: I'm currently creating symbol files for Octave in Debian. And quite frankly, the way symbol files for C++ libraries are handled and (especially) documented is totally frustrating. There's exactly zero precise documentation on how to maintain symbol files there. When the topic is brought up on mailing list, people point at some tools from the KDE packages, but still there's no documentation on the problem itself. The best example is dpkg-gensymbols(1) itself: did you read the link I posted? That's currently the best docs we have for pkgkde-symbolshelper. Better docs is always a nice thing to have. Please help improve it. Yeah, great. Which instantiations cannot be marked as optional and how do I recognize them? And if it's actually impossible to maintain symbol files for C++ libraries (as Florian Weimer has claimed in this thread), why doesn't the manpage just say so? It is not at all impossible to maintain symbol files for c++ libraries. /Sune -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnji3pji.p7v.nos...@sshway.ssh.pusling.com
Do symbols make sense for C++
I'm currently working on the Policy modification to document (and recommend) use of symbols instead of shlibs, but I'd only personally used symbols with C libraries. Today I decided that I should try adding a symbols file to a C++ library, particularly if I'm going to recommend everyone do it. I tried this exercise with xml-security-c, which is, I think, a reasonably typical C++ library. Not the sort of core C++ library that would sit at the center of the distribution, but a random software package that's in Debian because other things use it. The experience was rather interesting, and I ended up uploading the new version without a symbols file and continuing to just use shlibs. That's for the following reasons: 1. The generated symbols file was HUGE. Hundreds of lines. This is a marked difference from the typical C symbols file, which is of quite manageable size. Some of that is that the library provides a lot of different classes, but some of it is that C++ just generates a lot of exported symbols. There's no way that I could do what I would do with a C library and understand those symbols, why they're there, and whether they are likely to have changed between revisions. 2. Generating a reasonable symbols file was a pain. Generating an unreasonable symbols file that just contains all of the mangled symbols is largely mechanical and uninteresting, but that symbols file doesn't seem to me to convey useful information. So I did some scripting to translate the symbols back with c++filt, and add (c++) tags, and then try to understand what I was looking at and figure out whether I should sort the symbols list because the default sort is by mangled name, which is meaningless. This is a rather unappealing process. It's not particularly difficult, but it's very awkward and feels like it's missing vital tools. 3. The resulting symbols file is incomprehensible to someone without strong knowledge of C++. It's full of opaque entries that don't make sense to the non-C++ programmer, wihch I suspect is a substantial number of people who package C++ libraries for Debian. I know enough C++ from school that I can evaluate security fixes, make simple patches, and review upstream changes, and I think that's all that should be needed to package things for Debian. But I'm deeply uncomfortable producing a symbols file on my own that contains entries for things that I know nothing about and cannot evaluate when they've last changed, like non-virtual thunk to FooClass::~FooClass@Base. 4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file? 5. The exported symbols of the library contained many symbols that obviously weren't really from that library, but instead were artifacts of the C++ compilation process, things like instantiations of std::vector. Do those go into the symbols file? Do they change from architecture to architecture? If they disappear again, is that actually an ABI break? How do I know? It's all very mysterious, and while shlibs provides the same semantics as just ignoring this, at least I'm not then including in the package data, generated by me, things that I'm just blindly ignoring. I came away from this experience thinking that I should revise the Policy amendment to say that symbols files are really for C libraries and for C++ libraries with either a tightly maintained symbol export list or maintained by a C++ expert, and that most C++ library maintainers should just not bother with this and use shlibs, bumping the shlibs version or not based on their impression of how good upstream is at maintaining ABI equivalence. But that feels like a result contrary to what I had previously thought was the intended direction, so I wanted to ask the Debian development community as a whole: am I missing something? Are these symbols files actually useful? Am I missing some trick to make them useful? -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Do symbols make sense for C++
Hi, On Wed, 25 Jan 2012, Russ Allbery wrote: 4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file? The main benefit of the symbols file is that new symbols (API extension) are listed with the version where they have been introduced. So that seriously reduces the number of packages which would be affected by the equivalent shlibs bump. Otherwise I agree with you, maintainers of C++ libs are unlikely to be able to (preemptively) detect when a given symbol should have had its version bumped... so you should really trust upstream and hope to catch API extension that preserve ABI in the upstream changelog. 5. The exported symbols of the library contained many symbols that obviously weren't really from that library, but instead were artifacts of the C++ compilation process, things like instantiations of std::vector. Do those go into the symbols file? Do they change from architecture to architecture? If they disappear again, is that actually an ABI break? How do I know? It's all very mysterious, and while shlibs provides the same semantics as just ignoring this, at least I'm not then including in the package data, generated by me, things that I'm just blindly ignoring. It's also very mysterious to me. :) But Modestas Vainius has still managed to maintain symbols files for many KDE libraries. He might be able to tell you more. (He's also the one who implemented all the required support in dpkg-gensymbols) Cheers, -- Raphaël Hertzog ◈ Debian Developer Pre-order a copy of the Debian Administrator's Handbook and help liberate it: http://debian-handbook.info/liberation/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120126075855.gl18...@rivendell.home.ouaza.com
