Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Johannes Schauer (2014-07-07 13:51:00) we would like to propose an MBF with severity wishlist to drop unused build dependencies in a number of source packages I fixed many of the previous false positives of build dependencies on meta packages (not the file contents of the package are required for the build but its dependencies) which, if not present during build, would make the build not fail but just let the resulting binary be generated with less features. In contrast to the initial results, this means that 70 fewer build dependencies were found to be unneeded (25% of the original amount). The following binary packages were found to be meta packages and are thus not false positives in the result anymore: bison, ca-certificates, default-jdk, doxygen-latex, g++-4.8-multilib, gcc-multilib, gcj-jdk, gfortran, gir1.2-freedesktop, gir1.2-glib-2.0, g++-multilib, iasl, jadetex, libblas-dev, libboost-dev, libcorosync-dev, libdb-dev, libgmp3-dev, libgphoto2-2-dev, libkrb5-dev, libopenais-dev, libpcap-dev, libreadline-dev, libsonic-dev, libtasn1-3-bin, libtasn1-3-dev, libxi-dev, libxmlrpc-c3-dev, libxmu-dev, libxt-dev, lynx, mysql-server, portaudio19-dev, python, python2.7-dev, python3, python3-all, python3-all-dbg, python3-all-dev, python3-minimal, python-all, python-all-dbg, python-all-dev, python-gobject, ruby, ruby-dev, swig, tcl-dev, texinfo, texlive, texlive-latex-recommended, tk-dev, valac, x11-utils, zip The classification is done automatically (without a blacklist) based on their debtags (role::metapackage or role::dummy) or whether they ships any regular file except for those in /usr/share/doc. If they were found to be a meta package by this method, then they will be considered to contain the files of the binary packages that were used to satisfy their dependencies. indicated by the attached two text files and the dd-list output. I attached the updated list of droppable build dependencies and build dependencies that can be moved to Build-Depends-Indep together with dd-list output. The results exclude python-defaults and python3-defaults (as requested by Scott Kitterman) and openldap (as requested by Ryan Tandy) and lirc (as requested by Stefan Lippers-Hollmann). Thank you Scott, Ryan and Stefan for already having fixed your packaging! Here is the updated MBF template email --%--- Subject: Please consider removing the build dependencies on $foo, $bar and $baz Severity: wishlist Usertag: unusedbd User: bootst...@lists.debian.org Dear Maintainer, the following build dependencies of this source package do not seem to be needed during an --arch-all build with sbuild: - $foo - $bar And the following do not seem to be needed during a --no-arch-all build with sbuild: - $baz Neither are any of their files accessed during the respective build nor are their dependencies on other binary packages required. This can mean that either one of those build dependencies 1. is useless old cruft 2. is architecture dependent and does not occur on amd64 but on other architectures 3. indicates a bug in the packaging where you intend to use the build dependency but where that is not happening in practice during the build 4. is a false positive of a dependency which is used as a meta package (only its dependencies are used but not its files) but not detected as one and for which the build does not fail if it is not installed Depending on the case, you should 1. remove the build dependency (if it was found in a --arch-all build) or move it to Build-Depends-Indep (if it was found in a --no-arch-all build) 2. add an architecture restriction 3. fix your packaging to make use of the build dependency 4. make the build fail when the contents of the binary package are not found (for example by passing an explicit ./configure flag) or report the problem to j.scha...@email.de (reply to this bugreport) so that this false positive is not reported again You can find more detail about the procedures that were used to find this problem in the MBF announcement on debian-devel: $email --%--- Is it missing anything? Thank you! cheers, josch == apache2_2.4.9-1.arch-all.unusedbd == libcap-dev:amd64=1:2.22-1.2 == apparmor_2.8.0-5.arch-all.unusedbd == dejagnu=1.5-3 texlive-latex-recommended=2014.20140528-3 == apr-util_1.5.3-2.arch-all.unusedbd == libpcre3-dev:amd64=1:8.31-5 python=2.7.6-2 == apt_1.0.3.arch-all.unusedbd == automake=1:1.14.1-3 == at-spi2-atk_2.10.2-3.arch-all.unusedbd == libglib2.0-bin=2.40.0-3 == bc_1.06.95-8.arch-all.unusedbd == bison=2:3.0.2.dfsg-2 == bison_3.0.2.dfsg-2.arch-all.unusedbd == autotools-dev=20140510.1 == blcr_0.8.5-2.1.arch-all.unusedbd == autoconf=2.69-6 automake=1:1.14.1-3 autotools-dev=20140510.1 libtool=2.4.2-1.7 == bluez_4.101-4.1.arch-all.unusedbd == bison=2:3.0.2.dfsg-2
Re: possible MBF: automatically detecting unused build dependencies
On Mon, 28 Jul 2014 11:34:24 +0200, Johannes Schauer wrote: I attached the updated list of droppable build dependencies and build dependencies that can be moved to Build-Depends-Indep together with dd-list output. == libxml-parser-perl_2.41-1.arch-all.unusedbd == sharutils=1:4.14-2 Already fixed in 2.41-2. I assume you're planning to do a new run before actually filing the bugs? Cheers, gregor -- .''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06 : :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/ `. `' Member of VIBE!AT SPI, fellow of the Free Software Foundation Europe `- BOFH excuse #129: The ring needs another token -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140728094514.gk16...@colleen.colgarra.priv.at
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting gregor herrmann (2014-07-28 11:45:14) == libxml-parser-perl_2.41-1.arch-all.unusedbd == sharutils=1:4.14-2 Already fixed in 2.41-2. thanks! I assume you're planning to do a new run before actually filing the bugs? I cannot do a new run before September because I'm moving places twice within the next month and thus do not have a stable always-on machine available during that time. The only thing that could change this would be if I found easily accessible compute time elsewhere (I asked at debian-qa@l.d.o: http://lists.debian.org/20140726090503.4150.56356@hoothoot ) I thought that hardly any build dependencies get removed over time so that it would still be appropriate to fill bugreports for the June results now. If that would not be appreciated then I'll re-run the whole thing once I settled in at our new place some time in September. cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140728100758.4150.21535@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
Hi, I cannot believe I attached the wrong list once again. My sincere apologies to fill up your inboxes like that :( Attached are the right files and dd-list Quoting Johannes Schauer (2014-07-28 11:34:24) bison, ca-certificates, default-jdk, doxygen-latex, g++-4.8-multilib, gcc-multilib, gcj-jdk, gfortran, gir1.2-freedesktop, gir1.2-glib-2.0, g++-multilib, iasl, jadetex, libblas-dev, libboost-dev, libcorosync-dev, libdb-dev, libgmp3-dev, libgphoto2-2-dev, libkrb5-dev, libopenais-dev, libpcap-dev, libreadline-dev, libsonic-dev, libtasn1-3-bin, libtasn1-3-dev, libxi-dev, libxmlrpc-c3-dev, libxmu-dev, libxt-dev, lynx, mysql-server, portaudio19-dev, python, python2.7-dev, python3, python3-all, python3-all-dbg, python3-all-dev, python3-minimal, python-all, python-all-dbg, python-all-dev, python-gobject, ruby, ruby-dev, swig, tcl-dev, texinfo, texlive, texlive-latex-recommended, tk-dev, valac, x11-utils, zip and ignore this list. :( Sorry... josch == apache2_2.4.9-1.arch-all.unusedbd.real == libcap-dev:amd64=1:2.22-1.2 == apparmor_2.8.0-5.arch-all.unusedbd.real == dejagnu=1.5-3 texlive-latex-recommended=2014.20140528-3 == apr-util_1.5.3-2.arch-all.unusedbd.real == libpcre3-dev:amd64=1:8.31-5 == at-spi2-atk_2.10.2-3.arch-all.unusedbd.real == libglib2.0-bin=2.40.0-3 == bc_1.06.95-8.arch-all.unusedbd.real == bison=2:3.0.2.dfsg-2 == bison_3.0.2.dfsg-2.arch-all.unusedbd.real == autotools-dev=20140510.1 == blcr_0.8.5-2.1.arch-all.unusedbd.real == autoconf=2.69-6 automake=1:1.14.1-3 autotools-dev=20140510.1 libtool=2.4.2-1.7 == bluez_4.101-4.1.arch-all.unusedbd.real == bison=2:3.0.2.dfsg-2 gstreamer-tools=0.10.36-1.2 == boost-defaults_1.55.0.2.arch-all.unusedbd.real == libboost1.55-dev:amd64=1.55.0+dfsg-1 == ceph_0.80.1-1.arch-all.unusedbd.real == uuid-runtime=2.20.1-5.7 == chicken_4.8.0.5-1.arch-all.unusedbd.real == texinfo=5.2.0.dfsg.1-3 == cloog_0.18.2-1.arch-all.unusedbd.real == texinfo=5.2.0.dfsg.1-3 == cpio_2.11+dfsg-2.arch-all.unusedbd.real == texinfo=5.2.0.dfsg.1-3 == cunit_2.1-2.dfsg-1.arch-all.unusedbd.real == quilt=0.63-2 == cups-filters_1.0.53-1.arch-all.unusedbd.real == sharutils=1:4.14-2 == curl_7.37.0-1.arch-all.unusedbd.real == ca-certificates=20140325 openssh-server=1:6.6p1-5 python=2.7.6-2 == dbus-python_1.2.0-2.arch-all.unusedbd.real == xmlto=0.0.25-2 == dbus_1.8.2-1.arch-all.unusedbd.real == python=2.7.6-2 == devscripts_2.14.4.arch-all.unusedbd.real == libjson-perl=2.61-1 libterm-size-perl=0.207-1+b1 == doxygen_1.8.7-1.arch-all.unusedbd.real == texlive-extra-utils=2014.20140528-2 == e2fsprogs_1.42.10-1.arch-all.unusedbd.real == m4=1.4.17-4 == eglibc_2.18-7.arch-all.unusedbd.real == autoconf=2.69-6 quilt=0.63-2 == exiv2_0.23-1.arch-all.unusedbd.real == autotools-dev=20140510.1 dh-linktree=0.4 libjs-jquery=1.7.2+dfsg-3 == fftw3_3.3.4-1.arch-all.unusedbd.real == ghostscript=9.05~dfsg-8.1 transfig=1:3.2.5.e-3 == flite_1.4-release-8.arch-all.unusedbd.real == ghostscript=9.05~dfsg-8.1 texlive=2014.20140528-3 == fontconfig_2.11.0-5.arch-all.unusedbd.real == gperf=3.0.4-1 == freeglut_2.8.1-2.arch-all.unusedbd.real == libxt-dev:amd64=1:1.1.4-1 == gdb_7.6.2-1.1.arch-all.unusedbd.real == autoconf=2.69-6 flex=2.5.39-7 gcj-jdk=4:4.9.0-1 libtool=2.4.2-1.7 procps=1:3.3.9-5 python3-dev=3.4.1~rc1-1 == geoclue-2.0_2.1.8-1.arch-all.unusedbd.real == geoip-database=20140509-1 == geoip_1.6.0-1.arch-all.unusedbd.real == zlib1g-dev:amd64=1:1.2.8.dfsg-1 == ghostscript_9.05~dfsg-8.1.arch-all.unusedbd.real == freeglut3-dev:amd64=2.8.1-2 == gnome-keyring_3.12.0-2.arch-all.unusedbd.real == ca-certificates=20140325 docbook-xml=4.5-7.2 libglib2.0-doc=2.40.0-3 == gnome-vfs_2.24.4-4.arch-all.unusedbd.real == libattr1-dev:amd64=1:2.4.47-1 == gpm_1.20.4-6.1.arch-all.unusedbd.real == texi2html=1.82+dfsg1-3 texlive-base=2014.20140528-3 == graphviz_2.26.3-17.arch-all.unusedbd.real == libgd2-noxpm-dev=2.1.0-3 pdksh=49-2 == gsl_1.16+dfsg-1.arch-all.unusedbd.real == libtool=2.4.2-1.7 == gst-plugins-base0.10_0.10.36-1.1.arch-all.unusedbd.real == gir1.2-freedesktop=1.40.0-2 gir1.2-glib-2.0=1.40.0-2 == gst-plugins-base1.0_1.2.4-1.arch-all.unusedbd.real == gir1.2-freedesktop=1.40.0-2 == gstreamer1.0_1.2.4-1.arch-all.unusedbd.real == gir1.2-freedesktop=1.40.0-2 == gtest_1.7.0-3.arch-all.unusedbd.real == python=2.7.6-2 == gtk+2.0_2.24.23-1.arch-all.unusedbd.real == libatk1.0-doc=2.12.0-1 libcairo2-doc=1.12.16-2 libglib2.0-doc=2.40.0-3 libpango1.0-doc=1.36.3-1 == gtk+3.0_3.12.2-1.arch-all.unusedbd.real == gsettings-desktop-schemas=3.8.2-2 == gtkglext_1.2.0-3.1.arch-all.unusedbd.real == autotools-dev=20140510.1 == heimdal_1.6~rc2+dfsg-6.arch-all.unusedbd.real == libperl4-corelibs-perl=0.003-1 == hwloc_1.9-3.arch-all.unusedbd.real == autotools-dev=20140510.1 help2man=1.45.1 transfig=1:3.2.5.e-3 == ijs_0.35-10.arch-all.unusedbd.real == docbook-utils=0.6.14-3 docbook=4.5-5.1 ghostscript=9.05~dfsg-8.1 == klibc_2.0.3-1.arch-all.unusedbd.real == bison=2:3.0.2.dfsg-2 m4=1.4.17-4 ==
Re: possible MBF: automatically detecting unused build dependencies
On Monday, July 28, 2014 12:07:58 Johannes Schauer wrote: Hi, Quoting gregor herrmann (2014-07-28 11:45:14) == libxml-parser-perl_2.41-1.arch-all.unusedbd == sharutils=1:4.14-2 Already fixed in 2.41-2. thanks! I assume you're planning to do a new run before actually filing the bugs? I cannot do a new run before September because I'm moving places twice within the next month and thus do not have a stable always-on machine available during that time. The only thing that could change this would be if I found easily accessible compute time elsewhere (I asked at debian-qa@l.d.o: http://lists.debian.org/20140726090503.4150.56356@hoothoot ) I thought that hardly any build dependencies get removed over time so that it would still be appropriate to fill bugreports for the June results now. If that would not be appreciated then I'll re-run the whole thing once I settled in at our new place some time in September. It is quite common for people to fix things based on the initial discussion about an impending MBF, so I think it would be less than impolite to not acknowledge that by filing bugs on obsolete data. The two packages that I show up for are fixed as well. Scott K -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/1436982.fj5k9h6aVK@scott-latitude-e6320
Re: possible MBF: automatically detecting unused build dependencies
On Monday, July 28, 2014 08:54:29 Scott Kitterman wrote: On Monday, July 28, 2014 12:07:58 Johannes Schauer wrote: Hi, Quoting gregor herrmann (2014-07-28 11:45:14) == libxml-parser-perl_2.41-1.arch-all.unusedbd == sharutils=1:4.14-2 Already fixed in 2.41-2. thanks! I assume you're planning to do a new run before actually filing the bugs? I cannot do a new run before September because I'm moving places twice within the next month and thus do not have a stable always-on machine available during that time. The only thing that could change this would be if I found easily accessible compute time elsewhere (I asked at debian-qa@l.d.o: http://lists.debian.org/20140726090503.4150.56356@hoothoot ) I thought that hardly any build dependencies get removed over time so that it would still be appropriate to fill bugreports for the June results now. If that would not be appreciated then I'll re-run the whole thing once I settled in at our new place some time in September. It is quite common for people to fix things based on the initial discussion about an impending MBF, so I think it would be less than impolite to not acknowledge that by filing bugs on obsolete data. The two packages that I show up for are fixed as well. Scott K ... less than polite ... That'll teach me to write to -devel while still on the first cup of coffee. Scott K -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/6310597.iGnANyuyd6@scott-latitude-e6320
Re: possible MBF: automatically detecting unused build dependencies
On Mon, 28 Jul 2014 12:07:58 +0200, Johannes Schauer wrote: Quoting gregor herrmann (2014-07-28 11:45:14) == libxml-parser-perl_2.41-1.arch-all.unusedbd == sharutils=1:4.14-2 Already fixed in 2.41-2. thanks! Thanks to you for providing the lists :) I assume you're planning to do a new run before actually filing the bugs? I cannot do a new run before September because I'm moving places twice within the next month and thus do not have a stable always-on machine available during that time. The only thing that could change this would be if I found easily accessible compute time elsewhere (I asked at debian-qa@l.d.o: http://lists.debian.org/20140726090503.4150.56356@hoothoot ) It seems like there might be solution coming up in this thread - good! I thought that hardly any build dependencies get removed over time so that it would still be appropriate to fill bugreports for the June results now. My gut feeling (cf. also Scott's mail) is that some percentage of bugs already get fixed by sending a dd-list to -devel. If that would not be appreciated then I'll re-run the whole thing once I settled in at our new place some time in September. I could live with one bug that I have to close but re-running the tests would probably be preferrable, if possible. Cheers, gregor -- .''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06 : :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/ `. `' Member of VIBE!AT SPI, fellow of the Free Software Foundation Europe `- NP: Dhafer Youssef: Diaphanes signature.asc Description: Digital Signature
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Scott Kitterman (2014-07-28 14:54:29) It is quite common for people to fix things based on the initial discussion about an impending MBF, so I think it would be less than impolite to not acknowledge that by filing bugs on obsolete data. The two packages that I show up for are fixed as well. I know. That's why I wrote a few emails up: The results exclude python-defaults and python3-defaults (as requested by Scott Kitterman) and openldap (as requested by Ryan Tandy) and lirc (as requested by Stefan Lippers-Hollmann). Thank you Scott, Ryan and Stefan for already having fixed your packaging! Nevertheless, I'm currently talking with Holger Levsen and it seems that it should be possible to implement the machinery on jenkins.debian.net. If you think that I should not file bugs for the current results, then I'll wait until the first jenkins runs come in with fresh data. cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140728153328.4150.86639@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
* Johannes Schauer j.scha...@email.de, 2014-07-09, 16:50: == llvm-toolchain-3.4_3.4.1-4.arch-all.unusedbd.real == automake=1:1.14.1-3 autotools-dev=20140510.1 diffstat=1.58-1 doxygen=1.8.7-1 flex=2.5.39-7 lcov=1.10-1 libtool=2.4.2-1.7 patchutils=0.3.3-1 procps=1:3.3.9-5 sharutils=1:4.14-2 tcl=8.6.0+8 texinfo=5.2.0.dfsg.1-3 No, building with DEB_BUILD_OPTIONS=codecoverage enables at least some of them, should build dependencies which the source package only requires after setting some DEB_BUILD_OPTIONS go into Build-Depends? Probably not, unless it's one of the optioned blessed by Policy §4.9.1. :-) and applying patches might require the autotools to be re-run, so I think lots of the requirements are sane. My naive assumption was that the Build-Depends line contains a list of binary packages needed to build the package. Not binary packages that might be needed in some situations during the lifetime of a source package? Agreed. But here I'd recommend regenerating auto* stuff unconditionally, rather than dropping the build-dependencies. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140710104536.gb5...@jwilk.net
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Jakub Wilk (2014-07-10 12:45:36) * Johannes Schauer j.scha...@email.de, 2014-07-09, 16:50: should build dependencies which the source package only requires after setting some DEB_BUILD_OPTIONS go into Build-Depends? Probably not, unless it's one of the optioned blessed by Policy §4.9.1. :-) And even those are probably options which, if enabled, require less build dependencies than for a full build rather than more. Once build profiles arrive in stable, it will be possible to add !profile.nocheck to build dependencies which are not needed when specifying DEB_BUILD_OPTIONS=nocheck. My naive assumption was that the Build-Depends line contains a list of binary packages needed to build the package. Not binary packages that might be needed in some situations during the lifetime of a source package? Agreed. But here I'd recommend regenerating auto* stuff unconditionally, rather than dropping the build-dependencies. Yes, of course. cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140710110704.14505.27633@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Steve Langasek (2014-07-09 00:32:18) But it absolutely does have this effect if we add bootstrap-specific metadata unnecessarily, because: - it introduces a syntax incompatibility with older versions of the package tools we are currently trying to get a minimal change to dpkg, apt and python-apt into wheezy backports to solve this problem: http://lists.debian.org/20140706211617.14505.38487@hoothoot - it makes the packaging more complex to understand While this is true in one sense, one could also argue that annotating build dependencies with what they are used for (with !profile.nodoc or !profile.nocheck) does also add some clarity. Though I guess you were mainly talking about complexity in debian/rules. Sure, more conditionals add complexity. The latter point is as likely to cause problems for the bootstrappers themselves as it is for the maintainers, since the more maintainers who have to get this metadata right and keep it up to date, the more it's going to bitrot. If you are worried about bitrot, then we have to throw more continuous testing at it. With botch we can create a build order and then verify it once enough source packages have build profile information attached. Should Debian not have sufficient resources for that I am willing to do those rebuilds on my own hardware. The bitrot will happen even if bootstrap data is added out of necessity. Due to changing dependencies, some stage1 information might not be needed anymore at some point because it becomes better to break cycles at another point of the graph. So continuous testing is needed in any case. I'm happy that tools like botch exist and think botch has been a useful accelerator for bootstrappability of Debian. However, my own goal is to see that future port bootstraps can be done using the standard buildd infrastructure (for each of Debian and Ubuntu), which doesn't currently make use of such techniques - rather, they work by building everything which is buildable. If you plan to wire up botch to wanna-build for archive-friendly bootstrappability, that would be great to see! I would need to know far more about wanna-build until I can do that. I'm also not too sure whether wanna-build is the right machinery to do bootstraps from scratch? Maybe it is in the sense that botch could provide the info of which source package to best build with reduced build dependencies once nothing is buildable anymore. But it would probably be prudent to show that such an automated bootstrap is possible without wanna-build in the first place. And we are still quite far away from being able to do automated bootstraps, sadly :( But until that's happened, I stand by my claim that stage1 data not needed for breaking build-dep loops is counterproductive for bootstrapping ports. I agree with you that it is unwise to add such extra information to more packages than needed (currently about 60 source packages) before there is enough infrastructure to run and test everything. But again, except for self-cycles there are no hard requirements on a source package needing stage1 annotations. Botch can show which source packages, if modified accordingly could solve the problem with a (close to) minimal number of source packages changed. But if one source package cannot be changed because of technical reasons or because the maintainer refuses to implement these changes, then there are ways to work around that by modifying other source packages. But I guess that's what you meant by need. I guess either once jessie is released or once the wheezy backport happens, build profiles will slowly be introduced with the most obvious packages first. Then will come the other, harder packages until we can for example do a native amd64 bootstrap, starting from a minimal build system. Once we are that far (and that will probably take another release at least) we can talk again about adding bootstrap-specific metadata unnecessarily. :) So let me retract my claim from my earlier email and instead say that I agree that adding !profile.stage1 annotation should be done very selectively and only if necessary. Nevertheless the generated information should be stored somewhere so that a bootstrapper can use it as a source of information which build dependencies can be dropped without much effort from a source package if so necessary. cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140710120943.14505.4470@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
op 07-07-14 14:20, Johannes Schauer schreef: Hi, sorry I attached two wrong files containing the many false positives already noticed by some of the replies. Here the actual results. Sorry. cheers, josch == llvm-toolchain-3.4_3.4.1-4.arch-all.unusedbd.real == automake=1:1.14.1-3 autotools-dev=20140510.1 diffstat=1.58-1 doxygen=1.8.7-1 flex=2.5.39-7 lcov=1.10-1 libtool=2.4.2-1.7 patchutils=0.3.3-1 procps=1:3.3.9-5 sharutils=1:4.14-2 tcl=8.6.0+8 texinfo=5.2.0.dfsg.1-3 No, building with DEB_BUILD_OPTIONS=codecoverage enables at least some of them, and applying patches might require the autotools to be re-run, so I think lots of the requirements are sane. doxygen is probably used for llvm-3.4-doc, so I think it might not be unused either. texinfo probably the same. diffutils, tcl, flex, patchutils, procps could possibly be unused, although not 100% sure. :-) ~Maarten -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/53bd4831.7030...@canonical.com
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Maarten Lankhorst (2014-07-09 15:48:33) == llvm-toolchain-3.4_3.4.1-4.arch-all.unusedbd.real == automake=1:1.14.1-3 autotools-dev=20140510.1 diffstat=1.58-1 doxygen=1.8.7-1 flex=2.5.39-7 lcov=1.10-1 libtool=2.4.2-1.7 patchutils=0.3.3-1 procps=1:3.3.9-5 sharutils=1:4.14-2 tcl=8.6.0+8 texinfo=5.2.0.dfsg.1-3 No, building with DEB_BUILD_OPTIONS=codecoverage enables at least some of them, should build dependencies which the source package only requires after setting some DEB_BUILD_OPTIONS go into Build-Depends? and applying patches might require the autotools to be re-run, so I think lots of the requirements are sane. My naive assumption was that the Build-Depends line contains a list of binary packages needed to build the package. Not binary packages that might be needed in some situations during the lifetime of a source package? doxygen is probably used for llvm-3.4-doc, so I think it might not be unused either. texinfo probably the same. The traces show that no file of the doxygen or texinfo package are run during a normal build (including building architecture:all packages). You say they are used. So maybe this indicates a bug where you think they should be used but are in fact not? Could you make sure? diffutils, tcl, flex, patchutils, procps could possibly be unused, although not 100% sure. :-) the only kind of false positive that was pointed out this method has is that it discovers otherwise empty packages (meta packages) which depend on other packages without which the build will nevertheless succeed because it will just silently fail if the tools provided by them are found to not be present. The package tcl is such a case and might thus be a false positive. The other packages are containing actual usable content and should be legitimate. It would be great if you could point out if there was yet another undiscovered flaw in my method. Thank you! cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140709145041.14505.85252@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Steve Langasek (2014-07-07 20:22:42) Ah. No, it only means that the package build does not *fail* if the build-dependency is removed. That is not the same thing as saying that the build-dependency is not used. It would of course be better if packages were resilient against breakage in their build-dependencies, instead of misbuilding with features disabled or with wrong fallback code. But this isn't easy to do in all build systems, and anyway this isn't something we routinely audit about our packages; we shouldn't regard this as a bug to be reported without a lot more discussion of how we're going to systematically stay on top of those issues in the future. For your purposes, a better method would be: - identify those build-dependencies which are empty except for /usr/share/doc - for each such package, look at the contents of its direct dependencies - check the build against those contents it turns out that 13% of my results are packages with no other content than in /usr/share/doc. Namely: libreadline-dev, default-jdk-doc, doxygen-latex, gcj-native-helper, g++-multilib, gobjc, libboost-dev, libboost-system-dev, libgmp3-dev, libgphoto2-2-dev, libopenais-dev, libtasn1-3-dev, libxmlrpc-c3-dev, lynx, mysql-server, python3-all-dbg, python3-all-dev, libdb-dev, python-all, python-gobject, python-all-dbg, python-all-dev I'll re-run the whole caboodle later this year but consider the file content of empty packages to be the sum of the content of packages it depends on. This should reduce the number of this kind of false positives. Do you think I should fill bugs for all non-empty packages that were already found? Or do you think there is another high chance of false positives for that kind of packages too? cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140708074302.14505.28772@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
On Tue, Jul 08, 2014 at 09:43:02AM +0200, Johannes Schauer wrote: Do you think I should fill bugs for all non-empty packages that were already found? Or do you think there is another high chance of false positives for that kind of packages too? The only other likely sources of false positives I can think of would also be bugs, just bugs of a different sort - i.e., if you're build-depending on a -dev package which you don't use and which is not a metapackage, but you are using its dependencies. So I think it's fine to file bugs for the others, unless someone else raises objections here. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 07, 2014 at 01:51:00PM +0200, Johannes Schauer wrote: Kurt Roeckx k...@roeckx.be libtool == libtool_2.4.2-1.7.arch-all.unusedbd == gfortran=4:4.8.2-4 gfortran Depends on gfortran-4.8, and that is being used. openssl (U) == openssl_1.0.1g-4.arch-all.unusedbd == m4=1.4.17-4 From the changelog: * Add Build-Dependency on m4, since sparc needs it to generate it's assembler files. (Closes: #334542) Kurt -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140708223637.ga14...@roeckx.be
Re: possible MBF: automatically detecting unused build dependencies
On Tue, Jul 08, 2014 at 06:22:49AM +0200, Johannes Schauer wrote: Quoting Steve Langasek (2014-07-08 00:07:29) On Mon, Jul 07, 2014 at 09:14:44PM +0200, Johannes Schauer wrote: Nevertheless, those false positives that were generated this way are still useful to be later marked with !profile.stage1 once build profiles are allowed in the archive because they are obviously droppable. No, marking build-dependencies with build profiles should be driven by what is needed to break build-dep loops, not by what build-deps are droppable without further modification of the packages. Excessive stage1 profile tagging will result in unnecessary extra builds during a bootstrap. Bootstrapping should not get into the way of the normal packaging. But it absolutely does have this effect if we add bootstrap-specific metadata unnecessarily, because: - it introduces a syntax incompatibility with older versions of the package tools - it makes the packaging more complex to understand The latter point is as likely to cause problems for the bootstrappers themselves as it is for the maintainers, since the more maintainers who have to get this metadata right and keep it up to date, the more it's going to bitrot. The downside you list unnecessary extra builds are easy to avoid in practice. Botch contains algorithms to find a close to minimum set of source packages to modify to make a dependency graph acyclic (a feedback vertex set). Even if all source packages in the dependency graph had removable build dependencies it would only choose a small set of them (currently 60-70 are needed) to actually build with a stage1 profile active. I'm happy that tools like botch exist and think botch has been a useful accelerator for bootstrappability of Debian. However, my own goal is to see that future port bootstraps can be done using the standard buildd infrastructure (for each of Debian and Ubuntu), which doesn't currently make use of such techniques - rather, they work by building everything which is buildable. If you plan to wire up botch to wanna-build for archive-friendly bootstrappability, that would be great to see! But until that's happened, I stand by my claim that stage1 data not needed for breaking build-dep loops is counterproductive for bootstrapping ports. I did not plan to run this script more often yet. I'll probably do another run after having added a detection for meta packages as suggested by you below. Running it more regularly might not require any big discussion because the problem size might be small enough that maintaining a white list would be enough. If you really want to systematically detect misbuilds on missing build-dependencies, it's a much bigger problem than just detecting this for the build-dependencies which are metapackages. Why? The build dependencies which are not metapackages are not listed as false positives because they get weeded out in the first phase. Because package misbuilds when a metapackage it build-depends on is broken is an arbitrary subset of package misbuilds when build-depends are broken, and it does not benefit Debian to have maintainers focusing on such an arbitrary subset. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Kurt Roeckx (2014-07-09 00:36:37) On Mon, Jul 07, 2014 at 01:51:00PM +0200, Johannes Schauer wrote: Kurt Roeckx k...@roeckx.be libtool == libtool_2.4.2-1.7.arch-all.unusedbd == gfortran=4:4.8.2-4 gfortran Depends on gfortran-4.8, and that is being used. indeed this is then a false positive of a nearly-meta package. Unfortunately the gfortran package does ship some files besides /usr/share/doc (both symlinks) and would thus not be classified as a meta package. Maybe I should also check debtags. gfortran is tagged role::dummy. I also found role::metapackage. Is there another debtag or method in general that would allow to consider the dependencies of a package instead and avoid this kind of false positive? Maybe a package shipping only symlinks besides /usr/share/doc is another way of finding meta packages. On the other hand it's harder to check the type of file a package ships as it needs downloading and unpacking first. I'm not aware whether tools like apt-file display information about the file type. openssl (U) == openssl_1.0.1g-4.arch-all.unusedbd == m4=1.4.17-4 From the changelog: * Add Build-Dependency on m4, since sparc needs it to generate it's assembler files. (Closes: #334542) Then it makes sense that my amd64 build caught that. Could you make m4 an architecture specific dependency if it's only used on sparc? cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140709043719.14505.38892@hoothoot
possible MBF: automatically detecting unused build dependencies
Hi, we would like to propose an MBF with severity wishlist to drop unused build dependencies in a number of source packages indicated by the attached two text files and the dd-list output. TLDR; We searched a selection of source packages relevant to bootstrapping for build dependencies which are never used during the package build. Fewer build dependencies help making bootstrapping easier so we want to submit bugs for the results we generated. The results were generated automatically by a script using fatrace(1) and fake packages. It follows the MBF template email and a detailed explanation of our procedures. Attached is the list of the unused build dependencies we found and the output of dd-list. We would like you to review the list, drop unused build dependencies or report errors. MBF template email: --%--- Subject: Please consider removing the build dependencies on $foo, $bar and $baz Severity: wishlist Usertag: unusedbd User: bootst...@lists.debian.org Dear Maintainer, the build dependencies $foo, $bar and $baz of this source package do not seem to be needed. Neither are any of their files accessed during the build nor are their dependencies on other binary packages required. Please consider dropping those build dependencies to make bootstrapping Debian easier. You can find more detail about the procedures that were used to find this problem in the MBF announcement on debian-devel: $email --%--- Long version: Our bootstrappable Debian GSoC student Peter Pentchev found a number of source packages that did not make use of some of their build dependencies [1]. We thought we could automate this process so we set up a simple sbuild based package builder. The machinery works in two passes. In the first pass, all source packages are built normally but with fatrace(1) active. From the fatrace output we determine all build dependencies of which no files were used during the build. In the second pass we go through all source packages for which candidate droppable build dependencies where found with fatrace and test whether those build dependencies are really droppable by replacing them with an empty equivs package without dependencies one by one. The first pass makes sure that the files of a specific build dependency are not needed. This is to make sure not to produce false positives of build dependencies which, if not installed on the build system, do not result in a build failure but instead just in a reduction of functionality of the output binary. The second pass makes sure that also the dependencies of a specific build dependency are not needed. This makes sure to not produce false positives of meta or virtual packages. We executed the build on the amd64 snapshot 20140601T00Z of Debian Sid. A selection of 549 source was made from the source packages that are involved in the central strongly connected component of the bootstrapping dependency graph. We needed to patch fatrace to be able to handle files larger than 2 GB (bug #722901). We ran the whole process with `sbuild --arch-all` and then again with `sbuild --no-arch-all`. This way we can find build dependencies which can be dropped from full package builds as well as those which can be moved to Build-Depends-Indep. We did not use the results to also check for build dependencies that can be removed from Build-Depends-Indep. The source code to run the whole machinery can be found at [2]. Executing everything took two weeks and over 2000 package builds. We found 277 build dependencies could be dropped from the 549 tested source packages, affecting 158 source packages. We checked the impact on the bootstrap dependency graph. If all the found build dependencies were really droppable, then the minimum (strong) dependency graph would shrink from containing 483 source packages to 447 source packages. More importantly, the solution to the feedback arc set problem for making it acyclic would shrink from 74 to 56 source packages. Thus, the generated results are very relevant for making solving the bootstrap problem easier. You can find the results per source package in the attachment together with the dd-list output. The file drop-from-bd.txt lists the build dependencies that can be dropped from Build-Depends while move-to-bdi lists the build dependencies that can be moved from Build-Depends to Build-Depends-Indep. Can you spot obvious mistakes in the results or in the procedure used to generate them? cheers, josch [1] #749616 #749972 #751702 #751897 #752938 [2] https://github.com/josch/findunusedbd Adam C. Powell, IV hazel...@debian.org mpi-defaults (U) Adam Conrad adcon...@0c3.net cyrus-sasl2 (U) eglibc (U) freetds (U) Agustin Martin Domingo agmar...@debian.org linuxdoc-tools (U) Alan Woodland awoodl...@debian.org blcr Alessandro Ghedini gh...@debian.org curl valgrind
Re: possible MBF: automatically detecting unused build dependencies
On Mon, 07 Jul 2014, Johannes Schauer wrote: MBF template email: --%--- Subject: Please consider removing the build dependencies on $foo, $bar and $baz Severity: wishlist Usertag: unusedbd User: bootst...@lists.debian.org Dear Maintainer, the build dependencies $foo, $bar and $baz of this source package do not seem to be needed. Neither are any of their files accessed during the build nor are their dependencies on other binary packages required. Please consider dropping those build dependencies to make bootstrapping Debian easier. You can find more detail about the procedures that were used to find this problem in the MBF announcement on debian-devel: $email --%--- Please don't assume that the unused build dependency is always where the defect is. Rather, the MBF text should account for the possibility that the unused build-dependency should have been used in the first place, but something is broken in the build and it is being left unused. For example: something that is uselessly build-depending on autotools-dev might either: 1. Have an useless build-dependency or 2. Have a bug that is causing autoreconf to either not be called in the first place, or to fail to refresh the auto-tooling. While (1) is more likely, ignoring the possibility of (2) is not wise. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707120726.ga7...@khazad-dum.debian.net
Re: possible MBF: automatically detecting unused build dependencies
Le lundi 07 juillet 2014 à 09:07 -0300, Henrique de Moraes Holschuh a écrit : On Mon, 07 Jul 2014, Johannes Schauer wrote: MBF template email: --%--- Subject: Please consider removing the build dependencies on $foo, $bar and $baz Severity: wishlist Usertag: unusedbd User: bootst...@lists.debian.org Dear Maintainer, the build dependencies $foo, $bar and $baz of this source package do not seem to be needed. Neither are any of their files accessed during the build nor are their dependencies on other binary packages required. Please consider dropping those build dependencies to make bootstrapping Debian easier. You can find more detail about the procedures that were used to find this problem in the MBF announcement on debian-devel: $email --%--- Please don't assume that the unused build dependency is always where the defect is. Rather, the MBF text should account for the possibility that the unused build-dependency should have been used in the first place, but something is broken in the build and it is being left unused. Consider also the case when arch:all package require a build dependency on a package that only builds on some archs, to prevent the arch:all package being available on archs where its dependencies are not. nodejs and node-* packages are such an example. Jérémy. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/1404735139.4416.2.camel@imac.chaumes
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 7, 2014 at 1:51 PM, Johannes Schauer j.scha...@email.de wrote: Can you spot obvious mistakes in the results or in the procedure used to generate them? There seem to be a bunch of false positives for virtual/metapackages: == python-numpy_1.8.1-1.arch-all.unusedbd == gfortran=4:4.8.2-4 python-all-dbg=2.7.6-2 python-all-dev=2.7.6-2 python3-all-dbg=3.4.1~rc1-1 python3-all-dev=3.4.1~rc1-1 the -all packages are basically empty packages pulling in all python versions supported, those packages are definitely used during the build. Removing them during an arch-any build should fail the build. does sbuild --arch-all only build the indep part? If so that might explain why your pass2 did not remove these, but so far I know we have no way to declare this state in our control, we only have Build-Depends and Build-Depends-Indep, no Build-Depends-Arch. Similar fftw3: == fftw3_3.3.4-1.arch-all.unusedbd == gfortran=4:4.8.2-4 ghostscript=9.05~dfsg-8.1 mpi-default-dev=1.0.2+nmu1 transfig=1:3.2.5.e-3 mpi-default-dev is not used directly but its dependencies are required for the fully featured arch any build. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/CAK5FAtGP294ECYGOfh4dSEHcdYVi82jJDs2q67E=_ph-nla...@mail.gmail.com
Re: possible MBF: automatically detecting unused build dependencies
Hi josch, thank you very much for this effort, just two remarks: 1. +1 to what hmh said 2. You have missed at least one virtual package: libdb-dev is used to depend on latest libdbX.Y-dev, so if you can remove that before MBF... Ondrej On Mon, Jul 7, 2014, at 13:51, Johannes Schauer wrote: Hi, we would like to propose an MBF with severity wishlist to drop unused build dependencies in a number of source packages indicated by the attached two text files and the dd-list output. TLDR; We searched a selection of source packages relevant to bootstrapping for build dependencies which are never used during the package build. Fewer build dependencies help making bootstrapping easier so we want to submit bugs for the results we generated. The results were generated automatically by a script using fatrace(1) and fake packages. It follows the MBF template email and a detailed explanation of our procedures. Attached is the list of the unused build dependencies we found and the output of dd-list. We would like you to review the list, drop unused build dependencies or report errors. MBF template email: --%--- Subject: Please consider removing the build dependencies on $foo, $bar and $baz Severity: wishlist Usertag: unusedbd User: bootst...@lists.debian.org Dear Maintainer, the build dependencies $foo, $bar and $baz of this source package do not seem to be needed. Neither are any of their files accessed during the build nor are their dependencies on other binary packages required. Please consider dropping those build dependencies to make bootstrapping Debian easier. You can find more detail about the procedures that were used to find this problem in the MBF announcement on debian-devel: $email --%--- Long version: Our bootstrappable Debian GSoC student Peter Pentchev found a number of source packages that did not make use of some of their build dependencies [1]. We thought we could automate this process so we set up a simple sbuild based package builder. The machinery works in two passes. In the first pass, all source packages are built normally but with fatrace(1) active. From the fatrace output we determine all build dependencies of which no files were used during the build. In the second pass we go through all source packages for which candidate droppable build dependencies where found with fatrace and test whether those build dependencies are really droppable by replacing them with an empty equivs package without dependencies one by one. The first pass makes sure that the files of a specific build dependency are not needed. This is to make sure not to produce false positives of build dependencies which, if not installed on the build system, do not result in a build failure but instead just in a reduction of functionality of the output binary. The second pass makes sure that also the dependencies of a specific build dependency are not needed. This makes sure to not produce false positives of meta or virtual packages. We executed the build on the amd64 snapshot 20140601T00Z of Debian Sid. A selection of 549 source was made from the source packages that are involved in the central strongly connected component of the bootstrapping dependency graph. We needed to patch fatrace to be able to handle files larger than 2 GB (bug #722901). We ran the whole process with `sbuild --arch-all` and then again with `sbuild --no-arch-all`. This way we can find build dependencies which can be dropped from full package builds as well as those which can be moved to Build-Depends-Indep. We did not use the results to also check for build dependencies that can be removed from Build-Depends-Indep. The source code to run the whole machinery can be found at [2]. Executing everything took two weeks and over 2000 package builds. We found 277 build dependencies could be dropped from the 549 tested source packages, affecting 158 source packages. We checked the impact on the bootstrap dependency graph. If all the found build dependencies were really droppable, then the minimum (strong) dependency graph would shrink from containing 483 source packages to 447 source packages. More importantly, the solution to the feedback arc set problem for making it acyclic would shrink from 74 to 56 source packages. Thus, the generated results are very relevant for making solving the bootstrap problem easier. You can find the results per source package in the attachment together with the dd-list output. The file drop-from-bd.txt lists the build dependencies that can be dropped from Build-Depends while move-to-bdi lists the build dependencies that can be moved from Build-Depends to Build-Depends-Indep. Can you spot obvious mistakes in the results or in the procedure used to
Re: possible MBF: automatically detecting unused build dependencies
Hi, sorry I attached two wrong files containing the many false positives already noticed by some of the replies. Here the actual results. Sorry. cheers, josch == apache2_2.4.9-1.arch-all.unusedbd.real == libcap-dev:amd64=1:2.22-1.2 == apparmor_2.8.0-5.arch-all.unusedbd.real == dejagnu=1.5-3 texlive-latex-recommended=2014.20140528-3 == apr-util_1.5.3-2.arch-all.unusedbd.real == libpcre3-dev:amd64=1:8.31-5 == apt_1.0.3.arch-all.unusedbd.real == automake=1:1.14.1-3 == at-spi2-atk_2.10.2-3.arch-all.unusedbd.real == libglib2.0-bin=2.40.0-3 == avahi_0.6.31-4.arch-all.unusedbd.real == python-all-dev=2.7.6-2 == bc_1.06.95-8.arch-all.unusedbd.real == bison=2:3.0.2.dfsg-2 libreadline-dev:amd64=6.3-6 == bison_3.0.2.dfsg-2.arch-all.unusedbd.real == autotools-dev=20140510.1 == blcr_0.8.5-2.1.arch-all.unusedbd.real == autoconf=2.69-6 automake=1:1.14.1-3 autotools-dev=20140510.1 libtool=2.4.2-1.7 == bluez_4.101-4.1.arch-all.unusedbd.real == bison=2:3.0.2.dfsg-2 gstreamer-tools=0.10.36-1.2 == boost-defaults_1.55.0.2.arch-all.unusedbd.real == libboost1.55-dev:amd64=1.55.0+dfsg-1 == ceph_0.80.1-1.arch-all.unusedbd.real == libboost-dev:amd64=1.55.0.2 libboost-system-dev:amd64=1.55.0.2 python-all=2.7.6-2 uuid-runtime=2.20.1-5.7 == chicken_4.8.0.5-1.arch-all.unusedbd.real == texinfo=5.2.0.dfsg.1-3 == cloog_0.18.2-1.arch-all.unusedbd.real == texinfo=5.2.0.dfsg.1-3 == cpio_2.11+dfsg-2.arch-all.unusedbd.real == texinfo=5.2.0.dfsg.1-3 == cunit_2.1-2.dfsg-1.arch-all.unusedbd.real == quilt=0.63-2 == cups-filters_1.0.53-1.arch-all.unusedbd.real == sharutils=1:4.14-2 == curl_7.37.0-1.arch-all.unusedbd.real == ca-certificates=20140325 openssh-server=1:6.6p1-5 python=2.7.6-2 == db5.3_5.3.28-3.arch-all.unusedbd.real == gcj-native-helper=2:1.7-52 == dbus-python_1.2.0-2.arch-all.unusedbd.real == xmlto=0.0.25-2 == dbus_1.8.2-1.arch-all.unusedbd.real == python=2.7.6-2 == devscripts_2.14.4.arch-all.unusedbd.real == libjson-perl=2.61-1 libterm-size-perl=0.207-1+b1 == doxygen_1.8.7-1.arch-all.unusedbd.real == texlive-extra-utils=2014.20140528-2 == e2fsprogs_1.42.10-1.arch-all.unusedbd.real == m4=1.4.17-4 == ecj_3.9.0-2.arch-all.unusedbd.real == python=2.7.6-2 == eglibc_2.18-7.arch-all.unusedbd.real == autoconf=2.69-6 quilt=0.63-2 == exiv2_0.23-1.arch-all.unusedbd.real == autotools-dev=20140510.1 dh-linktree=0.4 libjs-jquery=1.7.2+dfsg-3 == fastjar_0.98-5.arch-all.unusedbd.real == texinfo=5.2.0.dfsg.1-3 == fftw3_3.3.4-1.arch-all.unusedbd.real == ghostscript=9.05~dfsg-8.1 transfig=1:3.2.5.e-3 == findlib_1.4.1-1.arch-all.unusedbd.real == gawk=1:4.1.1+dfsg-1 == flite_1.4-release-8.arch-all.unusedbd.real == ghostscript=9.05~dfsg-8.1 == fontconfig_2.11.0-5.arch-all.unusedbd.real == gperf=3.0.4-1 == freeglut_2.8.1-2.arch-all.unusedbd.real == libxt-dev:amd64=1:1.1.4-1 == freetds_0.91-6.arch-all.unusedbd.real == libreadline-dev:amd64=6.3-6 == gdb_7.6.2-1.1.arch-all.unusedbd.real == autoconf=2.69-6 flex=2.5.39-7 gcj-jdk=4:4.9.0-1 gobjc=4:4.8.2-4 libtool=2.4.2-1.7 procps=1:3.3.9-5 == geoclue-2.0_2.1.8-1.arch-all.unusedbd.real == geoip-database=20140509-1 == geoip_1.6.0-1.arch-all.unusedbd.real == zlib1g-dev:amd64=1:1.2.8.dfsg-1 == ghostscript_9.05~dfsg-8.1.arch-all.unusedbd.real == freeglut3-dev:amd64=2.8.1-2 == gnome-keyring_3.12.0-2.arch-all.unusedbd.real == ca-certificates=20140325 docbook-xml=4.5-7.2 libglib2.0-doc=2.40.0-3 libtasn1-3-dev=3.6-1 == gnome-vfs_2.24.4-4.arch-all.unusedbd.real == libattr1-dev:amd64=1:2.4.47-1 == gpac_0.5.0+svn5194~dfsg1-3.arch-all.unusedbd.real == libxmlrpc-c3-dev=1.16.33-3.2 == gpm_1.20.4-6.1.arch-all.unusedbd.real == texi2html=1.82+dfsg1-3 texlive-base=2014.20140528-3 == graphviz_2.26.3-17.arch-all.unusedbd.real == pdksh=49-2 == gsl_1.16+dfsg-1.arch-all.unusedbd.real == libtool=2.4.2-1.7 == gst-plugins-base0.10_0.10.36-1.1.arch-all.unusedbd.real == gir1.2-freedesktop=1.40.0-2 gir1.2-glib-2.0=1.40.0-2 == gst-plugins-base1.0_1.2.4-1.arch-all.unusedbd.real == gir1.2-freedesktop=1.40.0-2 == gstreamer1.0_1.2.4-1.arch-all.unusedbd.real == gir1.2-freedesktop=1.40.0-2 libgmp3-dev=2:6.0.0+dfsg-4 == gtest_1.7.0-3.arch-all.unusedbd.real == python=2.7.6-2 == gtk+2.0_2.24.23-1.arch-all.unusedbd.real == libatk1.0-doc=2.12.0-1 libcairo2-doc=1.12.16-2 libglib2.0-doc=2.40.0-3 libpango1.0-doc=1.36.3-1 == gtk+3.0_3.12.2-1.arch-all.unusedbd.real == gsettings-desktop-schemas=3.8.2-2 == gtkglext_1.2.0-3.1.arch-all.unusedbd.real == autotools-dev=20140510.1 == heimdal_1.6~rc2+dfsg-6.arch-all.unusedbd.real == libperl4-corelibs-perl=0.003-1 == hunspell_1.3.2-7.arch-all.unusedbd.real == libreadline-dev:amd64=6.3-6 == hwloc_1.9-3.arch-all.unusedbd.real == autotools-dev=20140510.1 doxygen-latex=1.8.7-1 help2man=1.45.1 transfig=1:3.2.5.e-3 == ijs_0.35-10.arch-all.unusedbd.real == docbook-utils=0.6.14-3 docbook=4.5-5.1 ghostscript=9.05~dfsg-8.1 == klibc_2.0.3-1.arch-all.unusedbd.real == bison=2:3.0.2.dfsg-2 m4=1.4.17-4 == lcms_1.19.dfsg1-1.3.arch-all.unusedbd.real ==
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Julian Taylor (2014-07-07 14:14:20) There seem to be a bunch of false positives for virtual/metapackages: == python-numpy_1.8.1-1.arch-all.unusedbd == gfortran=4:4.8.2-4 python-all-dbg=2.7.6-2 python-all-dev=2.7.6-2 python3-all-dbg=3.4.1~rc1-1 python3-all-dev=3.4.1~rc1-1 the -all packages are basically empty packages pulling in all python versions supported, those packages are definitely used during the build. Removing them during an arch-any build should fail the build. does sbuild --arch-all only build the indep part? Sorry, I attached the wrong files in my first email. I posted a follow up with the correct results which correctly do not contain the empty packages anymore. If so that might explain why your pass2 did not remove these, but so far I know we have no way to declare this state in our control, we only have Build-Depends and Build-Depends-Indep, no Build-Depends-Arch. Was Build-Depends-Arch not added with #629480? Similar fftw3: == fftw3_3.3.4-1.arch-all.unusedbd == gfortran=4:4.8.2-4 ghostscript=9.05~dfsg-8.1 mpi-default-dev=1.0.2+nmu1 transfig=1:3.2.5.e-3 mpi-default-dev is not used directly but its dependencies are required for the fully featured arch any build. Same story as above. Sorry for the confusion :( cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707122328.14505.92338@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Henrique de Moraes Holschuh (2014-07-07 14:07:26) Please don't assume that the unused build dependency is always where the defect is. Rather, the MBF text should account for the possibility that the unused build-dependency should have been used in the first place, but something is broken in the build and it is being left unused. you are correct, this should be mentioned. Here the updated text: --%--- Dear Maintainer, the build dependencies $foo, $bar and $baz of this source package do not seem to be needed. Neither are any of their files accessed during the build nor are their dependencies on other binary packages required. This can either mean that the build dependency is superfluous and should be removed to make bootstrapping easier or it indicates a bug where a package should be used but is in fact not. Please remove the build dependency from the Build-Depends in the former case or fix the build procedure in the latter. You can find more detail about the procedures that were used to find this problem in the MBF announcement on debian-devel: $email --%--- thank you! cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707123210.14505.41481@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
and the updated dd-list Sorry for not having attached the right files in my initial email :( cheres, josch Adam C. Powell, IV hazel...@debian.org mpi-defaults (U) Adam Conrad adcon...@0c3.net eglibc (U) freetds (U) Alan Woodland awoodl...@debian.org blcr Alessandro Ghedini gh...@debian.org curl valgrind Alessio Treglia ales...@debian.org gpac (U) Alexander Wirt formo...@debian.org rrdtool (U) Andreas Barth a...@not.so.argh.org netpbm-free Andreas Henriksson andr...@fatal.se gnome-keyring (U) gtk+3.0 (U) libarchive (U) libgnome-keyring (U) libsecret (U) libsoup2.4 (U) Andreas Metzler ametz...@debian.org libtasn1-6 (U) Andreas Rottmann ro...@debian.org libglade2 Andres Mejia ame...@debian.org gpac (U) libarchive (U) Andrew Moise ch...@demiurgestudios.com open-iscsi (U) Anibal Monsalve Salazar ani...@debian.org cpio libidn (U) libmnl xfsprogs (U) Anton Gladky gl...@debian.org freeglut APT Development Team de...@lists.debian.org apt python-apt Arnaud Fontaine ar...@debian.org xcb-util (U) xcb-util-image (U) xcb-util-keysyms (U) xcb-util-renderutil (U) xcb-util-wm (U) Arno Töll a...@debian.org apache2 (U) Aron Xu a...@debian.org libxml2 (U) netcat-openbsd Atsuhito KOHDA ko...@debian.org lynx-cur Aurelien Jarno aure...@debian.org eglibc (U) libusb libusb-1.0 qemu (U) Bart Martens ba...@debian.org gtkglext Bastian Blank wa...@debian.org parted (U) redhat-cluster (U) xen (U) Bastien ROUCARIÈS roucaries.bastien+deb...@gmail.com ghostscript (U) Benjamin Drung bdr...@debian.org devscripts (U) Benjamin Kaduk ka...@mit.edu krb5 (U) Bernd Zeimetz b...@debian.org rrdtool (U) Brian May b...@debian.org heimdal Ceph Maintainers ceph-maintain...@lists.ceph.com ceph Chris Halls ha...@debian.org hunspell (U) Christian Perrier bubu...@debian.org apt (U) Christoph Martin christoph.mar...@uni-mainz.de openssl (U) Chuan-kai Lin ck...@debian.org bison Clint Adams cl...@debian.org eglibc (U) Colin Watson cjwat...@debian.org parted (U) Craig Andrews candr...@integralblue.com geoclue-2.0 (U) Cyril Brulebois k...@debian.org libxkbcommon (U) xfonts-utils (U) Damien Raude-Morvan draz...@debian.org openjdk-7 (U) Daniel Leidert (dale) daniel.leid...@wgdd.de libxml-parser-perl (U) xmlto (U) Dave Beckett daj...@debian.org redland Davide Puricelli (evo) e...@debian.org chicken Debian Accessibility Team debian-accessibil...@lists.debian.org at-spi2-atk flite Debian Apache Maintainers debian-apa...@lists.debian.org apache2 apr-util Debian Berkeley DB Group pkg-db-de...@lists.alioth.debian.org db5.3 Debian Bluetooth Maintainers pkg-bluetooth-maintain...@lists.alioth.debian.org bluez Debian Boost Team pkg-boost-de...@lists.alioth.debian.org boost-defaults Debian GCC Maintainers debian-...@lists.debian.org cloog libffi Debian GNOME Maintainers pkg-gnome-maintain...@lists.alioth.debian.org gnome-keyring (U) gnome-vfs (U) gtk+2.0 gtk+3.0 libglade2 (U) libgnome-keyring libsecret libsoup2.4 pango1.0 pygobject-2 (U) pygtk (U) Debian GnuTLS Maintainers pkg-gnutls-ma...@lists.alioth.debian.org libtasn1-6 Debian HA Maintainers debian-ha-maintain...@lists.alioth.debian.org redhat-cluster Debian iSCSI Maintainers pkg-iscsi-maintain...@lists.alioth.debian.org open-iscsi Debian Java Maintainers pkg-java-maintain...@lists.alioth.debian.org ecj zookeeper Debian KDE Extras Team pkg-kde-ext...@lists.alioth.debian.org exiv2 Debian Libarchive Maintainers ah-libarch...@debian.org libarchive Debian Libidn Team help-lib...@gnu.org libidn Debian LibreOffice Maintainers debian-openoff...@lists.debian.org hunspell Debian Multimedia Maintainers pkg-multimedia-maintain...@lists.alioth.debian.org gpac x264 Debian OCaml Maintainers debian-ocaml-ma...@lists.debian.org findlib ocaml Debian OpenLDAP Maintainers pkg-openldap-de...@lists.alioth.debian.org openldap Debian OpenSSL Team pkg-openssl-de...@lists.alioth.debian.org openssl Debian Perl Group pkg-perl-maintain...@lists.alioth.debian.org libxml-parser-perl Debian PHP Maintainers pkg-php-ma...@lists.alioth.debian.org php5 Debian Printing Team debian-print...@lists.debian.org cups-filters ghostscript ijs Debian Python Modules Team python-modules-t...@lists.alioth.debian.org markupsafe (U) python-numpy Debian QA Group packa...@qa.debian.org graphviz readline5 Debian QEMU Team pkg-qemu-de...@lists.alioth.debian.org qemu Debian Qt/KDE Maintainers debian-qt-...@lists.debian.org qca2 qt4-x11 qtbase-opensource-src qtwebkit Debian RRDtool Team rrdt...@ml.snow-crash.org rrdtool Debian Samba Maintainers pkg-samba-ma...@lists.alioth.debian.org tdb Debian Science Team debian-science-maintain...@lists.alioth.debian.org fftw3
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 7, 2014 at 2:23 PM, Johannes Schauer j.scha...@email.de wrote: Hi, Quoting Julian Taylor (2014-07-07 14:14:20) If so that might explain why your pass2 did not remove these, but so far I know we have no way to declare this state in our control, we only have Build-Depends and Build-Depends-Indep, no Build-Depends-Arch. Was Build-Depends-Arch not added with #629480? neat, I have been looking in the policy for the tag where it is currently not mentioned (probably bug #727610) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/cak5fateoce4zpayfrq1upse9a6fn-+egdnueao9rootvnot...@mail.gmail.com
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Jérémy Lal (2014-07-07 14:12:19) Consider also the case when arch:all package require a build dependency on a package that only builds on some archs, to prevent the arch:all package being available on archs where its dependencies are not. nodejs and node-* packages are such an example. could you clarify this case? An arch:all package cannot require a build dependency because an arch:all package is a binary package. I can also not find any nodejs packages in the lists I posted. In fact, nodejs package have not yet been a problem with respect to bootstrapping - probably because they are mostly arch:all? cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707133254.14505.99230@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
Le lundi 07 juillet 2014 à 15:32 +0200, Johannes Schauer a écrit : Hi, Quoting Jérémy Lal (2014-07-07 14:12:19) Consider also the case when arch:all package require a build dependency on a package that only builds on some archs, to prevent the arch:all package being available on archs where its dependencies are not. nodejs and node-* packages are such an example. could you clarify this case? An arch:all package cannot require a build dependency because an arch:all package is a binary package. I can also not find any nodejs packages in the lists I posted. In fact, nodejs package have not yet been a problem with respect to bootstrapping - probably because they are mostly arch:all? Yep, sorry for the confusion. Jérémy. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/1404740047.4416.11.camel@imac.chaumes
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 7, 2014 at 4:51 AM, Johannes Schauer j.scha...@email.de wrote: == openldap_2.4.39-1.arch-all.unusedbd == debconf-utils=1.5.53 I think that's valid. According to debian/changelog, that B-D was added long ago for debconf-mergetemplate, but if I'm reading correctly it seems to be unused since switching to dh_installdebconf. A test build with debconf-utils removed succeeded. Fixed in git, thanks! -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/CAMXH3QCfC_4LOipR2ioRLAi=hipyiv+4ro-kzta0ieqyipb...@mail.gmail.com
Re: possible MBF: automatically detecting unused build dependencies
Hi Johannes, On Mon, Jul 07, 2014 at 02:37:02PM +0200, Johannes Schauer wrote: Steve Langasek vor...@debian.org freetds openldap (U) pam unixodbc There seem to still be some false positives here. pam is on the list because of a build-dependency on libdb-dev, freetds and unixodbc are there because of a build-dependency on libreadline-dev. Both of these are metapackages that pull in version-specific -dev packages. So something seems to be wrong with the detection of empty packages yet? Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Steve Langasek (2014-07-07 18:36:50) There seem to still be some false positives here. pam is on the list because of a build-dependency on libdb-dev, freetds and unixodbc are there because of a build-dependency on libreadline-dev. Both of these are metapackages that pull in version-specific -dev packages. So something seems to be wrong with the detection of empty packages yet? Empty packages are not detected. The first phase will find empty packages because they do not contain any files and thus they are detected as build dependencies of which no files were used. Since empty packages are mostly meta packages and we do not want to include them, we replace them by a fake equivs package without dependencies. If the build still succeeds, that means that the meta package is indeed not needed. Lets find out what happens here. Steps to reproduce: $ sudo debootstrap sid debian-sid # the pam build needs /proc mounted $ sudo mount -o bind /proc debian-sid/proc $ sudo chroot debian-sid $ echo deb-src http://ftp.us.debian.org/debian sid main /etc/apt/sources.list $ apt-get update $ apt-get install build-essential equivs $ apt-cache show libdb-dev | grep -v ^Depends: | grep -v ^Conflicts: | equivs-build - $ dpkg -i libdb-dev_5.3.0_amd64.deb $ apt-get build-dep pam $ dpkg -l | grep libdb ii libdb-dev 5.3.0amd64 Dummy package to fulfill package dependencies ii libdb5.3:amd64 5.3.28-5 amd64 Berkeley v5.3 Database Libraries [runtime] $ apt-get source --build pam [...] $ echo $? 0 I get the same effect when replacing pam by freetds and unixodbc and libdb-dev by libreadline-dev. Can you point out at which step my error is? Thanks! cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707180506.14505.22612@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 07, 2014 at 08:05:06PM +0200, Johannes Schauer wrote: Quoting Steve Langasek (2014-07-07 18:36:50) There seem to still be some false positives here. pam is on the list because of a build-dependency on libdb-dev, freetds and unixodbc are there because of a build-dependency on libreadline-dev. Both of these are metapackages that pull in version-specific -dev packages. So something seems to be wrong with the detection of empty packages yet? Empty packages are not detected. The first phase will find empty packages because they do not contain any files and thus they are detected as build dependencies of which no files were used. Since empty packages are mostly meta packages and we do not want to include them, we replace them by a fake equivs package without dependencies. If the build still succeeds, that means that the meta package is indeed not needed. Ah. No, it only means that the package build does not *fail* if the build-dependency is removed. That is not the same thing as saying that the build-dependency is not used. It would of course be better if packages were resilient against breakage in their build-dependencies, instead of misbuilding with features disabled or with wrong fallback code. But this isn't easy to do in all build systems, and anyway this isn't something we routinely audit about our packages; we shouldn't regard this as a bug to be reported without a lot more discussion of how we're going to systematically stay on top of those issues in the future. For your purposes, a better method would be: - identify those build-dependencies which are empty except for /usr/share/doc - for each such package, look at the contents of its direct dependencies - check the build against those contents Lets find out what happens here. Steps to reproduce: $ sudo debootstrap sid debian-sid # the pam build needs /proc mounted $ sudo mount -o bind /proc debian-sid/proc $ sudo chroot debian-sid $ echo deb-src http://ftp.us.debian.org/debian sid main /etc/apt/sources.list $ apt-get update $ apt-get install build-essential equivs $ apt-cache show libdb-dev | grep -v ^Depends: | grep -v ^Conflicts: | equivs-build - $ dpkg -i libdb-dev_5.3.0_amd64.deb $ apt-get build-dep pam $ dpkg -l | grep libdb ii libdb-dev 5.3.0amd64 Dummy package to fulfill package dependencies ii libdb5.3:amd64 5.3.28-5 amd64 Berkeley v5.3 Database Libraries [runtime] $ apt-get source --build pam [...] $ echo $? 0 I get the same effect when replacing pam by freetds and unixodbc and libdb-dev by libreadline-dev. Can you point out at which step my error is? For the case of pam, I would be interested in seeing the full build log to understand how in the world this built successfully without libdb. That's definitely a packaging error on my part, because without libdb, pam_userdb.so should not build, which in turn *should* be treated as a build failure. But I guess I'm not accounting for individual modules in the build output, since in general the greater risk is failing to keep this list in sync with new upstream modules, rather than misbuilding and losing a module from the tree. Thanks, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: possible MBF: automatically detecting unused build dependencies
On Mon, 07 Jul 2014, Johannes Schauer wrote: Empty packages are not detected. The first phase will find empty packages because they do not contain any files and thus they are detected as build dependencies of which no files were used. Since empty packages are mostly meta packages and we do not want to include them, we replace them by a fake equivs package without dependencies. If the build still succeeds, that means that the meta package is indeed not needed. Unfortunately, this is not necessarily the case; some builds systems disable optional functionality if the required build dependency is not present, and still let the build complete. -- Don Armstrong http://www.donarmstrong.com Mozart tells us what it's like to be human, Beethoven tells us what it's like to be Beethoven, and Bach tells us what it's like to be the universe. -- Douglas Adams -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707183337.gn9...@teltox.donarmstrong.com
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Don Armstrong (2014-07-07 20:33:37) On Mon, 07 Jul 2014, Johannes Schauer wrote: Empty packages are not detected. The first phase will find empty packages because they do not contain any files and thus they are detected as build dependencies of which no files were used. Since empty packages are mostly meta packages and we do not want to include them, we replace them by a fake equivs package without dependencies. If the build still succeeds, that means that the meta package is indeed not needed. Unfortunately, this is not necessarily the case; some builds systems disable optional functionality if the required build dependency is not present, and still let the build complete. correct. The exact problem here is that a meta package without any actual files depends on a package without which the build will still successfully complete. Usually, optional build dependencies are not registered as false positives because the first pass does a full build with all build dependencies present. Even if a build dependency could be optional it will not register in this pass because its files will still be used. The problem here is that the optional build dependency is not a direct dependency of the source package. I cannot think of an automated way to catch dependencies of meta packages without which the build will still succeed. This of course except if there was an easy way to compare the build output against the original binary package. But that's doesnt seem possible yet without reproducible builds. cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707190347.14505.76518@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Steve Langasek (2014-07-07 20:22:42) Ah. No, it only means that the package build does not *fail* if the build-dependency is removed. That is not the same thing as saying that the build-dependency is not used. you are correct. I expanded more on this in my other reply to Don Armstrong. It would of course be better if packages were resilient against breakage in their build-dependencies, instead of misbuilding with features disabled or with wrong fallback code. But this isn't easy to do in all build systems, and anyway this isn't something we routinely audit about our packages; we shouldn't regard this as a bug to be reported without a lot more discussion of how we're going to systematically stay on top of those issues in the future. I agree that it should not be a bug if a package does not fail if a certain build dependency is not installed. Nevertheless, those false positives that were generated this way are still useful to be later marked with !profile.stage1 once build profiles are allowed in the archive because they are obviously droppable. About systematically staying on top of those issues I do not know how to proceed. I guess for that we would first need to know how many source packages depend on meta packages which are not completely empty (besides /usr/share/doc) and still silently fail and continue building with reduced features. I did not plan to run this script more often yet. I'll probably do another run after having added a detection for meta packages as suggested by you below. Running it more regularly might not require any big discussion because the problem size might be small enough that maintaining a white list would be enough. For your purposes, a better method would be: - identify those build-dependencies which are empty except for /usr/share/doc - for each such package, look at the contents of its direct dependencies - check the build against those contents While this would often work it would still miss some meta packages which do ship some more files than /usr/share/doc but are otherwise mostly important because of the packages they depend on. Though I guess this would already tremendously decrease the amount of false positive (however high their number is) and I should implement that for the next time I recalculate this. For the case of pam, I would be interested in seeing the full build log to understand how in the world this built successfully without libdb. That's definitely a packaging error on my part, because without libdb, pam_userdb.so should not build, which in turn *should* be treated as a build failure. But I guess I'm not accounting for individual modules in the build output, since in general the greater risk is failing to keep this list in sync with new upstream modules, rather than misbuilding and losing a module from the tree. Sorry, I already deleted my chroot :( cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707191444.14505.34985@hoothoot
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 7, 2014 at 11:22:42 -0700, Steve Langasek wrote: For the case of pam, I would be interested in seeing the full build log to understand how in the world this built successfully without libdb. That's definitely a packaging error on my part, because without libdb, pam_userdb.so should not build, which in turn *should* be treated as a build failure. But I guess I'm not accounting for individual modules in the build output, since in general the greater risk is failing to keep this list in sync with new upstream modules, rather than misbuilding and losing a module from the tree. dh_install --fail-missing would help for the case where new modules are added and need to be accounted for in the packaging. Cheers, Julien signature.asc Description: Digital signature
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 07, 2014 at 09:46:47PM +0200, Julien Cristau wrote: On Mon, Jul 7, 2014 at 11:22:42 -0700, Steve Langasek wrote: For the case of pam, I would be interested in seeing the full build log to understand how in the world this built successfully without libdb. That's definitely a packaging error on my part, because without libdb, pam_userdb.so should not build, which in turn *should* be treated as a build failure. But I guess I'm not accounting for individual modules in the build output, since in general the greater risk is failing to keep this list in sync with new upstream modules, rather than misbuilding and losing a module from the tree. dh_install --fail-missing would help for the case where new modules are added and need to be accounted for in the packaging. Yes, I'm aware of the available techniques here; I am just not convinced that a 40+ line debian/libpam-modules.install, with architecture-dependent contents, is preferable to a two-line file with a single glob in terms of overall maintainability. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: possible MBF: automatically detecting unused build dependencies
On Mon, Jul 07, 2014 at 09:14:44PM +0200, Johannes Schauer wrote: Quoting Steve Langasek (2014-07-07 20:22:42) Ah. No, it only means that the package build does not *fail* if the build-dependency is removed. That is not the same thing as saying that the build-dependency is not used. you are correct. I expanded more on this in my other reply to Don Armstrong. It would of course be better if packages were resilient against breakage in their build-dependencies, instead of misbuilding with features disabled or with wrong fallback code. But this isn't easy to do in all build systems, and anyway this isn't something we routinely audit about our packages; we shouldn't regard this as a bug to be reported without a lot more discussion of how we're going to systematically stay on top of those issues in the future. I agree that it should not be a bug if a package does not fail if a certain build dependency is not installed. Nevertheless, those false positives that were generated this way are still useful to be later marked with !profile.stage1 once build profiles are allowed in the archive because they are obviously droppable. No, marking build-dependencies with build profiles should be driven by what is needed to break build-dep loops, not by what build-deps are droppable without further modification of the packages. Excessive stage1 profile tagging will result in unnecessary extra builds during a bootstrap. I did not plan to run this script more often yet. I'll probably do another run after having added a detection for meta packages as suggested by you below. Running it more regularly might not require any big discussion because the problem size might be small enough that maintaining a white list would be enough. If you really want to systematically detect misbuilds on missing build-dependencies, it's a much bigger problem than just detecting this for the build-dependencies which are metapackages. In my personal opinion, this is not worth doing. But if you are going to do it, be comprehensive - anything less than that is counterproductive. For your purposes, a better method would be: - identify those build-dependencies which are empty except for /usr/share/doc - for each such package, look at the contents of its direct dependencies - check the build against those contents While this would often work it would still miss some meta packages which do ship some more files than /usr/share/doc but are otherwise mostly important because of the packages they depend on. Though I guess this would already tremendously decrease the amount of false positive (however high their number is) and I should implement that for the next time I recalculate this. There certainly will be other cases that this technique won't cover, but it won't cause any false-negatives for your purposes. I don't know what the numbers will look like overall, but 3 out of 4 packages listed by my name were false-positives that can be filtered out this way, so I definitely think it's worth a rerun before MBF. For the case of pam, I would be interested in seeing the full build log to understand how in the world this built successfully without libdb. That's definitely a packaging error on my part, because without libdb, pam_userdb.so should not build, which in turn *should* be treated as a build failure. But I guess I'm not accounting for individual modules in the build output, since in general the greater risk is failing to keep this list in sync with new upstream modules, rather than misbuilding and losing a module from the tree. Sorry, I already deleted my chroot :( No worries... as previously suggested this is not high on my priority list ;) -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: possible MBF: automatically detecting unused build dependencies
+++ Steve Langasek [2014-07-07 15:07 -0700]: On Mon, Jul 07, 2014 at 09:14:44PM +0200, Johannes Schauer wrote: I agree that it should not be a bug if a package does not fail if a certain build dependency is not installed. Nevertheless, those false positives that were generated this way are still useful to be later marked with !profile.stage1 once build profiles are allowed in the archive because they are obviously droppable. No, marking build-dependencies with build profiles should be driven by what is needed to break build-dep loops, not by what build-deps are droppable without further modification of the packages. Excessive stage1 profile tagging will result in unnecessary extra builds during a bootstrap. Whilst I agree that a build-dep being droppable is not necessarily sufficient reason to mark it as a stage1 profile, I don't agree with your reasoning. Having a profile available does not mean that it will necessarily be used/built. The path through the bootstrap is determined by currently-buildable and profilable packages. Having more profiles marked just potentially gives us more possible routes through the dependecny graph, which (up to a point) is generally a useful thing with a new arch as you don't always know in advance which packages will be most trouble. The chosen route is unlikely to use all the profiles available unless there really are only just enough to do it at all. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140707224831.gt10...@stoneboat.aleph1.co.uk
Re: possible MBF: automatically detecting unused build dependencies
Hi Johannes, On Montag, 7. Juli 2014, Johannes Schauer wrote: About systematically staying on top of those issues I do not know how to proceed. I guess for that we would first need to know how many source packages depend on meta packages which are not completely empty (besides /usr/share/doc) and still silently fail and continue building with reduced features. I did not plan to run this script more often yet. I'll probably do another run after having added a detection for meta packages as suggested by you below. Running it more regularly might not require any big discussion because the problem size might be small enough that maintaining a white list would be enough. I'd be happy to assist you getting it to run on jenkins.d.n, which is now a 15 core / 30gb host, which can easily get a job configured testing this every three months (or whenever) on half the ressources or so Please read the existing documentation (about link etc) and ping me (best via (#)debian-qa) for further kickstarting help..! cheers, Holger signature.asc Description: This is a digitally signed message part.
Re: possible MBF: automatically detecting unused build dependencies
Hi, Quoting Steve Langasek (2014-07-08 00:07:29) On Mon, Jul 07, 2014 at 09:14:44PM +0200, Johannes Schauer wrote: Nevertheless, those false positives that were generated this way are still useful to be later marked with !profile.stage1 once build profiles are allowed in the archive because they are obviously droppable. No, marking build-dependencies with build profiles should be driven by what is needed to break build-dep loops, not by what build-deps are droppable without further modification of the packages. Excessive stage1 profile tagging will result in unnecessary extra builds during a bootstrap. Bootstrapping should not get into the way of the normal packaging. There should be strong reasons before a disruptive patch is applied to make a package build with fewer build dependencies. This class of cases that is found by this script (and it could be abused to find even more by omitting the first phase) sounds to me as if only trivial changes were needed to build the package with fewer build dependencies. Thus in case the maintainer does not have any strong objections, there is no harm applying them. The downside you list unnecessary extra builds are easy to avoid in practice. Botch contains algorithms to find a close to minimum set of source packages to modify to make a dependency graph acyclic (a feedback vertex set). Even if all source packages in the dependency graph had removable build dependencies it would only choose a small set of them (currently 60-70 are needed) to actually build with a stage1 profile active. Furthermore, the only time when there is a hard requirement to remove a dependency without alternative are self cycles which currently involve only about 30 source packages. For all other source packages there are always alternatives. So you will mostly not get into a situation where you absolutely need to break a build-dep loop as you describe it above (aside from these 30 source packages). I did not plan to run this script more often yet. I'll probably do another run after having added a detection for meta packages as suggested by you below. Running it more regularly might not require any big discussion because the problem size might be small enough that maintaining a white list would be enough. If you really want to systematically detect misbuilds on missing build-dependencies, it's a much bigger problem than just detecting this for the build-dependencies which are metapackages. Why? The build dependencies which are not metapackages are not listed as false positives because they get weeded out in the first phase. There certainly will be other cases that this technique won't cover, but it won't cause any false-negatives for your purposes. I don't know what the numbers will look like overall, but 3 out of 4 packages listed by my name were false-positives that can be filtered out this way, so I definitely think it's worth a rerun before MBF. You are right. I'll see what I can do. Thanks a lot for your input! cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140708042249.14505.27900@hoothoot
