Re: jessie release goal: verbose build logs
Le 14/06/2013 15:19, Timo Juhani Lindfors a écrit : This doesn't really help when trying to diagnose things, and even for successful builds it's valuable to have the complete build log, including the parts how the upstream build system is called from the Debian packaging. This is a useful goal. However, since fixing many different source packages is a lot of work I recently explored some alternative approaches. For example, with systemtap we can simply log all execve() invocations complete with timing information and end up with fully structured build logs. Here's example output from the proof of concept from a few years ago: http://lindi.iki.fi/lindi/structured-buildlogs/logs/hello-2.6-1_amd64.build How do you do that exactly? Cheers, -- Stéphane -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/52837b52.4030...@debian.org
Re: jessie release goal: verbose build logs
Stéphane Glondu glo...@debian.org writes: http://lindi.iki.fi/lindi/structured-buildlogs/logs/hello-2.6-1_amd64.build How do you do that exactly? Can't remember the exact details but you can get the script with git clone http://lindi.iki.fi/lindi/git/structured-buildlogs.git/ -Timo -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/848uwshwfm@sauna.l.org
Re: jessie release goal: verbose build logs
(Resending; first copy failed due to MIME non-8-bit-cleanliness damage.) Joey Hess writes (Re: jessie release goal: verbose build logs): I've attached a simple proof of concept you can try it out with building your own packages. For example: dpkg-buildpackage | ssh (The implementation needs to be improved; it should read both stdout and stderr and multiplex them properly. And it should check if stdout is not to a TTY, and if so avoid munging the build log output. The only other problem is that `make` outputs the lines it runs to stderr and so those continue to pollute the build display.) I like the idea, but you need to make this an adverbial command, not a pipe thing. This is so that if you _aren't_ \r-mangling, you can pass dpkg-buildpackage the same actual fd for both stdin and stdout, so that the interleaving of stdout and stderr is correct. Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/21011.33534.616618.445...@chiark.greenend.org.uk
Re: jessie release goal: verbose build logs
On Wed, 14 Aug 2013, Guillem Jover wrote: Definitely, and planned (#476221) to be fixed pretty soon. But the problem is that this can not be enabled by default (only through an option initially, until the upper layers rely on a dpkg-buildpackage with such option and use it) because the different tools might fight over the file and might either truncate it or write duplicate data, etc. Using a different path might be an issue for builds producing huge log files. Great to see that you want to take care of this, but can you let us know how you plan to let dpkg-buildpackage cooperate with the upper layer tools to have a full log file that can be added to .changes? If we don't care about adding it to .changes, then it's trivial, dpkg-buildpackage can just ignore the file, but otherwise we need some way to transfer the control of the log file or something similar. The requirements IMO is to have the start of the log from the upper layer (i.e. sbuild setup of build dependencies) but also have the log properly closed at the time dpkg-buildpackage runs dpkg-genchanges, because once it has been added to the .changes it should not be further modified obviously. Cheers, -- Raphaël Hertzog ◈ Debian Developer Discover the Debian Administrator's Handbook: → http://debian-handbook.info/get/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130818155954.gb7...@x230-buxy.home.ouaza.com
Re: Opt-out or Opt-in vebose build logs Re: jessie release goal: verbose build logs
Dmitrijs Ledkovs wrote: We have build log analyzers running for the build logs. Checking heuristically for problems you know about does not help find the class of problems you don't know about yet. How about a new DEB_BUILD_OPTION=silent which opts into silent build log? Does that sound reasonable. I don't see how it could seem reasonable if you've read my message and followed my reasoning. -- see shy jo signature.asc Description: Digital signature
Re: Opt-out or Opt-in vebose build logs Re: jessie release goal: verbose build logs
Am 13.08.2013 21:10, schrieb Adam Borowski: So I guess it would be best to put the threshold at automated vs human-supervised builds. What about setting the flag per-tool rather than per-deployment? For example, pbuilder would default to verbose (as you can't restart builds) while dpkg-buildpackage in the absence of inherited settings would default to terse. This way we'd be spared the spam during development. If you want to go this road, then it would be per use-case, not per tool. When looking at a QA maintained package, or looking at an unfixed RC issue in a package I do not maintain, I want to have as much information as possible. Filtering this information on my own is an easy task, however trying to figure out how to make a build to print all information usually takes a bit longer. So probably the package maintainer likes the terse build logs too, but he is usually the one who can make them terse without any effort. However people occasionally looking at a package are much better served with a complete build log. Working all the time with dpkg-buildpackage, so even there the default should be the one which is the most useful for the majority of use cases. Matthias -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/520c9fab.5060...@debian.org
Re: jessie release goal: verbose build logs
Hi, On Tue, 13 Aug 2013, Russ Allbery wrote: debuild, pbuilder, and related packages currently use ../package_version_arch.build for this output. Maybe the lower-level build architecture should take over maintaining that log file instead and those tools should be modified to assume that's already happening and move that log file around like the other build products? While I agree that dpkg-buildpackage should have that feature, it's also clear that those higher level tools have their output of their own that is likely to be interesting as well. I really want the build log to include the installation of the build dependencies for instance. So there should be some coordination between both levels. Maybe the higher level tools could hand over a file descriptor for the build log that they started filling before calling dpkg-buildpackage. Also if we use the same .build extensions for dpkg-buildpackage there will need to have some sort of transition to ensure all other tools creating those are correctly cooperating with dpkg-buildpackage. Cheers, -- Raphaël Hertzog ◈ Debian Developer Discover the Debian Administrator's Handbook: → http://debian-handbook.info/get/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130815101620.ga11...@x230-buxy.home.ouaza.com
Re: jessie release goal: verbose build logs
Hi! On Tue, 2013-08-13 at 15:16:49 -0700, Russ Allbery wrote: Joey Hess jo...@debian.org writes: Also it could save the whole log to someplace when minimizing the console output, to refer back to. Perhaps ../$package_$version.buildlog (which gets a step closer to including that in the dsc and uploading local build logs..) debuild, pbuilder, and related packages currently use ../package_version_arch.build for this output. Maybe the lower-level build architecture should take over maintaining that log file instead and those tools should be modified to assume that's already happening and move that log file around like the other build products? Definitely, and planned (#476221) to be fixed pretty soon. But the problem is that this can not be enabled by default (only through an option initially, until the upper layers rely on a dpkg-buildpackage with such option and use it) because the different tools might fight over the file and might either truncate it or write duplicate data, etc. Using a different path might be an issue for builds producing huge log files. Thanks, Guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130814091907.ga31...@gaara.hadrons.org
Re: jessie release goal: verbose build logs
Hi! On Tue, 2013-08-13 at 22:08:34 +0200, Joey Hess wrote: At DebConf, Enrico came up with this idea: When the package is building with the display going to the console, something could intercept the stdout and convert \n to \r. Let stderr through untouched. The result would be a build that makes it *very* easy to focus on the warnings, while still providing a progress display of what is going on at any given moment in the build. I'd like to see dpkg-buildpackage modified to use something like this by default, because it would reduce the level of cruft we wade through (often up to our eyeballs) when building packages, and allow concentrating on what's important. At the same time, it allows adding as verbose output as we like and getting that saved in the build logs for when a detailed analysis is needed. I'd probably be fine with something like this as an option in dpkg-buildpackage, which ideally could be also enabled through its future config file. But I'd really not be comfortable enabling it by default, because this seems fragile and assumes upstream build systems are sane and have consistent behavior, or the build might be so fast you cannot see what's going on anyway compared to being able to scroll back a bit (w/o reading the build log file). Thanks, Guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130814105632.gb31...@gaara.hadrons.org
Re: Opt-out or Opt-in vebose build logs Re: jessie release goal: verbose build logs
Adam Borowski kilobyte at angband.pl writes: Non-spammy builds are better for builds done by a human. No. They’re better for builds done by the developers of the upstream software on their own systems where they know what they’re doing. And even then, it’s a big maybe. Every other human is a packager or porter (in a broad sense… if I install Linux From Scratch then compile something I want to use on it, I’m one), and those *require* verbose output, e.g. to verify that the buildsystem didn’t eat $CFLAGS. I can live with an optional 'silent' build option, but I personally think everyone – especially(!) the package maintainers – should have no need to use it. bye, //mirabilos -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/loom.20130814t173844-...@post.gmane.org
Re: jessie release goal: verbose build logs
On 17 June 2013 23:58, Dmitrijs Ledkovs x...@debian.org wrote: tags 680686 patch thanks On 14 June 2013 12:35, Matthias Klose d...@debian.org wrote: - Fix debhelper not passing --disable-silent-rules by default. #680686 I think cdbs already does this. Patch attached for autoconf dh build system. cmake dh build system seems to be already enabling verbose makefiles. Is there any reason this hasn't been applied yet? Can I NMU this, as debhelper is marked as LowNMU package. The patch itself is a trivial one-liner and fixes a Jessie release goal. The bug report itself was filed a little over one year ago. Regards, Dmitrijs. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/canbhluij3-_x_fe0dq6v2nu7duma9gfsh_ze7yeogit9abf...@mail.gmail.com
Re: jessie release goal: verbose build logs
Dmitrijs Ledkovs wrote: Is there any reason this hasn't been applied yet? Can I NMU this, as debhelper is marked as LowNMU package. Not for reasons such as allowing patches like this. Making all builds verbose by default has both advantages and disadvantages. The disadvantages include making builds possibly so noisy that when one runs them during daily work, once ignores all output. Including important compiler warnings. (This is the same reason why it's a bad idea to let a codebase accumulate a lot of compiler warnings!) I'd be ok with DH_VERBOSE enabling the verbose behavior, and the buildds could then enable it. -- see shy jo signature.asc Description: Digital signature
Opt-out or Opt-in vebose build logs Re: jessie release goal: verbose build logs
On 13 August 2013 14:36, Joey Hess jo...@debian.org wrote: Dmitrijs Ledkovs wrote: Is there any reason this hasn't been applied yet? Can I NMU this, as debhelper is marked as LowNMU package. Not for reasons such as allowing patches like this. Ok. Making all builds verbose by default has both advantages and disadvantages. I agree, there should be a way to toggle this. The disadvantages include making builds possibly so noisy that when one runs them during daily work, once ignores all output. Including important compiler warnings. We have build log analyzers running for the build logs. And the important compiler warnings (errors) fail the build. If we make this opt-in, we will fail to achieve this goal. As when one is debugging a failed build (e.g. the failure in the last hour of webkit/qt/libreoffice compile on armhf porter box, or by hand ./debian/rules build) that's when one would start caring wishing to have the verbose output would have been set, but it would be too late. The way I interpret this goal is to have the build logs verbose by default and/or opt-out. Making this opt-in, will need many machines modified - potentially all the builders / CI integration. It's not just about Debian buildd network, but anyone who rebuilds debian packages or derives from Debian. Including all the ways people integrate and build debian packages. (This is the same reason why it's a bad idea to let a codebase accumulate a lot of compiler warnings!) I'd be ok with DH_VERBOSE enabling the verbose behavior, and the buildds could then enable it. I'm against re-using DH_VERBOSE variable: * DH_VERBOSE has explicit meaning to control output of dh_* commands. DH_VEBOSE=1 should print how/what dh_auto_* commands invokes. It should not change the commands it invokes. * this debian goal is not debhelper/cdbs/etc specific, but helper/build-system agnostic. How about a new DEB_BUILD_OPTION=silent which opts into silent build log? Does that sound reasonable. For a policy change, I am hoping to mandatory require verbose build by default, and optionally supporting DEB_BUILD_OPTION=silent. Regards, Dmitrijs. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/canbhlujxxzznj6tbmwvosrgra4jbdaa2oxtqi6futyf7t7z...@mail.gmail.com
Re: Opt-out or Opt-in vebose build logs Re: jessie release goal: verbose build logs
[why oh why are you breaking threading?] On Tue, Aug 13, 2013 at 19:51:52 +0200, Dmitrijs Ledkovs wrote: We have build log analyzers running for the build logs. And the important compiler warnings (errors) fail the build. If we make this opt-in, we will fail to achieve this goal. As when one is debugging a failed build (e.g. the failure in the last hour of webkit/qt/libreoffice compile on armhf porter box, or by hand ./debian/rules build) that's when one would start caring wishing to have the verbose output would have been set, but it would be too late. I don't see how. Either dpkg-buildpackage -nc or re-running debian/rules build with the option set would give you what you want. Cheers, Julien signature.asc Description: Digital signature
Re: Opt-out or Opt-in vebose build logs Re: jessie release goal: verbose build logs
On 13 August 2013 19:59, Julien Cristau jcris...@debian.org wrote: [why oh why are you breaking threading?] On Tue, Aug 13, 2013 at 19:51:52 +0200, Dmitrijs Ledkovs wrote: We have build log analyzers running for the build logs. And the important compiler warnings (errors) fail the build. If we make this opt-in, we will fail to achieve this goal. As when one is debugging a failed build (e.g. the failure in the last hour of webkit/qt/libreoffice compile on armhf porter box, or by hand ./debian/rules build) that's when one would start caring wishing to have the verbose output would have been set, but it would be too late. I don't see how. Either dpkg-buildpackage -nc or re-running debian/rules build with the option set would give you what you want. For well behaved build-systems that would only show flags for the yet to compile object, where the bug might be in the flags/libs used for the already compiled objects. (i don't have an example here, something like mixing with/without -fPIC but we get warnings which objects are at fault anyway) For less behaved build-systems this may cause a reconfigure and restart the whole build. Which is suboptimal. In the past there have been random bugs / build failures which are not reproducible on re-runs (but I've yet to see one which depended on build-flags, unless one gets different flags =/ on reruns which wouldn't know about) In controlled environments, one doesn't get to re-run a build, as the instances are stripped-down and destroyed on build failure E.g. all the jenkins instances running debian package builds, PPAs, auto-package-tests, automated cross-builders. One would have to modify each and every deployment of those... Somehow we lived fine with verbose build-logs, until automake silent rules and a few other build-systems started to become more silent. I can understand the appeal of silent builds for upstream developers, but it makes zero sense for a distribution or package maintainers or our downstream. Regards, Dmitrijs. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CANBHLUi1+Q8PVTJF2TvhGKBLQh-y=elhrzpuv-7ntjlypch...@mail.gmail.com
Re: Opt-out or Opt-in vebose build logs Re: jessie release goal: verbose build logs
On Tue, Aug 13, 2013 at 08:30:16PM +0200, Dmitrijs Ledkovs wrote: In controlled environments, one doesn't get to re-run a build, as the instances are stripped-down and destroyed on build failure E.g. all the jenkins instances running debian package builds, PPAs, auto-package-tests, automated cross-builders. One would have to modify each and every deployment of those... Somehow we lived fine with verbose build-logs, until automake silent rules and a few other build-systems started to become more silent. I can understand the appeal of silent builds for upstream developers, but it makes zero sense for a distribution or package maintainers or our downstream. Non-spammy builds are better for builds done by a human. Spammy builds work around the inability to restart on a failure in automated builds. Trying to spot an irregularity can be really hard visually if you have half a page of switches per every file. With a terse build like: CC foo CC bar CC baz CC quux LD blah any messages stand out. And if a failure does happen, it's a matter of, typically, make V=y. Too bad you don't have this option on a buildd. So I guess it would be best to put the threshold at automated vs human-supervised builds. What about setting the flag per-tool rather than per-deployment? For example, pbuilder would default to verbose (as you can't restart builds) while dpkg-buildpackage in the absence of inherited settings would default to terse. This way we'd be spared the spam during development. -- ᛊᚨᚾᛁᛏᚣ᛫ᛁᛊ᛫ᚠᛟᚱ᛫ᚦᛖ᛫ᚹᛖᚨᚲ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130813191008.ga...@angband.pl
Re: jessie release goal: verbose build logs
Joey Hess wrote:
Making all builds verbose by default has both advantages and
disadvantages.
The disadvantages include making builds possibly so noisy that when one
runs them during daily work, once ignores all output. Including
important compiler warnings.
(This is the same reason why it's a bad idea to let a codebase
accumulate a lot of compiler warnings!)
At DebConf, Enrico came up with this idea: When the package is building
with the display going to the console, something could intercept the
stdout and convert \n to \r. Let stderr through untouched. The result
would be a build that makes it *very* easy to focus on the warnings, while
still providing a progress display of what is going on at any given
moment in the build.
I've attached a simple proof of concept you can try it out with building
your own packages. For example:
dpkg-buildpackage | ssh
(The implementation needs to be improved; it should read both stdout and
stderr and multiplex them properly. And it should check if stdout is not
to a TTY, and if so avoid munging the build log output. The only other
problem is that `make` outputs the lines it runs to stderr and so those
continue to pollute the build display.)
I'd like to see dpkg-buildpackage modified to use something like this by
default, because it would reduce the level of cruft we wade through
(often up to our eyeballs) when building packages, and allow
concentrating on what's important. At the same time, it allows adding as
verbose output as we like and getting that saved in the build logs for
when a detailed analysis is needed.
--
see shy jo
#!/usr/bin/perl
$|=1;
my $width=$ENV{COLUMNS} || 80;
while () {
chomp;
print substr($_, 0, $width - 1).( x ($width - 1 - length $_)).\r;
}
signature.asc
Description: Digital signature
Re: jessie release goal: verbose build logs
Joey Hess wrote: (The implementation needs to be improved; it should read both stdout and stderr and multiplex them properly. And it should check if stdout is not to a TTY, and if so avoid munging the build log output. The only other problem is that `make` outputs the lines it runs to stderr and so those continue to pollute the build display.) Also it could save the whole log to someplace when minimizing the console output, to refer back to. Perhaps ../$package_$version.buildlog (which gets a step closer to including that in the dsc and uploading local build logs..) -- see shy jo signature.asc Description: Digital signature
Re: jessie release goal: verbose build logs
Joey Hess jo...@debian.org writes: Joey Hess wrote: (The implementation needs to be improved; it should read both stdout and stderr and multiplex them properly. And it should check if stdout is not to a TTY, and if so avoid munging the build log output. The only other problem is that `make` outputs the lines it runs to stderr and so those continue to pollute the build display.) Also it could save the whole log to someplace when minimizing the console output, to refer back to. Perhaps ../$package_$version.buildlog (which gets a step closer to including that in the dsc and uploading local build logs..) debuild, pbuilder, and related packages currently use ../package_version_arch.build for this output. Maybe the lower-level build architecture should take over maintaining that log file instead and those tools should be modified to assume that's already happening and move that log file around like the other build products? -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87siyd6xta@windlord.stanford.edu
Re: jessie release goal: verbose build logs
This is now documented at http://wiki.debian.org/ReleaseGoals/VerboseBuildLogs -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/51d6afef.8060...@debian.org
Re: jessie release goal: verbose build logs
On 14/06/2013 13:49, Wookey wrote: +++ Matthias Klose [2013-06-14 13:35 +0200]: Much more often than I do like it, I see bug reports for the toolchain just pointing to a build log. Then looking at the build log, you often just see CC ... CCLD ... (sometimes even colorized) [...] I'll second this. Like Doko, I've spent way too much time recently(ish) staring at build-logs. The short-form build logs are often useless for working out why something isn't working, or comparing successful native builds with failed cross builds, for example. Same as Wookey. This would save me some time working on the analysis of rebuilds with clang. Late, but still: I am also very much in favour of this, as the lack of full command lines has already caused quite a bit of overhead in my experiments with our goto-cc compiler infrastructure. Best, Michael pgp8GGKBTjSqy.pgp Description: PGP signature
Re: jessie release goal: verbose build logs
On Fri, 14 Jun 2013 14:14:39 +0200, Jakub Wilk wrote: - File and track issues for packages not enabling verbose builds. https://buildd.debian.org/~brlink/bytag/W-compiler-flags-hidden.html I attached a dd-list for the lazy. But note that false positives are possible, especially when building in parallel. Debian Perl Group pkg-perl-maintain...@lists.alioth.debian.org libconvert-binary-c-perl Fixed and uploaded. libhtml-template-pro-perl False positive. libwx-scintilla-perl Fixed in git. Cheers, gregor -- .''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06 : :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/ `. `' Member of VIBE!AT SPI, fellow of the Free Software Foundation Europe `- NP: Bruce Springsteen: I'm Goin' Down signature.asc Description: Digital signature
Re: jessie release goal: verbose build logs
On Fri, Jun 14, 2013 at 14:14:39 +0200, Jakub Wilk wrote: Debian X Strike Force debia...@lists.debian.org [long list] Working on fixing these (adding --disable-silent-rules) as I update them. Will take a while though... Cheers, Julien signature.asc Description: Digital signature
jessie release goal: verbose build logs
Much more often than I do like it, I see bug reports for the toolchain just pointing to a build log. Then looking at the build log, you often just see CC ... CCLD ... (sometimes even colorized) This doesn't really help when trying to diagnose things, and even for successful builds it's valuable to have the complete build log, including the parts how the upstream build system is called from the Debian packaging. Verbose build logs allow to analyse package builds and give hints to more issues regarding the build, especially for the hardening flags. The lintian hardening checks are incomplete, because they rely on the inspection of the generated binaries, which may be incomplete especially for many plugins or dynamically loadable extensions. So I'm proposing for jessie: - File and track issues for packages not enabling verbose builds. https://buildd.debian.org/~brlink/bytag/W-compiler-flags-hidden.html - Fix debhelper not passing --disable-silent-rules by default. #680686 I think cdbs already does this. - Fix debhelper to show how the upstream build system is called. #680687 As an alternative buildds could run with DEB_BUILD_OPTIONS=verbose - Change Debian policy to recommend or require verbose build logs. #628515 This is not rocket science, but allows for almost free additional QA. Size of the build logs shouldn't be an issue, but if it is, I'm considering to disable compiler warning and errors messages by default, unless DEB_BUILD_OPTIONS=verbose is set. Matthias -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/51bb0006.4080...@debian.org
Re: jessie release goal: verbose build logs
+++ Matthias Klose [2013-06-14 13:35 +0200]: Much more often than I do like it, I see bug reports for the toolchain just pointing to a build log. Then looking at the build log, you often just see CC ... CCLD ... (sometimes even colorized) This doesn't really help when trying to diagnose things, and even for successful builds it's valuable to have the complete build log, including the parts how the upstream build system is called from the Debian packaging. So I'm proposing for jessie: - File and track issues for packages not enabling verbose builds. https://buildd.debian.org/~brlink/bytag/W-compiler-flags-hidden.html - Fix debhelper not passing --disable-silent-rules by default. #680686 I think cdbs already does this. - Fix debhelper to show how the upstream build system is called. #680687 As an alternative buildds could run with DEB_BUILD_OPTIONS=verbose - Change Debian policy to recommend or require verbose build logs. #628515 I'll second this. Like Doko, I've spent way too much time recently(ish) staring at build-logs. The short-form build logs are often useless for working out why something isn't working, or comparing successful native builds with failed cross builds, for example. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130614114927.gx14...@stoneboat.aleph1.co.uk
Re: jessie release goal: verbose build logs
Hi, On 14/06/13 13:35, Matthias Klose wrote: Much more often than I do like it, I see bug reports for the toolchain just pointing to a build log. Then looking at the build log, you often just see CC ... CCLD ... (sometimes even colorized) This doesn't really help when trying to diagnose things, and even for successful builds it's valuable to have the complete build log, including the parts how the upstream build system is called from the Debian packaging. Verbose build logs allow to analyse package builds and give hints to more issues regarding the build, especially for the hardening flags. The lintian hardening checks are incomplete, because they rely on the inspection of the generated binaries, which may be incomplete especially for many plugins or dynamically loadable extensions. So I'm proposing for jessie: - File and track issues for packages not enabling verbose builds. https://buildd.debian.org/~brlink/bytag/W-compiler-flags-hidden.html - Fix debhelper not passing --disable-silent-rules by default. #680686 I think cdbs already does this. It has done so since version 0.4.62 from 2009. - Fix debhelper to show how the upstream build system is called. #680687 As an alternative buildds could run with DEB_BUILD_OPTIONS=verbose - Change Debian policy to recommend or require verbose build logs. #628515 Completely agreed on everything. I thought this was already a requirement. This is not rocket science, but allows for almost free additional QA. Size of the build logs shouldn't be an issue, but if it is, I'm considering to disable compiler warning and errors messages by default, unless DEB_BUILD_OPTIONS=verbose is set. If size is an issue, xz-compression could probably help a lot. Cheers, Emilio -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/51bb0535.1080...@debian.org
Re: jessie release goal: verbose build logs
On 14/06/2013 13:49, Wookey wrote: +++ Matthias Klose [2013-06-14 13:35 +0200]: Much more often than I do like it, I see bug reports for the toolchain just pointing to a build log. Then looking at the build log, you often just see CC ... CCLD ... (sometimes even colorized) [...] I'll second this. Like Doko, I've spent way too much time recently(ish) staring at build-logs. The short-form build logs are often useless for working out why something isn't working, or comparing successful native builds with failed cross builds, for example. Same as Wookey. This would save me some time working on the analysis of rebuilds with clang. Thanks Sylvestre -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/51bb0507.5070...@debian.org
Re: jessie release goal: verbose build logs
On 2013-06-14, Jakub Wilk jw...@debian.org wrote: I attached a dd-list for the lazy. But note that false positives are possible, especially when building in parallel. I've just sample-checked 4 packages I'm involved in and 100% of those four I checked was false positives. We need a better detection. But beside that, I agree with wanting verbose build logs. /Sune -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnkrm2nj.j8.nos...@sshway.ssh.pusling.com
Re: jessie release goal: verbose build logs
Matthias Klose d...@debian.org writes: This doesn't really help when trying to diagnose things, and even for successful builds it's valuable to have the complete build log, including the parts how the upstream build system is called from the Debian packaging. This is a useful goal. However, since fixing many different source packages is a lot of work I recently explored some alternative approaches. For example, with systemtap we can simply log all execve() invocations complete with timing information and end up with fully structured build logs. Here's example output from the proof of concept from a few years ago: http://lindi.iki.fi/lindi/structured-buildlogs/logs/hello-2.6-1_amd64.build Don't take me wrong, I don't intend to replace any of the existing infrastructure, I'm simply exploring new ways to (ab)use systemtap :-) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/84y5aczv7c@sauna.l.org
Re: jessie release goal: verbose build logs
Hi, I completely agree on the goal and disable silent rules where I notice them, but: On Fri, Jun 14, 2013 at 01:35:34PM +0200, Matthias Klose wrote: - Change Debian policy to recommend or require verbose build logs. #628515 Change buildds. Do we have autosiging now for all buildds? TTBOMK one ia64 buildd maintainer doesn't want it, so buildd log size restrictions block bigger logs (and that will happen e.g. for LO, bddt). That's why LO does verbose on normal build and not-verbose on the buildds. Regards, Rene -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130614140840.ga26...@rene-engelhard.de
Re: jessie release goal: verbose build logs
Hi I agree we need good build logs. On 14-06-13 14:14, Jakub Wilk wrote: * Matthias Klose d...@debian.org, 2013-06-14, 13:35: So I'm proposing for jessie: - File and track issues for packages not enabling verbose builds. https://buildd.debian.org/~brlink/bytag/W-compiler-flags-hidden.html I attached a dd-list for the lazy. But note that false positives are possible, especially when building in parallel. False positives also happen for other compilers, at least for Free Pascal. AIUI, that is why I show up in that list. Paul signature.asc Description: OpenPGP digital signature
Re: jessie release goal: verbose build logs
* Matthias Klose d...@debian.org [130614 13:36]: Verbose build logs allow to analyse package builds and give hints to more issues regarding the build, especially for the hardening flags. The lintian hardening checks are incomplete, because they rely on the inspection of the generated binaries, which may be incomplete especially for many plugins or dynamically loadable extensions. While I agree that a build log without compiler arguments is essentially worthless, could we please not call something that is not too silent verbose? Verbose is something that uses more words as necessary, while compiler options are neccessary, without them users of the software cannot get help in forums or channels when they have problems compiling the software and with build logs porters can often hardly help if the old buildd log contains no information. Bernhard R. Link -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130614223601.gb3...@client.brlink.eu
