Re: Parallel build results
On 01/01/08 at 18:23 -0800, Russ Allbery wrote: Aurelien Jarno [EMAIL PROTECTED] writes: On the other hand, DEB_BUILD_OPTIONS=parallel=n was ignored by packages that have not been validated by the maintainers, and used by packages that have been tested by the maintainer. Also it was possible to use only on some parts of the build. For example the make install part is sometimes problematic, and mainly depends on I/O throughput of the machine, and not CPU. This make some problems difficult to detect, as they happen only in rare case on some machines. Unfortunately dpkg-buildpackage now looks for DEB_BUILD_OPTIONS and call debian/rules with -j and the value it founds in this environment variable. That makes DEB_BUILD_OPTIONS=parallel=n useless, as it has the same effect as dpkg-buildpackage -j. I personally think that supporting DEB_BUILD_OPTIONS=parallel=n on some packages is a first goal to achieve. Supporting this option in the 100 packages which take the longest to build would probably be a huge improvement in build time of the whole archive, while being easier than fixing all the packages in the archive. I agree with this analysis. I think the MAKEFLAGS setting in dpkg-buildpackage should be removed and instead calling dpkg-buildpackage with the -j option should just set the parallel=n flag in DEB_BUILD_OPTIONS. From there, we can leave it to package maintainers to decide how to implement this. I filed bug #458589 against dpkg-dev about this. I think that both following solutions are fine: - dpkg-buildpackage must not consider DEB_BUILD_OPTIONS=parallel=n as the same thing as calling dpkg-buildpackage -j n. - dpkg-buildpackage -j must not set MAKEFLAGS. But if this is choosen, dpkg-buildpackage -j will simply become another way of doing 'export DEB_BUILD_OPTIONS=parallel=n'. Is it really worth keeping in this case? If someone *really* wants to try forcing make to do a parallel build, they can always set MAKEFLAGS themselves directly. True ; maybe dpkg-buildpackage -j is completely useless, actually. -- | Lucas Nussbaum | [EMAIL PROTECTED] http://www.lucas-nussbaum.net/ | | jabber: [EMAIL PROTECTED] GPG: 1024D/023B3F4F | -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Tuesday 01 January 2008 07:36:34 pm Aurelien Jarno wrote: Did you compare the contents of the package with and without -j? I am almost sure some of the successfully built packages are actually not correctly built and some files are missing. For example I remember having seen some python packages building one flavour after the other in different targets. I imagine this could result in one of the flavour being overwritten by the other, and thus not present in the resulting .deb file. I compared the control files and file lists from the packages and marked the build broken if it found significant differences. So that would have caught the case above. But not a case where, for example, the compiler runs were started before patches were completely applied, probably resulting in a mix of patched and non-patched code. However, due to the issues raised in the previous thread about 100% bit-by-bit reproducibility of package builds, I don't see any good way to detect that case automatically. -- Daniel Schepler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Monday 31 December 2007 04:07:15 pm Robert Millan wrote: On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Why -j3 ? It's already quite an effort to aim for -j2. Shouldn't we try to fix those first? That's just my habit, the idea being to increase the probability that both CPUs have work to do while I/O is going on. And besides, I don't see that much of a qualitative difference between -j2 and -j3, for reasons already pointed out by others in the thread. -- Daniel Schepler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Hi, On Wed, 2008-01-02 at 14:23:01 +0100, Lucas Nussbaum wrote: On 01/01/08 at 18:23 -0800, Russ Allbery wrote: Aurelien Jarno [EMAIL PROTECTED] writes: On the other hand, DEB_BUILD_OPTIONS=parallel=n was ignored by packages that have not been validated by the maintainers, and used by packages that have been tested by the maintainer. Also it was possible to use only on some parts of the build. For example the make install part is sometimes problematic, and mainly depends on I/O throughput of the machine, and not CPU. This make some problems difficult to detect, as they happen only in rare case on some machines. Well on my part I think I've made all my packages to properly build in parallel but didn't add the explicit parsing on the rules file. Unfortunately dpkg-buildpackage now looks for DEB_BUILD_OPTIONS and call debian/rules with -j and the value it founds in this environment variable. That makes DEB_BUILD_OPTIONS=parallel=n useless, as it has the same effect as dpkg-buildpackage -j. No, there's build systems out there that are not make based, those would still need to parse DEB_BUILD_OPTIONS and activate parallelism if supported by those systems. I personally think that supporting DEB_BUILD_OPTIONS=parallel=n on some packages is a first goal to achieve. Supporting this option in the 100 packages which take the longest to build would probably be a huge improvement in build time of the whole archive, while being easier than fixing all the packages in the archive. I agree with this analysis. I think the MAKEFLAGS setting in dpkg-buildpackage should be removed and instead calling dpkg-buildpackage with the -j option should just set the parallel=n flag in DEB_BUILD_OPTIONS. From there, we can leave it to package maintainers to decide how to implement this. I filed bug #458589 against dpkg-dev about this. I think that both following solutions are fine: - dpkg-buildpackage must not consider DEB_BUILD_OPTIONS=parallel=n as the same thing as calling dpkg-buildpackage -j n. Ok, I've just commited a change for dpkg-buildpackage not to automatically behave as if -j had been passed as argument if DEB_BUILD_OPTIONS contains parallel=n. - dpkg-buildpackage -j must not set MAKEFLAGS. But if this is choosen, dpkg-buildpackage -j will simply become another way of doing 'export DEB_BUILD_OPTIONS=parallel=n'. Is it really worth keeping in this case? If someone *really* wants to try forcing make to do a parallel build, they can always set MAKEFLAGS themselves directly. True ; maybe dpkg-buildpackage -j is completely useless, actually. It's as useless as dpkg-buildpackage itself, you can also directly call debian/rules ... regards, guillem -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 1014 succeeded, but with jobserver warnings No that DEB_BUILD_OPTIONS=parallel=n has been superseded by dpkg-buildpackage -j, I am trying to fix the glibc package so that it can be built in parallel. Unfortunately I failed to get it built on parallel (whereas with DEB_BUILD_OPTIONS it was possible), I am getting: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule. This basically means only the debian/rules file is run in parallel, the real build is not, whereas with DEB_BUILD_OPTIONS it was the contrary. You can imagine the difference in term of build time. This is due to the following command: $(call logme, -a $(log_build), $(MAKE) -C $(DEB_BUILDDIR)) where logme is defined as: define logme (exec 31; exit `( ( ( $(2) ) 21 3-; echo $$? 4) | tee $(1) 3) 41`) endef Any idea how to get this working for parallel builds with dpkg-buildpackage -j? I fail to see a solution. -- .''`. Aurelien Jarno | GPG: 1024D/F1BCDB73 : :' : Debian developer | Electrical Engineer `. `' [EMAIL PROTECTED] | [EMAIL PROTECTED] `-people.debian.org/~aurel32 | www.aurel32.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 204 built BROKEN packages 1408 FAILED 230 FAILED, even with regular build 8986 succeeded 1014 succeeded, but with jobserver warnings These are not encouraging statistics, especially considering the fact that there are undoubtedly many false negatives, so I'll hold off on submitting bug reports for now. Did you compare the contents of the package with and without -j? I am almost sure some of the successfully built packages are actually not correctly built and some files are missing. For example I remember having seen some python packages building one flavour after the other in different targets. I imagine this could result in one of the flavour being overwritten by the other, and thus not present in the resulting .deb file. -- .''`. Aurelien Jarno | GPG: 1024D/F1BCDB73 : :' : Debian developer | Electrical Engineer `. `' [EMAIL PROTECTED] | [EMAIL PROTECTED] `-people.debian.org/~aurel32 | www.aurel32.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Aurelien Jarno [EMAIL PROTECTED] writes: On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 1014 succeeded, but with jobserver warnings No that DEB_BUILD_OPTIONS=parallel=n has been superseded by dpkg-buildpackage -j, I am trying to fix the glibc package so that it can be built in parallel. The effect of dpkg-buildpackage -j is to set DEB_BUILD_OPTIONS. The package should continue to look at DEB_BUILD_OPTIONS to determine whether parallel builds should be done. See http://bugs.debian.org/209008 which is awaiting final reviews and seconds and should be in the next version of Policy, and the dpkg-buildpackage man page under the -j option. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Russ Allbery a écrit : Aurelien Jarno [EMAIL PROTECTED] writes: On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 1014 succeeded, but with jobserver warnings No that DEB_BUILD_OPTIONS=parallel=n has been superseded by dpkg-buildpackage -j, I am trying to fix the glibc package so that it can be built in parallel. The effect of dpkg-buildpackage -j is to set DEB_BUILD_OPTIONS. The package should continue to look at DEB_BUILD_OPTIONS to determine whether parallel builds should be done. As far as I understand, the main effect is to call debian/rules with -j, but it also set DEB_BUILD_OPTIONS (but I fail to see why). Anyway, if the package continue to look to DEB_BUILD_OPTIONS, as the make children could not communicate with the jobserver, some strange results may appear. On architectures that have more than one flavour, that just mean the number of jobs is actually (value of -j times number of flavours). -- .''`. Aurelien Jarno | GPG: 1024D/F1BCDB73 : :' : Debian developer | Electrical Engineer `. `' [EMAIL PROTECTED] | [EMAIL PROTECTED] `-people.debian.org/~aurel32 | www.aurel32.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Aurelien Jarno [EMAIL PROTECTED] writes: Russ Allbery a écrit : The effect of dpkg-buildpackage -j is to set DEB_BUILD_OPTIONS. The package should continue to look at DEB_BUILD_OPTIONS to determine whether parallel builds should be done. As far as I understand, the main effect is to call debian/rules with -j, but it also set DEB_BUILD_OPTIONS (but I fail to see why). It also sets MAKEFLAGS. I'm not sure that part is a good idea, since packages that don't anticipate this can't reject the provided MAKEFLAGS in the case that the upstream build system doesn't support parallel builds. Basically, this means that using dpkg-buildpackage -j isn't generally safe. It can only be used with packages that you already know support parallel builds. Anyway, if the package continue to look to DEB_BUILD_OPTIONS, as the make children could not communicate with the jobserver, some strange results may appear. On architectures that have more than one flavour, that just mean the number of jobs is actually (value of -j times number of flavours). Ah, interesting. This is a subtlety of how make implements parallelism that I wasn't aware of. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/
Re: Parallel build results
Aurelien Jarno [EMAIL PROTECTED] writes: On the other hand, DEB_BUILD_OPTIONS=parallel=n was ignored by packages that have not been validated by the maintainers, and used by packages that have been tested by the maintainer. Also it was possible to use only on some parts of the build. For example the make install part is sometimes problematic, and mainly depends on I/O throughput of the machine, and not CPU. This make some problems difficult to detect, as they happen only in rare case on some machines. Unfortunately dpkg-buildpackage now looks for DEB_BUILD_OPTIONS and call debian/rules with -j and the value it founds in this environment variable. That makes DEB_BUILD_OPTIONS=parallel=n useless, as it has the same effect as dpkg-buildpackage -j. I personally think that supporting DEB_BUILD_OPTIONS=parallel=n on some packages is a first goal to achieve. Supporting this option in the 100 packages which take the longest to build would probably be a huge improvement in build time of the whole archive, while being easier than fixing all the packages in the archive. I agree with this analysis. I think the MAKEFLAGS setting in dpkg-buildpackage should be removed and instead calling dpkg-buildpackage with the -j option should just set the parallel=n flag in DEB_BUILD_OPTIONS. From there, we can leave it to package maintainers to decide how to implement this. If someone *really* wants to try forcing make to do a parallel build, they can always set MAKEFLAGS themselves directly. The only drawback that I can see is that, without special intervention by the package, this doesn't run debian/rules itself in parallel. But I think the latter generates a lot of bugs and doesn't save a lot of time in practice, so I'm okay with that. The most difficult part probably being changing the policy (sigh) as some of them already support that, but through a specific environment variable. My intention is to have parallel=n be specified in Policy 3.7.4. If you have a moment to review the wording in Bug#209008, feedback and seconds are certainly welcome. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Why -j3 ? It's already quite an effort to aim for -j2. Shouldn't we try to fix those first? -- Robert Millan GPLv2 I know my rights; I want my phone call! DRM What use is a phone call, if you are unable to speak? (as seen on /.) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Mon, Dec 31, 2007 at 10:07:15PM +0100, Robert Millan wrote: On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Why -j3 ? It's already quite an effort to aim for -j2. Shouldn't we try to fix those first? Once you go over -j1, nearly all usual race conditions can be potentially triggered. You would need a quite contrived build system to get a three-way race which cannot fail with -j2. And races caused by doing more than one task at once are pretty rare, in most in the cases the failure is due to wrong dependences. The root of the problem is, people are used to a single-tasked make which always builds everything from left to right. Things won't break until -j or -k is used. What about changing make so it builds prerequisites in a random order, or at least right-to-left which is typically the worst case? This would help spot dependency problems other than true races. (an example which will not be caught by hypothethical make --order=random but will likely break for -j higher than 1: all: foo bar foo: foo.c gcc src.c mv a.out $(target)/foo bar: bar.c gcc bar.c mv a.out $(target)/bar Another example could be building two flavours of the same package.) True race conditions are non-deterministic, so even repeated test builds are likely to find most of them, but wrong dependencies are a lot easier to catch. -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Sun, Dec 02, 2007 at 11:13:53PM +, Ben Hutchings wrote: It appears that ExtUtils::MakeMaker, a standard Perl module commonly used to generate Makefiles for Perl modules, emits the rule: install :: all pure_install doc_install This appears to account for the failure of some of my Perl packages, and probably many others. This should be fixed in MakeMaker, not in the packages that use it. About the only option I can see is to use something like: install :: all $(MAKE) pure_install doc_install which I'm not terribly keen on. I'm open to better suggestions. --bod -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Sun, Dec 30, 2007 at 11:43:00PM +1100, Brendan O'Dea wrote: On Sun, Dec 02, 2007 at 11:13:53PM +, Ben Hutchings wrote: It appears that ExtUtils::MakeMaker, a standard Perl module commonly used to generate Makefiles for Perl modules, emits the rule: install :: all pure_install doc_install install :: all $(MAKE) pure_install doc_install Both are buggy, because neither makes sure any of the _install targets have their files built. This means, trying to run X_install before or instead of full install can break; that can happen if someone works on the package or if there are separate binary-arch and binary-indep targets. What about this: install :: pure_install doc_install pure_install: pure_build doc_install: doc_build (either pure_build or doc_build can be all if there's no easy way to separate the main and doc builds) -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Brendan O'Dea [EMAIL PROTECTED] writes: On Sun, Dec 02, 2007 at 11:13:53PM +, Ben Hutchings wrote: It appears that ExtUtils::MakeMaker, a standard Perl module commonly used to generate Makefiles for Perl modules, emits the rule: install :: all pure_install doc_install This appears to account for the failure of some of my Perl packages, and probably many others. This should be fixed in MakeMaker, not in the packages that use it. About the only option I can see is to use something like: install :: all $(MAKE) pure_install doc_install which I'm not terribly keen on. I'm open to better suggestions. As long as you also have: pure_install:: all doc_install:: all won't make figure it out even with the first form? It may try to run all three at the same time, but it will then realize that the second two depend on all anyway, so all should be forced to go first and complete before either of the other targets run. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Dec 02, Daniel Schepler [EMAIL PROTECTED] wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Can you explain the meaning of this failure and how it should be fixed? make[2]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule. -- ciao, Marco signature.asc Description: Digital signature
Re: Parallel build results
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2 Dec 2007, at 2:21 am, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 204 built BROKEN packages 1408 FAILED 230 FAILED, even with regular build 8986 succeeded 1014 succeeded, but with jobserver warnings These are not encouraging statistics, especially considering the fact that there are undoubtedly many false negatives, so I'll hold off on submitting bug reports for now. Well, am-utils lists as building broken packages, but when I looked at the log, it was just that the parallel build had produced the multiple binary packages in a different order from the serial build. At least, that's my interpretation of the following from debdiff: Files moved or copied from at least TWO packages or to at least TWO packages - - -rw-r--r-- root/root DEBIAN/control From packages: am-utils-doc, am-utils, libamu4, libamu-dev To packages: am-utils-doc, libamu4, libamu-dev, am-utils - -rw-r--r-- root/root DEBIAN/md5sums From packages: am-utils-doc, am-utils, libamu4, libamu-dev To packages: am-utils-doc, libamu4, libamu-dev, am-utils - -rwxr-xr-x root/root DEBIAN/postinst From packages: am-utils-doc, am-utils, libamu4 To packages: am-utils-doc, libamu4, am-utils - -rwxr-xr-x root/root DEBIAN/postrm From packages: am-utils, libamu4 To packages: libamu4, am-utils Or does that mean something more basic that's wrong with the packages? Tim -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQEVAwUBR1PWWBypeFo2odvPAQLydAgArlhmYhkRt8s4FDKg8i5QYa6DlgrKJCz3 fKifS0P0w8yR3E7NLkpS5co+lFZ2Htb2xs4MMD7pWmRBXCxIsh6+6bIPnUfc/mp1 C3VfdfPwPAoER/A6EryfoBA2Y+tof7ewVjEDFaViIlWxsDhaRQgLAck/xx1m51Yd pbnscakO7P8BK9yzirhHHJ6VmlXgU4HkES2xkzZzAumxnmrklkzRWIg52iCBhi8h kGwTmBVAMeRlXP1Iu6sGZFGOIh72wjO2EQ7M5GsbVPYkwbPN/FoyrceSe+2hz+CO 2sA/w9f3wRhIH1WfgiLZTj4d9YGM95iNvkqhl4Wlc6A7IEkgSMvdHg== =QcpJ -END PGP SIGNATURE- -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
* Marco d'Itri [EMAIL PROTECTED] [071203 09:53]: Can you explain the meaning of this failure and how it should be fixed? make[2]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule. That usually means a new make is started without the calling make realizing it is starting a make. In most of the cases that means that some makefile (like debian/rules) is calling make instead of $(MAKE). There are also some special other cases (like when $(MAKE) is used, but not directly but indirectly referenced, then this might be fixed by prefixing the command with a +, but beware of the other meanings this has.), which normally dictate a closer look, as when something is doing such absurdities, there might be some reason to it. (well, or not) Hochachtungsvoll, Bernhard R. Link -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
* Patrick Schoenfeld [EMAIL PROTECTED] [071202 20:43]:
Relevant parts for detox are:
/usr/bin/install -c -d /tmp/buildd/detox-1.1.1/debian/tmp/etc
/usr/bin/install: cannot create regular file
`/tmp/buildd/detox-1.1.1/debian/tmp/etc/detoxrc.sample': No such file or
directory
So I assume the first install command for debian/tmp/etc was not
successful? I'm not very experienced with parallel builds and I cannot
reproduce this on my single-core-systems. So: Is there some
documentation or hints on how to make package building be parallel-safe?
The problem is in upstream's Makefile.in:
install: install-base install-safe-config install-sample-config
install-base: detox
...snip
${INSTALL} -d ${DESTDIR}${sysconfdir}
...snip
install-safe-config:
@if [ ! -f ${DESTDIR}${sysconfdir}/detoxrc ]; then \
${INSTALL} detoxrc ${DESTDIR}${sysconfdir}; \
..snip...
As you can see, install says it needs install-base and install-sample-config
but install-sample-config does not say it wants install-base to be run before
it is run, so nothing enforces sysconfigdir is actually created.
Only when install-safe-config is run while install-base had not yet had enough
time to complete directory creating, this produces an error.
The fix for this problem is just to tell install-safe-config (and the other
install-* stuff needing a directory) to either create the directory first or
depend on the rule that creates it.
Catching those race conditions is hard, even a -jN run might just not trigger it
if the directory creating is fast enough. Perhaps someone would like to write a
patched make that does reorderable target in reversed order and try to build the
whole archive with it (and -j1), to catch those problems more reliable.
Hochachtungsvoll,
Bernhard R. Link
--
Never contain programs so few bugs, as when no debugging tools are available!
Niklaus Wirth
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Daniel Schepler [EMAIL PROTECTED] writes: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Which said: shishi: succeeded, but with jobserver warnings and in the logs: ... make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule. ... I believe I've fixed this in the debian package CVS. The problem was use of 'make' rather than '$(MAKE)' in rules. Thanks, Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Monday 03 December 2007 05:11:35 am Tim Cutts wrote:
Well, am-utils lists as building broken packages, but when I looked at
the log, it was just that the parallel build had produced the multiple
binary packages in a different order from the serial build. At least,
that's my interpretation of the following from debdiff:
Files moved or copied from at least TWO packages or to at least TWO
packages
---
- -rw-r--r-- root/root DEBIAN/control
From packages: am-utils-doc, am-utils, libamu4, libamu-dev
To packages: am-utils-doc, libamu4, libamu-dev, am-utils
-rw-r--r-- root/root DEBIAN/md5sums
From packages: am-utils-doc, am-utils, libamu4, libamu-dev
To packages: am-utils-doc, libamu4, libamu-dev, am-utils
-rwxr-xr-x root/root DEBIAN/postinst
From packages: am-utils-doc, am-utils, libamu4
To packages: am-utils-doc, libamu4, am-utils
-rwxr-xr-x root/root DEBIAN/postrm
From packages: am-utils, libamu4
To packages: libamu4, am-utils
Or does that mean something more basic that's wrong with the packages?
The relevant difference is actually:
Control files of package am-utils: lines which differ (wdiff format)
Depends: libamu4 (= 6.1.5-7), portmap, {+libamu4,+} libc6 (= 2.6.1-1),
libgdbm3, libhesiod0, libldap2 (= 2.1.17-1), libwrap0, perl, debconf (=
1.2.0), ucf, debianutils (= 1.6)
So the Depends line got changed in the parallel build.
--
Daniel Schepler
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 3 Dec 2007, at 12:43 pm, Daniel Schepler wrote:
Control files of package am-utils: lines which differ (wdiff format)
Depends: libamu4 (= 6.1.5-7), portmap, {+libamu4,+} libc6 (=
2.6.1-1),
libgdbm3, libhesiod0, libldap2 (= 2.1.17-1), libwrap0, perl,
debconf (=
1.2.0), ucf, debianutils (= 1.6)
So the Depends line got changed in the parallel build.
Sure, yes, I realise that one. Don't know why it's happening though.
It may change though as I make changes do deal with the shlibdeps
stuff that's also going on at the moment, so I'm going to deal with
that first, and see whether this problem persists afterwards.
Tim
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (Darwin)
iQEVAwUBR1QEaBypeFo2odvPAQJ1WQgAizNT66pUw/GB72oB8zc1+RVzYrseaRFo
pqZIerbj1mr3G4rKur+wuYAfVEJDR9zgSWEmebwGuklHsa+eztWcYAgWfD8OBvmD
1vSyhIhfgN0tARQQORX7Ba9G3lM7MnLs8/Rvn3APm0/vacJDDKCucEwdsTqDGpDH
yBh6/9yxF8lLiaumiomGO+vFZTkREXk2CtxNI8nh44ijgYkSdj1QZj9lZ7cuCC3W
8FnHeLlit9l2K3EVH/0dE/3YcBt2uf3QRfURVKdtPrwpNMuGh0FkuHdScm+BPh/L
oDHtYuSxkElPam3q25a8wno15wVt5qaz+nJ43TtvsC6WGJNxIqNorw==
=I2Ko
-END PGP SIGNATURE-
--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Hi Bernhard, On Mon, Dec 03, 2007 at 12:21:45PM +0100, Bernhard R. Link wrote: The problem is in upstream's Makefile.in: thanks for the good explanation. Also to Daniel whose hint already pointed me in the right direction. Its already fixed in latest upload. Thanks, Best Regards Patrick -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Hello, Marco d'Itri [EMAIL PROTECTED] wrote: On Dec 02, Daniel Schepler [EMAIL PROTECTED] wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Can you explain the meaning of this failure and how it should be fixed? make[2]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule. Quoting from http://lists.samba.org/archive/distcc/2004q1/002160.html There are (at least?) three reasons why this might be so: 1) The parent make didn't realize that the child process it was invoking was actually a make program. In this case, it won't allow the child to connect to the jobserver. This is typically because the command to invoke the child uses make or similar rather than the correct $(MAKE). See the docs on the + token to understand the algorithm GNU make uses to determine if the child is a sub-make or not. Bye, Jörg. -- Wenn du nur einen Hammer hast, sieht jedes Problem aus wie ein Nagel. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: [...] BTW, Daniel, it would be nice if you provided a detailed description of the exact build environment you use, i.e., kernel version, libc version, gcc, ... Thanks, Michael pgpkghRLTogCM.pgp Description: PGP signature
Re: Parallel build results
I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: [...] Wouldn't it have been wise to use lenny (or even etch) instead of sid? Apparently your statistics include all the failures due to normal bugs entirely unrelated to the parallel build process. Best, Michael pgpnf9JaJMYby.pgp Description: PGP signature
Re: Parallel build results
On 01/12/07 at 21:21 -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 204 built BROKEN packages 1408 FAILED 230 FAILED, even with regular build 8986 succeeded 1014 succeeded, but with jobserver warnings These are not encouraging statistics, especially considering the fact that there are undoubtedly many false negatives, so I'll hold off on submitting bug reports for now. According to those numbers, I don't really see how it could become a viable solution to require that packages support building using dpkg-buildpackage -j. This option is nice for maintainers, to build their own packages, but it seems better to push #209008 instead, so packages that take a long time to build have a common interface to specify that parallel builds. -- | Lucas Nussbaum | [EMAIL PROTECTED] http://www.lucas-nussbaum.net/ | | jabber: [EMAIL PROTECTED] GPG: 1024D/023B3F4F | -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
* Daniel Schepler [EMAIL PROTECTED] [071202 03:39]: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . |xbuffy: succeeded, but with jobserver warnings thanks for catching this, should be fixed with the upload of today. These are not encouraging statistics, especially considering the fact that there are undoubtedly many false negatives, so I'll hold off on submitting bug reports for now. Glancing over some of the failed build logs, suprisingly many of them seem even be caused by broken debian/rules files. I'm suprised stuff like | build-arch: unpack-stamp patch-stamp configure build-arch-stamp | build-arch-stamp: |...some commands... actually works with -j1. (Well, it will only work when special targets are specified). I think going after such errors has much more advantages than just allowing hosts with multiple processors to build faster, such stuff should never be allowed to sneak in. Hochachtungsvoll, Bernhard R. Link -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
Hi, On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote: i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . I am not sure how to handle these problems with my packages. Currently there are two packages failing. detox and dnsproxy. dnsproxy should be simply (moving config.status depend to build-stamp instead of build), but with detox I don't really know what to do. Relevant parts for detox are: /usr/bin/install -c -d /tmp/buildd/detox-1.1.1/debian/tmp/etc /usr/bin/install: cannot create regular file `/tmp/buildd/detox-1.1.1/debian/tmp/etc/detoxrc.sample': No such file or directory make[1]: *** [install-sample-config] Error 1 make[1]: *** Waiting for unfinished jobs /usr/bin/install: `/tmp/buildd/detox-1.1.1/debian/tmp/etc' exists but is not a directory make[1]: *** [install-base] Error 1 make[1]: Leaving directory `/tmp/buildd/detox-1.1.1' So I assume the first install command for debian/tmp/etc was not successful? I'm not very experienced with parallel builds and I cannot reproduce this on my single-core-systems. So: Is there some documentation or hints on how to make package building be parallel-safe? Regards, Patrick -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Parallel build results
On Sat, 2007-12-01 at 21:21 -0500, Daniel Schepler wrote: I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 204 built BROKEN packages 1408 FAILED 230 FAILED, even with regular build 8986 succeeded 1014 succeeded, but with jobserver warnings These are not encouraging statistics, especially considering the fact that there are undoubtedly many false negatives, so I'll hold off on submitting bug reports for now. It appears that ExtUtils::MakeMaker, a standard Perl module commonly used to generate Makefiles for Perl modules, emits the rule: install :: all pure_install doc_install This appears to account for the failure of some of my Perl packages, and probably many others. This should be fixed in MakeMaker, not in the packages that use it. Ben. -- Ben Hutchings The most exhausting thing in life is being insincere. - Anne Morrow Lindberg signature.asc Description: This is a digitally signed message part
Re: Parallel build results
On Sunday 02 December 2007 02:12:40 pm Patrick Schoenfeld wrote: Relevant parts for detox are: /usr/bin/install -c -d /tmp/buildd/detox-1.1.1/debian/tmp/etc /usr/bin/install: cannot create regular file `/tmp/buildd/detox-1.1.1/debian/tmp/etc/detoxrc.sample': No such file or directory make[1]: *** [install-sample-config] Error 1 make[1]: *** Waiting for unfinished jobs /usr/bin/install: `/tmp/buildd/detox-1.1.1/debian/tmp/etc' exists but is not a directory make[1]: *** [install-base] Error 1 make[1]: Leaving directory `/tmp/buildd/detox-1.1.1' Earlier, the Makefile is trying to execute /usr/bin/install -c detoxrc /tmp/buildd/detox-1.1.1/debian/tmp/etc/detoxrc.sample which failed because debian/tmp/etc was not yet created. So it appears the upstream package's Makefile is not parallel-safe, at least with regards to the install target. -- Daniel Schepler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Parallel build results
I finally got through the test builds of all the source packages in sid for i386 using dpkg-buildpackage -j3 on a dual core machine. The results as before are at http://people.debian.org/~schepler/build-logs/bymaint.html . Some statistics: 204 built BROKEN packages 1408 FAILED 230 FAILED, even with regular build 8986 succeeded 1014 succeeded, but with jobserver warnings These are not encouraging statistics, especially considering the fact that there are undoubtedly many false negatives, so I'll hold off on submitting bug reports for now. -- Daniel Schepler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
