Hi guys,
the following patch is a first implementation for you to review and comment on.
It's been tested successfully on Linux/x86, Linux/x86_64, and Darwin/x86. The
following changes were made:
1) Nix-env provides the environment variable $NIX_BUILD_CORES. The default
value is 1, which is equivalent to building with parallelism disabled.
2) The generic builder checks whether $enableParallelBuilding is set. If it
is, "-j{NIX_BUILD_CORES} -l{NIX_BUILD_CORES}" is added to $makeFlags.
3) The packages gcc, glibc, gmp, mpfr, coreutils, perl, git, and qt4 now
define $enableParallelBuilding.
Let me know what you think, please.
Take care,
Peter
>From b392671f88d56ea6f43b9c2fb7089436ad9926b9 Mon Sep 17 00:00:00 2001
From: Peter Simons <[email protected]>
Date: Tue, 22 Jun 2010 13:09:33 +0200
Subject: [PATCH 1/3] pkgs/tools/package-management/nix/unstable.nix: added
support for impure NIX_BUILD_CORES variable
This patch adds the configuration file variable "build-cores", and the
command line argument "--cores" to nix-env. The variables specifies the
number of CPU cores to utilize for parellel building within a job, i.e.
by passing an appropriate "-j" argument to GNU Make. The default value
is 1, which means that paralllel building is *disabled*. If the number
of build cores is specified as 0, then that value is normalized to 1.
The value of this setting is available to builders in the environment
variable $NIX_BUILD_CORES. The contents of that variable does *not*
influence the hash that goes into the $out store path, i.e. the number
of build cores to be utilized can be changed at will without requiring
any re-builds.
---
...rt-for-passing-an-impure-NIX_BUILD_CORES-.patch | 99 ++++++++++++++++++++
pkgs/tools/package-management/nix/unstable.nix | 1 +
2 files changed, 100 insertions(+), 0 deletions(-)
create mode 100644
pkgs/tools/package-management/nix/0001-Added-support-for-passing-an-impure-NIX_BUILD_CORES-.patch
diff --git
a/pkgs/tools/package-management/nix/0001-Added-support-for-passing-an-impure-NIX_BUILD_CORES-.patch
b/pkgs/tools/package-management/nix/0001-Added-support-for-passing-an-impure-NIX_BUILD_CORES-.patch
new file mode 100644
index 0000000..047001d
--- /dev/null
+++
b/pkgs/tools/package-management/nix/0001-Added-support-for-passing-an-impure-NIX_BUILD_CORES-.patch
@@ -0,0 +1,99 @@
+From 898e90f1eaf531d55bb37b36be500ee366db18b4 Mon Sep 17 00:00:00 2001
+From: Peter Simons <[email protected]>
+Date: Tue, 22 Jun 2010 16:59:03 +0200
+Subject: [PATCH] Added support for passing an (impure) NIX_BUILD_CORES
variable to build expressions.
+
+---
+ nix.conf.example | 10 ++++++++++
+ src/libmain/shared.cc | 3 +++
+ src/libstore/build.cc | 3 +++
+ src/libstore/globals.cc | 1 +
+ src/libstore/globals.hh | 4 ++++
+ 5 files changed, 21 insertions(+), 0 deletions(-)
+
+diff --git a/nix.conf.example b/nix.conf.example
+index e17cf3c..96d47c1 100644
+--- a/nix.conf.example
++++ b/nix.conf.example
+@@ -59,6 +59,16 @@
+ #build-max-jobs = 1
+
+
++### Option `build-cores'
++#
++# This option defines the number of CPU cores to utilize in parallel
++# within a build job, i.e. by passing an appropriate `-jN' flag to
++# GNU make. The default is 1, meaning that parallel building within
++# job is disabled. This setting can be overridden using the
++# `--cores' command line switch.
++#build-cores = 1
++
++
+ ### Option `build-max-silent-time'
+ #
+ # This option defines the maximum number of seconds that a builder can
+diff --git a/src/libmain/shared.cc b/src/libmain/shared.cc
+index 3fbec4b..1491ee9 100644
+--- a/src/libmain/shared.cc
++++ b/src/libmain/shared.cc
+@@ -135,6 +135,7 @@ static void initAndRun(int argc, char * * argv)
+ /* Get some settings from the configuration file. */
+ thisSystem = querySetting("system", SYSTEM);
+ maxBuildJobs = queryIntSetting("build-max-jobs", 1);
++ buildCores = std::max(queryIntSetting("build-cores", 1u), 1u);
+ maxSilentTime = queryIntSetting("build-max-silent-time", 0);
+
+ /* Catch SIGINT. */
+@@ -226,6 +227,8 @@ static void initAndRun(int argc, char * * argv)
+ tryFallback = true;
+ else if (arg == "--max-jobs" || arg == "-j")
+ maxBuildJobs = getIntArg<unsigned int>(arg, i, args.end());
++ else if (arg == "--cores")
++ buildCores = getIntArg<unsigned int>(arg, i, args.end());
+ else if (arg == "--readonly-mode")
+ readOnlyMode = true;
+ else if (arg == "--max-silent-time")
+diff --git a/src/libstore/build.cc b/src/libstore/build.cc
+index f901c1f..a53c23b 100644
+--- a/src/libstore/build.cc
++++ b/src/libstore/build.cc
+@@ -1411,6 +1411,9 @@ void DerivationGoal::startBuilder()
+ in the store or in the build directory). */
+ env["NIX_STORE"] = nixStore;
+
++ /* The maximum number of cores to utilize for parallel building. */
++ env["NIX_BUILD_CORES"] = (format("%d") % buildCores).str();
++
+ /* Add all bindings specified in the derivation. */
+ foreach (StringPairs::iterator, i, drv.env)
+ env[i->first] = i->second;
+diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
+index cc0e44e..75d2f69 100644
+--- a/src/libstore/globals.cc
++++ b/src/libstore/globals.cc
+@@ -22,6 +22,7 @@ bool keepGoing = false;
+ bool tryFallback = false;
+ Verbosity buildVerbosity = lvlInfo;
+ unsigned int maxBuildJobs = 1;
++unsigned int buildCores = 1;
+ bool readOnlyMode = false;
+ string thisSystem = "unset";
+ time_t maxSilentTime = 0;
+diff --git a/src/libstore/globals.hh b/src/libstore/globals.hh
+index d3388e3..0f42533 100644
+--- a/src/libstore/globals.hh
++++ b/src/libstore/globals.hh
+@@ -55,6 +55,10 @@ extern Verbosity buildVerbosity;
+ /* Maximum number of parallel build jobs. 0 means unlimited. */
+ extern unsigned int maxBuildJobs;
+
++/* Number of CPU cores to utilize in parallel within a build, i.e. by passing
++ this number to Make via '-j'. 0 is interpreted like 1. */
++extern unsigned int buildCores;
++
+ /* Read-only mode. Don't copy stuff to the store, don't change the
+ database. */
+ extern bool readOnlyMode;
+--
+1.7.1
+
diff --git a/pkgs/tools/package-management/nix/unstable.nix
b/pkgs/tools/package-management/nix/unstable.nix
index f45172b..c7f690c 100644
--- a/pkgs/tools/package-management/nix/unstable.nix
+++ b/pkgs/tools/package-management/nix/unstable.nix
@@ -13,6 +13,7 @@ stdenv.mkDerivation rec {
buildNativeInputs = [ perl ];
buildInputs = [ curl openssl ];
+ patches = [
./0001-Added-support-for-passing-an-impure-NIX_BUILD_CORES-.patch ];
configureFlags =
''
--
1.7.1
>From ffc48b7767b3b923e647cbfa2250b1f91ae32f15 Mon Sep 17 00:00:00 2001
From: Peter Simons <[email protected]>
Date: Tue, 22 Jun 2010 13:12:27 +0200
Subject: [PATCH 2/3] pkgs/stdenv/generic/setup.sh: added support for
enableParallelBuilding variable
If a build expressions has set "enableParallelBuilding = true", then the
generic builder may utilize more than one CPU core to build that particular
expression. This feature works out of the box for GNU Make. Expressions that
use other build drivers like Boost.Jam or SCons have to add appropriate flags
such as "-j${NIX_BUILD_CORES}" to $buildFlags or $buildFlagsArray themselves.
---
pkgs/stdenv/generic/setup.sh | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/pkgs/stdenv/generic/setup.sh b/pkgs/stdenv/generic/setup.sh
index 5e2fc7b..88bfc46 100644
--- a/pkgs/stdenv/generic/setup.sh
+++ b/pkgs/stdenv/generic/setup.sh
@@ -601,6 +601,11 @@ buildPhase() {
return
fi
+ # Enable parallel compilation if the expression allows it.
+ if test -n "$enableParallelBuilding"; then
+ makeFlags="-j${NIX_BUILD_CORES:-1} -l${NIX_BUILD_CORES:-1} $makeFlags"
+ fi
+
echo "make flags: $makeFlags ${makeflagsarr...@]} $buildFlags
${buildflagsarr...@]}"
make ${makefile:+-f $makefile} \
$makeFlags "${makeflagsarr...@]}" \
--
1.7.1
>From 832fe13ca922a950181cb17c5bc790497eead2d1 Mon Sep 17 00:00:00 2001
From: Peter Simons <[email protected]>
Date: Tue, 22 Jun 2010 13:28:20 +0200
Subject: [PATCH 3/3] Enable parallel building of gcc, glibc, gmp, mpfr,
coreutils, perl, git, and qt4.
---
.../git-and-tools/git/default.nix | 2 ++
pkgs/development/compilers/gcc-4.3/default.nix | 2 ++
pkgs/development/compilers/gcc-4.4/default.nix | 2 ++
pkgs/development/compilers/gcc-4.5/default.nix | 2 ++
pkgs/development/compilers/gcc-apple/default.nix | 2 ++
pkgs/development/compilers/gcc-apple64/default.nix | 2 ++
.../development/interpreters/perl-5.10/default.nix | 2 ++
pkgs/development/libraries/glibc-2.11/builder.sh | 2 +-
pkgs/development/libraries/glibc-2.11/builder2.sh | 2 +-
pkgs/development/libraries/glibc-2.11/common.nix | 2 ++
pkgs/development/libraries/glibc-2.5/builder.sh | 2 +-
pkgs/development/libraries/glibc-2.5/default.nix | 2 ++
pkgs/development/libraries/gmp/4.3.1.nix | 2 ++
pkgs/development/libraries/gmp/default.nix | 2 ++
pkgs/development/libraries/mpfr/default.nix | 2 ++
pkgs/development/libraries/qt-4.x/4.5/default.nix | 2 ++
pkgs/development/libraries/qt-4.x/4.6/default.nix | 2 ++
pkgs/development/libraries/qt-4.x/4.7/default.nix | 2 ++
pkgs/tools/misc/coreutils/default.nix | 2 ++
19 files changed, 35 insertions(+), 3 deletions(-)
diff --git a/pkgs/applications/version-management/git-and-tools/git/default.nix
b/pkgs/applications/version-management/git-and-tools/git/default.nix
index b7d3490..3cdf200 100644
--- a/pkgs/applications/version-management/git-and-tools/git/default.nix
+++ b/pkgs/applications/version-management/git-and-tools/git/default.nix
@@ -116,6 +116,8 @@ stdenv.mkDerivation rec {
'';
+ enableParallelBuilding = true;
+
meta = {
license = "GPLv2";
homepage = http://git-scm.com/;
diff --git a/pkgs/development/compilers/gcc-4.3/default.nix
b/pkgs/development/compilers/gcc-4.3/default.nix
index 73caa15..b9d0d9b 100644
--- a/pkgs/development/compilers/gcc-4.3/default.nix
+++ b/pkgs/development/compilers/gcc-4.3/default.nix
@@ -127,6 +127,8 @@ stdenv.mkDerivation ({
passthru = { inherit langC langCC langFortran langVhdl langTreelang
enableMultilib; };
+ enableParallelBuilding = true;
+
meta = {
homepage = "http://gcc.gnu.org/";;
license = "GPL/LGPL";
diff --git a/pkgs/development/compilers/gcc-4.4/default.nix
b/pkgs/development/compilers/gcc-4.4/default.nix
index bdf2e58..e4206ca 100644
--- a/pkgs/development/compilers/gcc-4.4/default.nix
+++ b/pkgs/development/compilers/gcc-4.4/default.nix
@@ -201,6 +201,8 @@ stdenv.mkDerivation ({
passthru = { inherit langC langCC langAda langFortran langTreelang langVhdl
enableMultilib version; };
+ enableParallelBuilding = true;
+
meta = {
homepage = http://gcc.gnu.org/;
license = "GPLv3+"; # runtime support libraries are typically LGPLv3+
diff --git a/pkgs/development/compilers/gcc-4.5/default.nix
b/pkgs/development/compilers/gcc-4.5/default.nix
index 1dcae1f..d7e0925 100644
--- a/pkgs/development/compilers/gcc-4.5/default.nix
+++ b/pkgs/development/compilers/gcc-4.5/default.nix
@@ -277,6 +277,8 @@ stdenv.mkDerivation ({
passthru = { inherit langC langCC langAda langFortran langTreelang langVhdl
enableMultilib version; };
+ enableParallelBuilding = true;
+
meta = {
homepage = http://gcc.gnu.org/;
license = "GPLv3+"; # runtime support libraries are typically LGPLv3+
diff --git a/pkgs/development/compilers/gcc-apple/default.nix
b/pkgs/development/compilers/gcc-apple/default.nix
index 8006bd4..3346a4e 100644
--- a/pkgs/development/compilers/gcc-apple/default.nix
+++ b/pkgs/development/compilers/gcc-apple/default.nix
@@ -21,6 +21,8 @@ stdenv.mkDerivation ({
sha256 =
"a7d8041e50e110f5a503e188a05cb217f0c99c51f248a0a1387cc07a0b6f167f";
}) ;
+ enableParallelBuilding = true;
+
sourceRoot = "gcc_42-5574/";
patches =
[./pass-cxxcpp.patch ./debug_list.patch]
diff --git a/pkgs/development/compilers/gcc-apple64/default.nix
b/pkgs/development/compilers/gcc-apple64/default.nix
index b87cd59..ec16fae 100644
--- a/pkgs/development/compilers/gcc-apple64/default.nix
+++ b/pkgs/development/compilers/gcc-apple64/default.nix
@@ -21,6 +21,8 @@ stdenv.mkDerivation ({
sha256 = "1fy6j41rhxdsm19sib9wygjl5l54g8pm13c6y5x13f40mavw1mma";
}) ;
+ enableParallelBuilding = true;
+
libstdcxx = "libstdcxx-39";
sourceRoot = "gcc-5646/";
patches =
diff --git a/pkgs/development/interpreters/perl-5.10/default.nix
b/pkgs/development/interpreters/perl-5.10/default.nix
index 3116829..76928d1 100644
--- a/pkgs/development/interpreters/perl-5.10/default.nix
+++ b/pkgs/development/interpreters/perl-5.10/default.nix
@@ -39,6 +39,8 @@ stdenv.mkDerivation rec {
dontAddPrefix = true;
+ enableParallelBuilding = true;
+
preConfigure =
''
configureFlags="$configureFlags -Dprefix=$out
-Dman1dir=$out/share/man/man1 -Dman3dir=$out/share/man/man3"
diff --git a/pkgs/development/libraries/glibc-2.11/builder.sh
b/pkgs/development/libraries/glibc-2.11/builder.sh
index f8da3b8..cbdb55b 100644
--- a/pkgs/development/libraries/glibc-2.11/builder.sh
+++ b/pkgs/development/libraries/glibc-2.11/builder.sh
@@ -17,7 +17,7 @@ postConfigure() {
postInstall() {
if test -n "$installLocales"; then
- make localedata/install-locales
+ make -j${NIX_BUILD_CORES:-1} -l${NIX_BUILD_CORES:-1}
localedata/install-locales
fi
test -f $out/etc/ld.so.cache && rm $out/etc/ld.so.cache
diff --git a/pkgs/development/libraries/glibc-2.11/builder2.sh
b/pkgs/development/libraries/glibc-2.11/builder2.sh
index f156e64..7d1a051 100644
--- a/pkgs/development/libraries/glibc-2.11/builder2.sh
+++ b/pkgs/development/libraries/glibc-2.11/builder2.sh
@@ -20,7 +20,7 @@ postConfigure() {
postInstall() {
if test -n "$installLocales"; then
- make localedata/install-locales
+ make -j${NIX_BUILD_CORES:-1} -l${NIX_BUILD_CORES:-1}
localedata/install-locales
fi
test -f $out/etc/ld.so.cache && rm $out/etc/ld.so.cache
diff --git a/pkgs/development/libraries/glibc-2.11/common.nix
b/pkgs/development/libraries/glibc-2.11/common.nix
index b792e84..00d9b99 100644
--- a/pkgs/development/libraries/glibc-2.11/common.nix
+++ b/pkgs/development/libraries/glibc-2.11/common.nix
@@ -30,6 +30,8 @@ stdenv.mkDerivation ({
inherit (stdenv) is64bit;
+ enableParallelBuilding = true;
+
patches =
stdenv.lib.optional (fetchgit == null)
/* Fix for NIXPKGS-79: when doing host name lookups, when
diff --git a/pkgs/development/libraries/glibc-2.5/builder.sh
b/pkgs/development/libraries/glibc-2.5/builder.sh
index 695a3a1..4cd0132 100644
--- a/pkgs/development/libraries/glibc-2.5/builder.sh
+++ b/pkgs/development/libraries/glibc-2.5/builder.sh
@@ -38,7 +38,7 @@ postConfigure() {
postInstall() {
if test -n "$installLocales"; then
- make localedata/install-locales
+ make -j${NIX_BUILD_CORES:-1} -l${NIX_BUILD_CORES:-1}
localedata/install-locales
fi
rm $out/etc/ld.so.cache
(cd $out/include && ln -s $kernelHeaders/include/* .) || exit 1
diff --git a/pkgs/development/libraries/glibc-2.5/default.nix
b/pkgs/development/libraries/glibc-2.5/default.nix
index abfc3d0..83447d8 100644
--- a/pkgs/development/libraries/glibc-2.5/default.nix
+++ b/pkgs/development/libraries/glibc-2.5/default.nix
@@ -30,6 +30,8 @@ stdenv.mkDerivation {
# the symbol __i686.get_pc_thunk.dx to be mangled.
NIX_CFLAGS_COMPILE = "-U__i686";
+ enableParallelBuilding = true;
+
meta = {
homepage = http://www.gnu.org/software/libc/;
description = "The GNU C Library";
diff --git a/pkgs/development/libraries/gmp/4.3.1.nix
b/pkgs/development/libraries/gmp/4.3.1.nix
index 17455b9..5f83d9a 100644
--- a/pkgs/development/libraries/gmp/4.3.1.nix
+++ b/pkgs/development/libraries/gmp/4.3.1.nix
@@ -16,6 +16,8 @@ stdenv.mkDerivation rec {
doCheck = true;
+ enableParallelBuilding = true;
+
meta = {
description = "A free library for arbitrary precision arithmetic,
operating on signed integers, rational numbers, and floating point numbers";
homepage = http://gmplib.org/;
diff --git a/pkgs/development/libraries/gmp/default.nix
b/pkgs/development/libraries/gmp/default.nix
index 0b608fb..ac20bc6 100644
--- a/pkgs/development/libraries/gmp/default.nix
+++ b/pkgs/development/libraries/gmp/default.nix
@@ -16,6 +16,8 @@ stdenv.mkDerivation rec {
doCheck = true;
+ enableParallelBuilding = true;
+
meta = {
description = "A free library for arbitrary precision arithmetic,
operating on signed integers, rational numbers, and floating point numbers";
homepage = http://gmplib.org/;
diff --git a/pkgs/development/libraries/mpfr/default.nix
b/pkgs/development/libraries/mpfr/default.nix
index dea0113..1bfd144 100644
--- a/pkgs/development/libraries/mpfr/default.nix
+++ b/pkgs/development/libraries/mpfr/default.nix
@@ -12,6 +12,8 @@ stdenv.mkDerivation rec {
doCheck = true;
+ enableParallelBuilding = true;
+
meta = {
homepage = http://www.mpfr.org/;
description = "GNU MPFR, a library for multiple-precision floating-point
arithmetic";
diff --git a/pkgs/development/libraries/qt-4.x/4.5/default.nix
b/pkgs/development/libraries/qt-4.x/4.5/default.nix
index 35960c4..3b29043 100644
--- a/pkgs/development/libraries/qt-4.x/4.5/default.nix
+++ b/pkgs/development/libraries/qt-4.x/4.5/default.nix
@@ -74,6 +74,8 @@ stdenv.mkDerivation {
${if keepDocumentation == false then "rm -rf $out/doc" else ""}
'';
+ enableParallelBuilding = true;
+
meta = {
homepage =
http://www.qtsoftware.com/downloads/opensource/appdev/linux-x11-cpp;
description = "A cross-platform application framework for C++";
diff --git a/pkgs/development/libraries/qt-4.x/4.6/default.nix
b/pkgs/development/libraries/qt-4.x/4.6/default.nix
index 9440765..3253e58 100644
--- a/pkgs/development/libraries/qt-4.x/4.6/default.nix
+++ b/pkgs/development/libraries/qt-4.x/4.6/default.nix
@@ -89,6 +89,8 @@ stdenv.mkDerivation rec {
postInstall = if useDocs then "rm -rf $out/share/doc/${name}/{html,src}"
else "";
+ enableParallelBuilding = true;
+
meta = {
homepage = http://qt.nokia.com/products;
description = "A cross-platform application framework for C++";
diff --git a/pkgs/development/libraries/qt-4.x/4.7/default.nix
b/pkgs/development/libraries/qt-4.x/4.7/default.nix
index fbcd8b2..70dd1de 100644
--- a/pkgs/development/libraries/qt-4.x/4.7/default.nix
+++ b/pkgs/development/libraries/qt-4.x/4.7/default.nix
@@ -96,6 +96,8 @@ stdenv.mkDerivation rec {
postInstall = if useDocs then "rm -rf $out/share/doc/${name}/{html,src}"
else "";
+ enableParallelBuilding = true;
+
meta = {
homepage = http://qt.nokia.com/products;
description = "A cross-platform application framework for C++";
diff --git a/pkgs/tools/misc/coreutils/default.nix
b/pkgs/tools/misc/coreutils/default.nix
index f4cf9fe..8052ce9 100644
--- a/pkgs/tools/misc/coreutils/default.nix
+++ b/pkgs/tools/misc/coreutils/default.nix
@@ -24,6 +24,8 @@ stdenv.mkDerivation (rec {
# and {Open,Free}BSD.
doCheck = (stdenv ? glibc) && (cross == null);
+ enableParallelBuilding = true;
+
meta = {
homepage = http://www.gnu.org/software/coreutils/;
description = "The basic file, shell and text manipulation utilities of
the GNU operating system";
--
1.7.1
_______________________________________________
nix-dev mailing list
[email protected]
https://mail.cs.uu.nl/mailman/listinfo/nix-dev
