For Your Consideration
======================
Hello,
this is an implementation of a new option for Make. It controls the
number of parallel jobs in regards to the amount of memory in use on a
host. It allows one to use Make's parallel job execution on hosts with
multiple CPU cores, but where memory is limited. It works similar to
the -l, --load-average option and has to be used together with the -j,
--jobs option.
For example, assuming a host with 2GB of memory, the following options
are equivalent:
-u25%, -u0.25, -u512m, or -u0.5g
In this example will Make stop creating new jobs when a host's memory
use reaches 512MB (-j1), and will resume when it drops back below the
threshold (-jN). A memory threshold can be specified as either an
absolute value or as a percentage of the host's total memory.
When the option is used in combination with -l, --load-average then
Make stops creating new jobs when either the load threshold has been
reached, or when the memory threshold has been reached. In addition
will Make use the load and memory thresholds as a hint for a job's
average memory use and scales the number of jobs (from -j1 to -jN).
This reduces load and memory spikes near the thresholds and stabilizes
the memory usage of a Make run.
A value of 0 turns it off (-u0) and sets it to unlimited, which is
also the default.
The attached patch is against release 4.3 of Make. It implements the
-u option for Linux using /proc/meminfo, and it enables the use of
/proc/loadavg for the -l option.
How well does it work?
======================
To show its effectiveness in practise did I create graphs for system
loads (upper graphs) and memory usages (lower graphs) of different
Make runs. In these graphs is an average shown (thicker line) together
with the raw values (thinner lines, spikes). Because the memory use
can vary between runs did I animate the graphs by using 5 Make runs.
The "std-" labels in the legends mean that the standard Make 4.3
release was used. The "new-" labels mean the patched version was used.
The following is a Linux kernel build on a 16-core CPU with 32GB of
memory. The first graphs (black) show standard runs with only the -j
option. The second graphs (red) show standard runs with -j and -l. The
third graphs (green) show runs with -j and -l, using /proc/loadavg
under Linux. The forth graphs (blue) show runs with -j, -l and -u
combined, using /proc/loadavg and /proc/meminfo. The additional graphs
show variations of the arguments to the -u option:
https://zippyimage.com/images/2022/02/12/183964b428e4a0f0530c647c923924b9.gif
The same kernel build on a 4-core SoC with 512MB:
https://zippyimage.com/images/2022/02/12/aa05eb906c4bb1f6eb969b5bc29b7731.gif
To test the limits did I use a Makefile from image processing. It
consists of 3 rules and processes 30 files, leaving little room for
Make to control its jobs. The jobs themselves use multiple threads,
which inflate the system load, and use a fair amount of memory:
https://zippyimage.com/images/2022/02/12/189b22aa75adc0a6b7a34b6b9cceabee.gif
A more practical test is a build of LLVM/Clang-13 with itself, using
link time optimization (LTO). The Makefiles are CMake-generated. Here
a link job with LTO can use up to 24GB of memory. The host itself has
got 32GB of RAM plus 16GB of ZRAM for swap, making it near impossible
to build with a fixed number of parallel jobs. A "make -j8" will
consistently run out of memory and out of swap space (red, failed). A
"make -j8 -u30%" will build it 5 times without failing. Other values
for -u work, too, however only builds, which succeeded at least 5
times and not failed once, are shown in the graphs. The combination of
-l with -u together works better and allows to build with "-j16 -l16
-u90%" in under an hour:
https://zippyimage.com/images/2022/02/12/9837a5551cc9350c17806c7b57efb991.gif
The last test is a LLVM/Clang-13 LTO build with "-j16 -l16 -u90%" and
CCACHE enabled. It speeds up the build time and brings the LTO link
jobs closer together, causing them to further overlap. It builds 5
times without failing:
https://zippyimage.com/images/2022/02/12/a55d2c3352a134edcb6af3feed8c2ff6.gif
The patch is provided "as is". I hope you enjoy it and find it useful.
Regards,
Sven
diff -r -u make-4.3/src/job.c make-4.3.1/src/job.c
--- make-4.3/src/job.c 2020-01-19 20:32:59.000000000 +0000
+++ make-4.3.1/src/job.c 2022-02-11 14:09:08.074210132 +0000
@@ -221,6 +221,7 @@
static void free_child (struct child *);
static void start_job_command (struct child *child);
static int load_too_high (void);
+static int memory_too_high (void);
static int job_next_command (struct child *);
static int start_waiting_job (struct child *);
@@ -1616,7 +1617,7 @@
/* If we are running at least one job already and the load average
is too high, make this one wait. */
if (!c->remote
- && ((job_slots_used > 0 && load_too_high ())
+ && ((job_slots_used > 0 && (load_too_high () || memory_too_high ()))
#ifdef WINDOWS32
|| process_table_full ()
#endif
@@ -2017,8 +2018,9 @@
OK, I'm not sure exactly how to handle that, but for sure we need to
clamp this value at the number of cores before this can be enabled.
*/
-#define PROC_FD_INIT -1
- static int proc_fd = PROC_FD_INIT;
+#ifdef linux
+ static int proc_fd = -2;
+#endif
double load, guess;
time_t now;
@@ -2032,6 +2034,7 @@
if (max_load_average < 0)
return 0;
+#ifdef linux
/* If we haven't tried to open /proc/loadavg, try now. */
#define LOADAVG "/proc/loadavg"
if (proc_fd == -2)
@@ -2093,6 +2096,7 @@
close (proc_fd);
proc_fd = -1;
}
+#endif /* linux */
/* Find the real system load average. */
make_access ();
@@ -2138,6 +2142,111 @@
#endif
}
+static int
+memory_too_high (void)
+{
+#ifdef linux
+ static int proc_fd = -2;
+
+ if (max_memory_used == 0.0)
+ return 0;
+
+#define MEMINFO "/proc/meminfo"
+ if (proc_fd == -2)
+ {
+ EINTRLOOP (proc_fd, open (MEMINFO, O_RDONLY));
+ if (proc_fd < 0)
+ DB (DB_JOBS, ("Cannot use " MEMINFO " as memory use detection
method.\n"));
+ else
+ {
+ DB (DB_JOBS, ("Using " MEMINFO " memory use detection method.\n"));
+ fd_noinherit (proc_fd);
+ }
+ }
+
+ if (proc_fd >= 0)
+ {
+ int r;
+
+ EINTRLOOP (r, lseek (proc_fd, 0, SEEK_SET));
+ if (r >= 0)
+ {
+#define PROC_MEMINFO_SIZE 256
+ char mem[PROC_MEMINFO_SIZE+1];
+
+ EINTRLOOP (r, read (proc_fd, mem, PROC_MEMINFO_SIZE));
+ if (r >= 0)
+ {
+ const char *p;
+ char *end;
+ double total = -1.0, avail = -1.0, used;
+
+ /* The structure of /proc/meminfo is:
+ MemTotal: ... kB\n
+ MemFree: ... kB\n
+ MemAvailable: ... kB\n
+ ...
+ The difference between MemTotal and MemAvailable is
+ the amount of memory currently used. */
+ mem[r] = '\0';
+ p = strchr (mem, ':');
+ if (p)
+ total = strtod (++p, &end);
+ if (end > p)
+ {
+ p = strchr (end, ':');
+ if (p)
+ p = strchr (p+1, ':');
+ if (p)
+ avail = strtod (p+1, &end);
+ }
+ used = total - avail;
+ if (total <= 0.0 || avail < 0.0 || used < 0.0)
+ {
+ /* The reported values in /proc/meminfo are invalid. */
+ DB (DB_JOBS, ("Mem = %.3f kB, Avail = %.3f kB, Used = %.3f
kB\n",
+ total, avail, used));
+ return 1;
+ }
+ if (max_memory_used <= 1.0)
+ /* We are using percentages, so scale down. */
+ used /= total;
+ if (used >= max_memory_used)
+ /* The limit has been reached, so no new jobs for now. */
+ return 1;
+ if (max_load_average >= 1.0)
+ {
+ /* When a load average is specified together with a
+ memory limit do we scale the number of jobs with
+ the amount of memory used in order to stabilize
+ the memory use and to reduce load and memory
+ spikes.
+
+ The function here creates a balance between speed
+ at a low memory use (creating slightly more jobs)
+ and linear scalability at a high memory use.
+ */
+ double rt = used / max_memory_used;
+
+ return job_slots_used + 1 >=
+ max_load_average * (1.0 - (2.0 - rt) * rt * rt);
+ }
+ return 0;
+ }
+ }
+
+ /* If we got here, something went wrong. Give up on this method. */
+ if (r < 0)
+ DB (DB_JOBS, ("Failed to read " MEMINFO ": %s\n", strerror (errno)));
+
+ close (proc_fd);
+ proc_fd = -1;
+ }
+#endif /* linux */
+
+ return 0;
+}
+
/* Start jobs that are waiting for the load to be lower. */
void
diff -r -u make-4.3/src/main.c make-4.3.1/src/main.c
--- make-4.3/src/main.c 2020-01-19 20:32:59.000000000 +0000
+++ make-4.3.1/src/main.c 2022-02-11 14:20:07.372986002 +0000
@@ -36,6 +36,9 @@
#ifdef HAVE_STRINGS_H
# include <strings.h> /* for strcasecmp */
#endif
+#ifdef HAVE_STDLIB_H
+# include <stdlib.h> /* for strtod */
+#endif
# include "pathstuff.h"
# include "sub_proc.h"
# include "w32err.h"
@@ -279,6 +282,13 @@
double max_load_average = -1.0;
double default_load_average = -1.0;
+/* Maximum memory usage upto which multiple jobs will be run. Zero
+ means unlimited, values between 0.0 and 1.0 are treated as a
+ percentage of total memory, greater values are treated as
+ absolute values in kilobytes. */
+const char *memory_used_arg = NULL;
+double max_memory_used = 0.0;
+
/* List of directories given with -C switches. */
static struct stringlist *directories = 0;
@@ -397,6 +407,9 @@
N_("\
--trace Print tracing information.\n"),
N_("\
+ -u [N], --memory-used=[N] Don't start multiple jobs when system memory
use\n\
+ is above N.\n"),
+ N_("\
-v, --version Print the version number of make and exit.\n"),
N_("\
-w, --print-directory Print the current directory.\n"),
@@ -451,6 +464,7 @@
&default_load_average, "load-average" },
{ 'o', filename, &old_files, 0, 0, 0, 0, 0, "old-file" },
{ 'O', string, &output_sync_option, 1, 1, 0, "target", 0, "output-sync" },
+ { 'u', string, &memory_used_arg, 1, 1, 0, "0", "0", "memory-used" },
{ 'W', filename, &new_files, 0, 0, 0, 0, 0, "what-if" },
/* These are long-style options. */
@@ -799,6 +813,58 @@
#endif
}
+/* Decode the -u, --memory-used argument. A number between 0.0 and 1.0
+ is take as a percentage of the reported total memory. Numbers
+ greater 1.0 can be followed by a unit such as k, m, or g, or the
+ percent sign. Absolute values greater than 1.0 are represented in
+ kilobytes internally. */
+
+static void
+decode_memory_used_arg (void)
+{
+ if (memory_used_arg != NULL)
+ {
+ double value;
+ char *end;
+
+ value = strtod (memory_used_arg, &end);
+ if (value > 0.0 && end != NULL && *end)
+ {
+ switch (tolower (*end))
+ {
+ case 'k': /* kilo */
+ break;
+ case 'm': /* mega */
+ value *= (double) (1UL<<10);
+ break;
+ case 'g': /* giga */
+ value *= (double) (1UL<<20);
+ break;
+ case 't': /* tera */
+ value *= (double) (1UL<<30);
+ break;
+ case 'p': /* peta */
+ value *= (double) (1UL<<40);
+ break;
+ case 'e': /* exa */
+ value *= (double) (1UL<<50);
+ break;
+ case '%':
+ /* Treat percentage values greater 100% as invalid. */
+ value *= (value <= 100.0) ? 0.01 : -1.0;
+ break;
+ default:
+ value /= 1024.0;
+ break;
+ }
+ }
+ if (value < 0.0)
+ OS (fatal, NILF,
+ _("Invalid system memory limit '%s'"), memory_used_arg);
+ max_memory_used = value;
+ }
+}
+
#ifdef WINDOWS32
#ifndef NO_OUTPUT_SYNC
@@ -3008,6 +3074,7 @@
/* If there are any options that need to be decoded do it now. */
decode_debug_flags ();
decode_output_sync_flags ();
+ decode_memory_used_arg ();
/* Perform any special switch handling. */
run_silent = silent_flag;
diff -r -u make-4.3/src/makeint.h make-4.3.1/src/makeint.h
--- make-4.3/src/makeint.h 2020-01-19 20:32:59.000000000 +0000
+++ make-4.3.1/src/makeint.h 2022-01-20 20:09:23.412527682 +0000
@@ -687,6 +687,7 @@
extern unsigned int job_slots;
extern double max_load_average;
+extern double max_memory_used;
extern const char *program;
