Re: [OE-core] [PATCH 1/5] classes: jobserver: support gnu make fifo jobserver
Hi Andreas,
On Thu, 2024-04-04 at 09:56 +0200, Andreas Helbech Kleist wrote:
> > + # avoid making make-native or its dependencies depend on make-
> > native itself
> > + if d.getVar("PN") in (
> > + "make-native",
> > + "libtool-native",
> > + "pkgconfig-native",
> > + "automake-native",
> > + "autoconf-native",
> > + "m4-native",
> > + "texinfo-dummy-native",
> > + "gettext-minimal-native",
> > + "quilt-native",
> > + "gnu-config-native",
>
> This seems a bit brittle, but I guess make's dependencies doesn't
> change often. So it is probably not a big issue.
>
> How does it break if a dependency is missing?
Bitbake errors out with a dependency loop.
// Martin
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197948):
https://lists.openembedded.org/g/openembedded-core/message/197948
Mute This Topic: https://lists.openembedded.org/mt/105304233/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 1/5] classes: jobserver: support gnu make fifo jobserver
Hi
Very interesting patch series. I stumbled upon a couple of spelling
errors while looking through it, and have a question about the
hardcoded dependency list.
On Wed, 2024-04-03 at 09:02 +0200, Martin Hundeb?ll wrote:
> Add a class to implement the gnu make fifo style jobserver. The class
> can be activated by symply adding an `INHERIT += "jobserver"` to the
s/symply/simply/
> local configuration.
>
> Furthermore, one can configure an external jobserver (i.e. a server
> shared between multiple builds), by configured the `JOBSERVER_FIFO`
> variable to point at an existing jobserver fifo.
>
> The jobserver class uses the fifo style jobserver, which doesn't require
> passing open file descriptors around. It does, however, require
> make-4.4, which isn't available in common distro yet. To work around
> this, the class make all recipes (except make and its dependencies
> itself) depend on `virtual/make-native`.
>
> Signed-off-by: Martin Hundebøll
> ---
> meta/classes-global/jobserver.bbclass | 80 +++
> meta/conf/bitbake.conf| 2 +-
> 2 files changed, 81 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes-global/jobserver.bbclass
>
> diff --git a/meta/classes-global/jobserver.bbclass
> b/meta/classes-global/jobserver.bbclass
> new file mode 100644
> index 00..c76909fe50
> --- /dev/null
> +++ b/meta/classes-global/jobserver.bbclass
> @@ -0,0 +1,80 @@
> +JOBSERVER_FIFO ?= ""
> +JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of
> creating a per-build server."
> +
> +addhandler jobserver_setup_fifo
> +jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
> +
> +python jobserver_setup_fifo() {
> +# don't setup a per-build fifo, if an external one is configured
> +if d.getVar("JOBSERVER_FIFO"):
> +return
> +
> +# don't use a job-server if no parallelism is configured
> +jobs = oe.utils.parallel_make(d)
> +if jobs in (None, 1):
> +return
> +
> +# reduce jobs by one as a token has implicitly been handed to the
> +# process requesting tokens
> +jobs -= 1
> +
> +fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
> +
> +# and old fifo might be lingering; remove it
> +if os.path.exists(fifo):
> +os.remove(fifo)
> +
> +# create a new fifo to use for communicating tokens
> +os.mkfifo(fifo)
> +
> +# fill the fifo with the number of tokens to hand out
> +wfd = os.open(fifo, os.O_RDWR)
> +written = os.write(wfd, b"+" * jobs)
> +if written != (jobs):
> +bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
> +
> +# configure the per-build fifo path to use
> +d.setVar("JOBSERVER_FIFO", fifo)
> +}
> +
> +python () {
> +# don't configure the fifo if none is defined
> +fifo = d.getVar("JOBSERVER_FIFO")
> +if not fifo:
> +return
> +
> +# avoid making make-native or its dependencies depend on make-native
> itself
> +if d.getVar("PN") in (
> +"make-native",
> +"libtool-native",
> +"pkgconfig-native",
> +"automake-native",
> +"autoconf-native",
> +"m4-native",
> +"texinfo-dummy-native",
> +"gettext-minimal-native",
> +"quilt-native",
> +"gnu-config-native",
This seems a bit brittle, but I guess make's dependencies doesn't
change often. So it is probably not a big issue.
How does it break if a dependency is missing?
> +):
> +return
> +
> +# don't make unwilling recipes depend on make-native
> +if d.getVar('INHIBIT_DEFAULT_DEPS', False):
> +return
> +
> +# make other recipes depend on make-native to make sure it is new enough
> to
> +# support the --jobserver-auth=fifo: syntax (from make-4.4 and
> onwards)
> +d.appendVar("DEPENDS", " virtual/make-native")
> +
> +# disable the "-j " flag, as that overrides the jobserver fifo
> tokens
> +d.setVar("PARALLEL_MAKE", "")
> +d.setVar("PARALLEL_MAKEINST", "")
> +
> +# set and export the jobserver in the environment
> +d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
> +d.setVarFlag("MAKEFLAGS", "export", "1")
> +
> +# ignore the joberserver argument part of MAKEFLAGS in the hash, as that
s/joberserver/jobserver/
> +# shouldn't change the build output
> +d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "|
> --jobserver-auth=fifo:" + fifo)
> +}
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 6f180d18b0..23a016b31e 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -960,7 +960,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
> BB_TASKHASH BBPATH BBSERVER DL_DI
> BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot
> DEPLOY_DIR \
> SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
> SSTATE_HASHEQUIV_OWNER
[OE-core] [PATCH 1/5] classes: jobserver: support gnu make fifo jobserver
Add a class to implement the gnu make fifo style jobserver. The class
can be activated by symply adding an `INHERIT += "jobserver"` to the
local configuration.
Furthermore, one can configure an external jobserver (i.e. a server
shared between multiple builds), by configured the `JOBSERVER_FIFO`
variable to point at an existing jobserver fifo.
The jobserver class uses the fifo style jobserver, which doesn't require
passing open file descriptors around. It does, however, require
make-4.4, which isn't available in common distro yet. To work around
this, the class make all recipes (except make and its dependencies
itself) depend on `virtual/make-native`.
Signed-off-by: Martin Hundebøll
---
meta/classes-global/jobserver.bbclass | 80 +++
meta/conf/bitbake.conf| 2 +-
2 files changed, 81 insertions(+), 1 deletion(-)
create mode 100644 meta/classes-global/jobserver.bbclass
diff --git a/meta/classes-global/jobserver.bbclass
b/meta/classes-global/jobserver.bbclass
new file mode 100644
index 00..c76909fe50
--- /dev/null
+++ b/meta/classes-global/jobserver.bbclass
@@ -0,0 +1,80 @@
+JOBSERVER_FIFO ?= ""
+JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of
creating a per-build server."
+
+addhandler jobserver_setup_fifo
+jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
+
+python jobserver_setup_fifo() {
+# don't setup a per-build fifo, if an external one is configured
+if d.getVar("JOBSERVER_FIFO"):
+return
+
+# don't use a job-server if no parallelism is configured
+jobs = oe.utils.parallel_make(d)
+if jobs in (None, 1):
+return
+
+# reduce jobs by one as a token has implicitly been handed to the
+# process requesting tokens
+jobs -= 1
+
+fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
+
+# and old fifo might be lingering; remove it
+if os.path.exists(fifo):
+os.remove(fifo)
+
+# create a new fifo to use for communicating tokens
+os.mkfifo(fifo)
+
+# fill the fifo with the number of tokens to hand out
+wfd = os.open(fifo, os.O_RDWR)
+written = os.write(wfd, b"+" * jobs)
+if written != (jobs):
+bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
+
+# configure the per-build fifo path to use
+d.setVar("JOBSERVER_FIFO", fifo)
+}
+
+python () {
+# don't configure the fifo if none is defined
+fifo = d.getVar("JOBSERVER_FIFO")
+if not fifo:
+return
+
+# avoid making make-native or its dependencies depend on make-native itself
+if d.getVar("PN") in (
+"make-native",
+"libtool-native",
+"pkgconfig-native",
+"automake-native",
+"autoconf-native",
+"m4-native",
+"texinfo-dummy-native",
+"gettext-minimal-native",
+"quilt-native",
+"gnu-config-native",
+):
+return
+
+# don't make unwilling recipes depend on make-native
+if d.getVar('INHIBIT_DEFAULT_DEPS', False):
+return
+
+# make other recipes depend on make-native to make sure it is new enough to
+# support the --jobserver-auth=fifo: syntax (from make-4.4 and
onwards)
+d.appendVar("DEPENDS", " virtual/make-native")
+
+# disable the "-j " flag, as that overrides the jobserver fifo tokens
+d.setVar("PARALLEL_MAKE", "")
+d.setVar("PARALLEL_MAKEINST", "")
+
+# set and export the jobserver in the environment
+d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
+d.setVarFlag("MAKEFLAGS", "export", "1")
+
+# ignore the joberserver argument part of MAKEFLAGS in the hash, as that
+# shouldn't change the build output
+d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "|
--jobserver-auth=fifo:" + fifo)
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6f180d18b0..23a016b31e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -960,7 +960,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH
BBPATH BBSERVER DL_DI
BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot
DEPLOY_DIR \
SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES
\
-OMP_NUM_THREADS BB_CURRENTTASK"
+OMP_NUM_THREADS BB_CURRENTTASK JOBSERVER_FIFO"
BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS
BUILDHISTORY_DIR \
SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS"
BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID
\
--
2.44.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197910):
https://lists.openembedded.org/g/openembedded-core/message/197910
Mute This Topic: https://lists.openembedded.org/mt/105304233/21656
Group Owner:
