Aug 28, 2023 14:48:46 Martin Hundebøll <[email protected]>:
> Add a class to implement the gnu make fifo style jobserver. The class
> can be activated by symply adding an `INHERIT += "jobserver"` to the
> local configuration.
>
> Furthermore, one can configure an external jobserver (i.e. a server
> shared between multiple builds), by configured the `JOBSERVER_FIFO`
> variable to point at an existing jobserver fifo.
>
> The jobserver class uses the fifo style jobserver, which doesn't require
> passing open file descriptors around. It does, however, require
> make-4.4, which isn't available in common distro yet. To work around
> this, the class make all recipes (except make and its dependencies
> itself) depend on `virtual/make-native`.
>
> Signed-off-by: Martin Hundebøll <[email protected]>
> ---
> meta/classes-global/jobserver.bbclass | 80 +++++++++++++++++++++++++++
> meta/conf/bitbake.conf | 2 +-
> 2 files changed, 81 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes-global/jobserver.bbclass
>
> diff --git a/meta/classes-global/jobserver.bbclass
> b/meta/classes-global/jobserver.bbclass
> new file mode 100644
> index 0000000000..c76909fe50
> --- /dev/null
> +++ b/meta/classes-global/jobserver.bbclass
> @@ -0,0 +1,80 @@
> +JOBSERVER_FIFO ?= ""
> +JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of
> creating a per-build server."
> +
> +addhandler jobserver_setup_fifo
> +jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
> +
> +python jobserver_setup_fifo() {
> + # don't setup a per-build fifo, if an external one is configured
> + if d.getVar("JOBSERVER_FIFO"):
> + return
> +
> + # don't use a job-server if no parallelism is configured
> + jobs = oe.utils.parallel_make(d)
> + if jobs in (None, 1):
> + return
> +
> + # reduce jobs by one as a token has implicitly been handed to the
> + # process requesting tokens
> + jobs -= 1
> +
> + fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
> +
> + # and old fifo might be lingering; remove it
> + if os.path.exists(fifo):
> + os.remove(fifo)
> +
> + # create a new fifo to use for communicating tokens
> + os.mkfifo(fifo)
> +
> + # fill the fifo with the number of tokens to hand out
> + wfd = os.open(fifo, os.O_RDWR)
> + written = os.write(wfd, b"+" * jobs)
> + if written != (jobs):
> + bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
> +
> + # configure the per-build fifo path to use
> + d.setVar("JOBSERVER_FIFO", fifo)
> +}
> +
> +python () {
> + # don't configure the fifo if none is defined
> + fifo = d.getVar("JOBSERVER_FIFO")
> + if not fifo:
> + return
> +
> + # avoid making make-native or its dependencies depend on make-native
> itself
> + if d.getVar("PN") in (
> + "make-native",
> + "libtool-native",
> + "pkgconfig-native",
> + "automake-native",
> + "autoconf-native",
> + "m4-native",
> + "texinfo-dummy-native",
> + "gettext-minimal-native",
> + "quilt-native",
> + "gnu-config-native",
> + ):
> + return
> +
> + # don't make unwilling recipes depend on make-native
> + if d.getVar('INHIBIT_DEFAULT_DEPS', False):
> + return
> +
> + # make other recipes depend on make-native to make sure it is new enough
> to
> + # support the --jobserver-auth=fifo:<path> syntax (from make-4.4 and
> onwards)
> + d.appendVar("DEPENDS", " virtual/make-native")
I would like some feedback on this part, i.e. changing package dependencies
depending on a pure build-configuration like this. I would prefer if the build
didn't change at all when enabling the jobserver class, which would require
adding virtual/make-native to the base dependencies regardless of this class.
Would that be acceptable?
> + # disable the "-j <jobs>" flag, as that overrides the jobserver fifo
> tokens
> + d.setVar("PARALLEL_MAKE", "")
> + d.setVar("PARALLEL_MAKEINST", "")
> +
> + # set and export the jobserver in the environment
> + d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
> + d.setVarFlag("MAKEFLAGS", "export", "1")
> +
> + # ignore the joberserver argument part of MAKEFLAGS in the hash, as that
> + # shouldn't change the build output
> + d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "|
> --jobserver-auth=fifo:" + fifo)
> +}
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index cf7ff3328c..8cf188270b 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -946,7 +946,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
> BB_TASKHASH BBPATH BBSERVER DL_DI
> BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot
> DEPLOY_DIR \
> SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
> SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE
> GIT_CEILING_DIRECTORIES \
> - OMP_NUM_THREADS BB_CURRENTTASK"
> + OMP_NUM_THREADS BB_CURRENTTASK JOBSERVER_FIFO"
> BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS
> BUILDHISTORY_DIR \
> SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS
> RUST_TARGET_SYS"
> BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME
> SSH_AGENT_PID \
> --
> 2.41.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186842):
https://lists.openembedded.org/g/openembedded-core/message/186842
Mute This Topic: https://lists.openembedded.org/mt/101009092/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
