from:"Martin Hundeb\?ll"

Re: [OE-core] [PATCH] qemu: upgrade 8.2.1 -> 9.0.0

2024-06-04 Thread Martin Hundeb?ll

On Tue, 2024-06-04 at 08:13 +0100, Richard Purdie wrote:
> On Wed, 2024-05-22 at 15:28 +0200, Martin Hundeb?ll via
> lists.openembedded.org wrote:
> > Remove pacthes accepted upstream, and refresh the remaining ones.
> > Remove
> > patches required by ubuntu 18.04, as those are unmaintainable.
> > 
> > Remove CVE status for CVEs not applicable to the new version.
> > 
> > Exclude hppa-firmware{,64}.img files from stripping during
> > packaging, as
> > those (binary) files are already stripped.
> > 
> > Signed-off-by: Martin Hundebøll 
> > ---
> > 
> > I'm no sure about the CVE changes, but the one I removed didn't
> > have
> > qemu-9.* listed as a vulnerable configuration.
> > 
> > This has been tested with runqemu on a standard qemux86-64 machine.
> > If
> > it breaks miserably elsewhere, then at least there's a starting
> > point.
> 
> Thanks for sending this patch. I've tweaked it to delete the hppa
> firmware files since we don't use that and they were causing multiple
> problems. I agree it is time to remove those 18.04 patches.

Removing those files seems like a reasonable approach.

> Testing seems to be otherwise working so we can probably take the
> upgrade :)

Great!

// Martin

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200298): 
https://lists.openembedded.org/g/openembedded-core/message/200298
Mute This Topic: https://lists.openembedded.org/mt/106242475/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] qemu: upgrade 8.2.1 -> 9.0.0

2024-05-27 Thread Martin Hundeb?ll

On Mon, 2024-05-27 at 20:38 +0200, Alexandre Belloni wrote:
> This conflicts with "qemu: enable parallel builds when using the
> jobserver class" that I'm currently carrying, should I drop it?
> 

It's.. complicated... :)

This qemu update includes the change you mention, so upgrading is
probably the preferred way forward.

But qemu is also a key component in parts of many builds, so upgrading
it might lead to regressions.

How sure are we about an upgrade like this if it passes the
autobuilder?

Anyways, you can drop the jobserver series regardless. I'm currently
upstreaming the ninja part of the series, so upgrading both qemu and
ninja will make the jobserver series smaller.

// Martin

> On 22/05/2024 15:28:38+0200, Martin Hundeb?ll wrote:
> > Remove pacthes accepted upstream, and refresh the remaining ones.
> > Remove
> > patches required by ubuntu 18.04, as those are unmaintainable.
> > 
> > Remove CVE status for CVEs not applicable to the new version.
> > 
> > Exclude hppa-firmware{,64}.img files from stripping during
> > packaging, as
> > those (binary) files are already stripped.
> > 
> > Signed-off-by: Martin Hundebøll 
> > ---
> > 
> > I'm no sure about the CVE changes, but the one I removed didn't
> > have
> > qemu-9.* listed as a vulnerable configuration.
> > 
> > This has been tested with runqemu on a standard qemux86-64 machine.
> > If
> > it breaks miserably elsewhere, then at least there's a starting
> > point.
> > 
> >  meta/conf/distro/include/tcmode-default.inc   |   2 +-
> >  ...u-native_8.2.1.bb => qemu-native_9.0.0.bb} |   0
> >  ...e_8.2.1.bb => qemu-system-native_9.0.0.bb} |   0
> >  meta/recipes-devtools/qemu/qemu.inc   |  58 +---
> >  ...4-Handle-the-vsyscall-page-in-open_s.patch |  56 
> >  ...n-environment-space-to-boot-loader-q.patch |  18 +-
> >  ... 0002-apic-fixup-fallthrough-to-PIC.patch} |  18 +-
> >  ...ongarch64-Remove-TARGET_FORCE_SHMLBA.patch |  43 ---
> >  ...d-pkg-config-handling-for-libgcrypt.patch} |  18 +-
> >  ...0003-linux-user-Add-strace-for-shmat.patch |  71 -
> >  ...0004-linux-user-Rewrite-target_shmat.patch | 236 --
> > -
> >  ...u-Do-not-include-file-if-not-exists.patch} |  18 +-
> >  ...r-space-mmap-tweaks-to-address-musl.patch} |  18 +-
> >  ...that-shmat-does-not-break-proc-self-.patch |  85 --
> >  ...atch => 0006-qemu-Determinism-fixes.patch} |  18 +-
> >  ...-use-relative-path-to-refer-to-file.patch} |  20 +-
> >  ...nd-MAP_SHARED_VALIDATE-on-needed-li.patch} |  16 +-
> >  ...e-use-of-lfs64-related-functions-an.patch} |  61 ++--
> >  ...ure-lookup-meson-exutable-from-PATH.patch} |  21 +-
> >  ...t-against-buggy-or-malicious-guest-d.patch |  40 ---
> >  ...round-for-missing-MAP_FIXED_NOREPLAC.patch | 282 --
> > 
> >  ...nd-the-python-venv-aren-t-used-for-.patch} |  26 +-
> >  ...round-for-missing-MAP_SHARED_VALIDAT.patch |  51 
> >  ...79ad8629b57a43daa62e46cc7af6e1078116.patch |  60 
> >  .../qemu/qemu/CVE-2023-6683.patch |  91 --
> >  .../qemu/{qemu_8.2.1.bb => qemu_9.0.0.bb} |   0
> >  26 files changed, 163 insertions(+), 1164 deletions(-)
> >  rename meta/recipes-devtools/qemu/{qemu-native_8.2.1.bb => qemu-
> > native_9.0.0.bb} (100%)
> >  rename meta/recipes-devtools/qemu/{qemu-system-native_8.2.1.bb =>
> > qemu-system-native_9.0.0.bb} (100%)
> >  delete mode 100644 meta/recipes-devtools/qemu/qemu/0001-linux-
> > user-x86_64-Handle-the-vsyscall-page-in-open_s.patch
> >  rename meta/recipes-devtools/qemu/qemu/{0003-apic-fixup-
> > fallthrough-to-PIC.patch => 0002-apic-fixup-fallthrough-to-
> > PIC.patch} (80%)
> >  delete mode 100644 meta/recipes-devtools/qemu/qemu/0002-linux-
> > user-loongarch64-Remove-TARGET_FORCE_SHMLBA.patch
> >  rename meta/recipes-devtools/qemu/qemu/{0004-configure-Add-pkg-
> > config-handling-for-libgcrypt.patch => 0003-configure-Add-pkg-
> > config-handling-for-libgcrypt.patch} (73%)
> >  delete mode 100644 meta/recipes-devtools/qemu/qemu/0003-linux-
> > user-Add-strace-for-shmat.patch
> >  delete mode 100644 meta/recipes-devtools/qemu/qemu/0004-linux-
> > user-Rewrite-target_shmat.patch
> >  rename meta/recipes-devtools/qemu/qemu/{0005-qemu-Do-not-include-
> > file-if-not-exists.patch => 0004-qemu-Do-not-include-file-if-not-
> > exists.patch} (67%)
> >  rename meta/recipes-devtools/qemu/qemu/{0006-qemu-Add-some-user-
> > space-mmap-tweaks-to-address-musl.patch => 0005-qemu-Add-some-user-
> > space-mmap-tweaks-to-address-musl.patch} (80%)
> >

[OE-core] [PATCH][master][scarthgap] classes: image_types: quote variable assignment needed by dash

2024-05-27 Thread Martin Hundeb?ll

The change in commit 39fc503036
("classes: image_types: apply EXTRA_IMAGECMD:squashfs* in oe_mksquashfs()")
assigns $@ to a local variable without quoting it. While this works with
bash, it fails with dash. Here, only the first token of $@ is assigned
to the variable, and the reamining tokens are passed as arguments to the
"local" keyword.

Fix it by adding the missing quotes.

Signed-off-by: Martin Hundebøll 
---
 meta/classes-recipe/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/image_types.bbclass 
b/meta/classes-recipe/image_types.bbclass
index b4a83ae284..2f948ecbf8 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -113,7 +113,7 @@ IMAGE_CMD:btrfs () {
 
 oe_mksquashfs () {
 local comp=$1; shift
-local extra_imagecmd=$@
+local extra_imagecmd="$@"
 
 if [ "$comp" = "zstd" ]; then
 suffix="zst"
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199900): 
https://lists.openembedded.org/g/openembedded-core/message/199900
Mute This Topic: https://lists.openembedded.org/mt/106329106/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] qemu: upgrade 8.2.1 -> 9.0.0

2024-05-22 Thread Martin Hundeb?ll

Remove pacthes accepted upstream, and refresh the remaining ones. Remove
patches required by ubuntu 18.04, as those are unmaintainable.

Remove CVE status for CVEs not applicable to the new version.

Exclude hppa-firmware{,64}.img files from stripping during packaging, as
those (binary) files are already stripped.

Signed-off-by: Martin Hundebøll 
---

I'm no sure about the CVE changes, but the one I removed didn't have
qemu-9.* listed as a vulnerable configuration.

This has been tested with runqemu on a standard qemux86-64 machine. If
it breaks miserably elsewhere, then at least there's a starting point.

 meta/conf/distro/include/tcmode-default.inc   |   2 +-
 ...u-native_8.2.1.bb => qemu-native_9.0.0.bb} |   0
 ...e_8.2.1.bb => qemu-system-native_9.0.0.bb} |   0
 meta/recipes-devtools/qemu/qemu.inc   |  58 +---
 ...4-Handle-the-vsyscall-page-in-open_s.patch |  56 
 ...n-environment-space-to-boot-loader-q.patch |  18 +-
 ... 0002-apic-fixup-fallthrough-to-PIC.patch} |  18 +-
 ...ongarch64-Remove-TARGET_FORCE_SHMLBA.patch |  43 ---
 ...d-pkg-config-handling-for-libgcrypt.patch} |  18 +-
 ...0003-linux-user-Add-strace-for-shmat.patch |  71 -
 ...0004-linux-user-Rewrite-target_shmat.patch | 236 ---
 ...u-Do-not-include-file-if-not-exists.patch} |  18 +-
 ...r-space-mmap-tweaks-to-address-musl.patch} |  18 +-
 ...that-shmat-does-not-break-proc-self-.patch |  85 --
 ...atch => 0006-qemu-Determinism-fixes.patch} |  18 +-
 ...-use-relative-path-to-refer-to-file.patch} |  20 +-
 ...nd-MAP_SHARED_VALIDATE-on-needed-li.patch} |  16 +-
 ...e-use-of-lfs64-related-functions-an.patch} |  61 ++--
 ...ure-lookup-meson-exutable-from-PATH.patch} |  21 +-
 ...t-against-buggy-or-malicious-guest-d.patch |  40 ---
 ...round-for-missing-MAP_FIXED_NOREPLAC.patch | 282 --
 ...nd-the-python-venv-aren-t-used-for-.patch} |  26 +-
 ...round-for-missing-MAP_SHARED_VALIDAT.patch |  51 
 ...79ad8629b57a43daa62e46cc7af6e1078116.patch |  60 
 .../qemu/qemu/CVE-2023-6683.patch |  91 --
 .../qemu/{qemu_8.2.1.bb => qemu_9.0.0.bb} |   0
 26 files changed, 163 insertions(+), 1164 deletions(-)
 rename meta/recipes-devtools/qemu/{qemu-native_8.2.1.bb => 
qemu-native_9.0.0.bb} (100%)
 rename meta/recipes-devtools/qemu/{qemu-system-native_8.2.1.bb => 
qemu-system-native_9.0.0.bb} (100%)
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0001-linux-user-x86_64-Handle-the-vsyscall-page-in-open_s.patch
 rename 
meta/recipes-devtools/qemu/qemu/{0003-apic-fixup-fallthrough-to-PIC.patch => 
0002-apic-fixup-fallthrough-to-PIC.patch} (80%)
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0002-linux-user-loongarch64-Remove-TARGET_FORCE_SHMLBA.patch
 rename 
meta/recipes-devtools/qemu/qemu/{0004-configure-Add-pkg-config-handling-for-libgcrypt.patch
 => 0003-configure-Add-pkg-config-handling-for-libgcrypt.patch} (73%)
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0003-linux-user-Add-strace-for-shmat.patch
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0004-linux-user-Rewrite-target_shmat.patch
 rename 
meta/recipes-devtools/qemu/qemu/{0005-qemu-Do-not-include-file-if-not-exists.patch
 => 0004-qemu-Do-not-include-file-if-not-exists.patch} (67%)
 rename 
meta/recipes-devtools/qemu/qemu/{0006-qemu-Add-some-user-space-mmap-tweaks-to-address-musl.patch
 => 0005-qemu-Add-some-user-space-mmap-tweaks-to-address-musl.patch} (80%)
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0005-tests-tcg-Check-that-shmat-does-not-break-proc-self-.patch
 rename meta/recipes-devtools/qemu/qemu/{0007-qemu-Determinism-fixes.patch => 
0006-qemu-Determinism-fixes.patch} (71%)
 rename 
meta/recipes-devtools/qemu/qemu/{0008-tests-meson.build-use-relative-path-to-refer-to-file.patch
 => 0007-tests-meson.build-use-relative-path-to-refer-to-file.patch} (76%)
 rename 
meta/recipes-devtools/qemu/qemu/{0009-Define-MAP_SYNC-and-MAP_SHARED_VALIDATE-on-needed-li.patch
 => 0008-Define-MAP_SYNC-and-MAP_SHARED_VALIDATE-on-needed-li.patch} (80%)
 rename 
meta/recipes-devtools/qemu/qemu/{0002-linux-user-Replace-use-of-lfs64-related-functions-an.patch
 => 0009-linux-user-Replace-use-of-lfs64-related-functions-an.patch} (81%)
 rename meta/recipes-devtools/qemu/qemu/{fixedmeson.patch => 
0010-configure-lookup-meson-exutable-from-PATH.patch} (53%)
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0010-hw-pvrdma-Protect-against-buggy-or-malicious-guest-d.patch
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0011-linux-user-workaround-for-missing-MAP_FIXED_NOREPLAC.patch
 rename meta/recipes-devtools/qemu/qemu/{no-pip.patch => 
0011-qemu-Ensure-pip-and-the-python-venv-aren-t-used-for-.patch} (70%)
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/0012-linux-user-workaround-for-missing-MAP_SHARED_VALIDAT.patch
 delete mode 100644 
meta/recipes-devtools/qemu/qemu/4a8579ad8629b57a43daa62e46cc7af6e1078116.patch
 delete mode 100644 meta/recipes-devtools/qemu/qemu/CVE-2023-6683.patch
 rename

Re: [OE-core] [AUH] qemu: upgrading to 9.0.0 FAILED

2024-05-22 Thread Martin Hundeb?ll

Upgrading to qemu-9.0.0 involves rebasing patches needed for ubuntu-
18.04. Is this really needed still?

Wouldn't it make sense to drop ubuntu-18.04 for the next release?

// Martin

On Wed, 2024-05-15 at 17:32 +, Auto Upgrade Helper wrote:
> Hello,
> 
> this email is a notification from the Auto Upgrade Helper
> that the automatic attempt to upgrade the recipe *qemu* to *9.0.0*
> has Failed (devtool error).
> 
> Detailed error information:
> 
> Running 'devtool upgrade' for recipe qemu failed.
> NOTE: Reconnecting to bitbake server...
> Loading cache...done.
> Loaded 1881 entries from dependency cache.
> NOTE: Resolving any missing task queue dependencies
> 
> Build Configuration:
> BB_VERSION   = "2.9.1"
> BUILD_SYS    = "x86_64-linux"
> NATIVELSBSTRING  = "universal"
> TARGET_SYS   = "x86_64-poky-linux"
> MACHINE  = "qemux86-64"
> DISTRO   = "poky"
> DISTRO_VERSION   = "5.0+snapshot-
> b42883fef514bff64dc37a5bcd44402eea630a93"
> TUNE_FEATURES    = "m64 core2"
> TARGET_FPU   = ""
> meta 
> meta-poky    
> meta-yocto-bsp   
> workspace    = "tmp-auh-
> upgrades:b42883fef514bff64dc37a5bcd44402eea630a93"
> 
> Initialising tasks...NOTE: The /proc/pressure files can't be read.
> Continuing build without monitoring pressure
> Sstate summary: Wanted 11 Local 11 Mirrors 0 Missed 0 Current 22
> (100% match, 100% complete)
> done.
> NOTE: Executing Tasks
> NOTE: Tasks Summary: Attempted 113 tasks of which 110 didn't need to
> be rerun and all succeeded.
> NOTE: Writing buildhistory
> NOTE: Writing buildhistory took: 3 seconds
> Loading cache...done.
> Loaded 1881 entries from dependency cache.
> Parsing recipes...done.
> Parsing of 922 .bb files complete (920 cached, 2 parsed). 1880
> targets, 35 skipped, 0 masked, 0 errors.
> NOTE: Resolving any missing task queue dependencies
> 
> Build Configuration:
> BB_VERSION   = "2.9.1"
> BUILD_SYS    = "x86_64-linux"
> NATIVELSBSTRING  = "universal"
> TARGET_SYS   = "x86_64-poky-linux"
> MACHINE  = "qemux86-64"
> DISTRO   = "poky"
> DISTRO_VERSION   = "5.0+snapshot-
> b42883fef514bff64dc37a5bcd44402eea630a93"
> TUNE_FEATURES    = "m64 core2"
> TARGET_FPU   = ""
> meta 
> meta-poky    
> meta-yocto-bsp   
> workspace    = "tmp-auh-
> upgrades:b42883fef514bff64dc37a5bcd44402eea630a93"
> 
> Initialising tasks...NOTE: The /proc/pressure files can't be read.
> Continuing build without monitoring pressure
> Sstate summary: Wanted 10 Local 9 Mirrors 0 Missed 1 Current 18 (90%
> match, 96% complete)
> done.
> NOTE: Executing Tasks
> NOTE: Tasks Summary: Attempted 93 tasks of which 90 didn't need to be
> rerun and all succeeded.
> NOTE: Writing buildhistory
> NOTE: Writing buildhistory took: 3 seconds
> DEBUG 5 [Errno 25] Inappropriate ioctl for device
> Adding changed files:   0% |   |
> ETA:  --:--:--
> Adding changed files:   0% |   |
> ETA:  --:--:--
> Adding changed files:   3% |#   |
> ETA:  0:00:13
> Adding changed files:   6% |##  |
> ETA:  0:00:09
> Adding changed files:   9% |### |
> ETA:  0:00:08
> Adding changed files:  13% |    |
> ETA:  0:00:07
> Adding changed files:  16% |#   |
> ETA:  0:00:07
> Adding changed files:  19% |### |
> ETA:  0:00:07
> Adding changed files:  22% |    |
> ETA:  0:00:06
> Adding changed files:  26% |#   |
> ETA:  0:00:06
> Adding changed files:  29% |##  |
> ETA:  0:00:06
> Adding changed files:  32% |### |
> ETA:  0:00:05
> Adding changed files:  35% |    |
> ETA:  0:00:05
> Adding changed files:  39% |##  |
> ETA:  0:00:05
> Adding changed files:  42% |### |
> ETA:  0:00:04
> Adding changed files:  45% |    |
> ETA:  0:00:04
> Adding changed files:  48% |#   |
> ETA:  0:00:04
> Adding changed files:  52% |##  |
> ETA:  0:00:03
> Adding changed files:  55% |### |
> ETA:  0:00:03
> Adding changed files:  58% |#   |
> ETA:  0:00:03
> Adding changed files:  61% |##  |
> ETA:  0:00:03
> Adding changed files:  65% |### |
> ETA:  0:00:03
> Adding changed files:  68% |    |
> ETA:  0:00:02
> Adding changed files:  71% |#   |
> ETA:  0:00:02
> Adding changed files:  75%

Re: [OE-core] [PATCH 29/42] ofono: upgrade 2.4 -> 2.7

2024-05-22 Thread Martin Hundeb?ll

Hi Wang,

I've sent a similar patch that also removes the two patches, which
aren't needed anymore:
https://lists.openembedded.org/g/openembedded-core/message/199693

// Martin

On Wed, 2024-05-22 at 17:14 +0800, wangmy via lists.openembedded.org
wrote:
> From: Wang Mingyu 
> 
> Changelog:
> ==
> - Fix issue with handling SIM7100 modem enable operation.
> - Fix issue with handling QMI NAS indications earlier.
> - Add support for handling QMI default bearer.
> - Add support for handling QMI DTMF tones.
> - Fix issue with handling ATD*99 for non-muxed ports.
> - Fix issue with handling QRTR discover timeout.
> - Add support for QMI voice call feature.
> - Add support for exposing modem capabilities.
> - Remove support for BlueZ 4.x services.
> - Fix issue with LTE provisioning support.
> - Fix issue with SMS and address length checks.
> - Add support for QRTR services.
> 
> Signed-off-by: Wang Mingyu 
> ---
>  meta/recipes-connectivity/ofono/{ofono_2.4.bb => ofono_2.7.bb} | 2
> +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>  rename meta/recipes-connectivity/ofono/{ofono_2.4.bb =>
> ofono_2.7.bb} (95%)
> 
> diff --git a/meta/recipes-connectivity/ofono/ofono_2.4.bb
> b/meta/recipes-connectivity/ofono/ofono_2.7.bb
> similarity index 95%
> rename from meta/recipes-connectivity/ofono/ofono_2.4.bb
> rename to meta/recipes-connectivity/ofono/ofono_2.7.bb
> index 5e1e5f3b6a..87f1958525 100644
> --- a/meta/recipes-connectivity/ofono/ofono_2.4.bb
> +++ b/meta/recipes-connectivity/ofono/ofono_2.7.bb
> @@ -13,7 +13,7 @@ SRC_URI = "\
> 
> file://0001-mbim-add-an-optional-TEMP_FAILURE_RETRY-macro-copy.patch 
> \
> 
> file://0002-mbim-Fix-build-with-ell-0.39-by-restoring-unlikely-m.patch
>  \
>  "
> -SRC_URI[sha256sum] =
> "93580adc1afd1890dc516efb069de0c5cdfef014415256ddfb28ab172df2d11d"
> +SRC_URI[sha256sum] =
> "dabf6ef06b94beaad65253200abe3887046a4e722f4fe373c4264f357ae47ad3"
>  
>  inherit autotools pkgconfig update-rc.d systemd gobject-
> introspection-data
>  
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199741): 
https://lists.openembedded.org/g/openembedded-core/message/199741
Mute This Topic: https://lists.openembedded.org/mt/106239640/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 1/2] ell: upgrade 0.64 -> 0.65

2024-05-22 Thread Martin Hundeb?ll

Remove do_configure:prepend() that creates build-aux, as that isn't
needed.

Signed-off-by: Martin Hundebøll 
---
 meta/recipes-core/ell/{ell_0.64.bb => ell_0.65.bb} | 6 +-
 1 file changed, 1 insertion(+), 5 deletions(-)
 rename meta/recipes-core/ell/{ell_0.64.bb => ell_0.65.bb} (82%)

diff --git a/meta/recipes-core/ell/ell_0.64.bb 
b/meta/recipes-core/ell/ell_0.65.bb
similarity index 82%
rename from meta/recipes-core/ell/ell_0.64.bb
rename to meta/recipes-core/ell/ell_0.65.bb
index c4f16905fd..9b559caf8a 100644
--- a/meta/recipes-core/ell/ell_0.64.bb
+++ b/meta/recipes-core/ell/ell_0.65.bb
@@ -15,8 +15,4 @@ DEPENDS = "dbus"
 inherit autotools pkgconfig
 
 SRC_URI = 
"https://mirrors.edge.kernel.org/pub/linux/libs/${BPN}/${BPN}-${PV}.tar.xz;
-SRC_URI[sha256sum] = 
"760f3901078409f66cadf1bb24c8bdc60f13d53f6dd66b88631221d2494f8405"
-
-do_configure:prepend () {
-mkdir -p ${S}/build-aux
-}
+SRC_URI[sha256sum] = 
"9ee7ac57b188d391cead705d3596a6d3240341786475149db297782a52269aa5"
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199692): 
https://lists.openembedded.org/g/openembedded-core/message/199692
Mute This Topic: https://lists.openembedded.org/mt/106239534/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 2/2] ofono: upgrade 2.4 -> 2.7

2024-05-22 Thread Martin Hundeb?ll

Remove 0001-mbim-add-an-optional-TEMP_FAILURE_RETRY-macro-copy.patch as
TEMP_FAILURE_RETRY usage has been removed in upstream commit 765c6655
("treewide: Use L_TFR macro").

Remove 0002-mbim-Fix-build-with-ell-0.39-by-restoring-unlikely-m.patch
as likely()/unlikely() has been removed in upstream commit dbbbebf9
("mbimmodem: Remove usage of likely and unlikely").

Remove the do_configure:prepend() that purges the bundled ell directory,
as it isn't needed when passing --enable-external-ell in EXTRA_OECONF.

Signed-off-by: Martin Hundebøll 
---
 ...tional-TEMP_FAILURE_RETRY-macro-copy.patch | 36 ---
 ...ith-ell-0.39-by-restoring-unlikely-m.patch | 28 ---
 .../ofono/{ofono_2.4.bb => ofono_2.7.bb}  |  9 +
 3 files changed, 1 insertion(+), 72 deletions(-)
 delete mode 100644 
meta/recipes-connectivity/ofono/ofono/0001-mbim-add-an-optional-TEMP_FAILURE_RETRY-macro-copy.patch
 delete mode 100644 
meta/recipes-connectivity/ofono/ofono/0002-mbim-Fix-build-with-ell-0.39-by-restoring-unlikely-m.patch
 rename meta/recipes-connectivity/ofono/{ofono_2.4.bb => ofono_2.7.bb} (82%)

diff --git 
a/meta/recipes-connectivity/ofono/ofono/0001-mbim-add-an-optional-TEMP_FAILURE_RETRY-macro-copy.patch
 
b/meta/recipes-connectivity/ofono/ofono/0001-mbim-add-an-optional-TEMP_FAILURE_RETRY-macro-copy.patch
deleted file mode 100644
index 8a5a300adc..00
--- 
a/meta/recipes-connectivity/ofono/ofono/0001-mbim-add-an-optional-TEMP_FAILURE_RETRY-macro-copy.patch
+++ /dev/null
@@ -1,36 +0,0 @@
-From 22b52db4842611ac31a356f023fc09595384e2ad Mon Sep 17 00:00:00 2001
-From: Khem Raj 
-Date: Thu, 23 May 2019 18:11:22 -0700
-Subject: [PATCH] mbim: add an optional TEMP_FAILURE_RETRY macro copy
-
-Fixes build on musl which does not provide this macro
-
-Upstream-Status: Submitted 
[https://lists.ofono.org/pipermail/ofono/2019-May/019370.html]
-Signed-off-by: Khem Raj 

- drivers/mbimmodem/mbim-private.h | 9 +
- 1 file changed, 9 insertions(+)
-
-diff --git a/drivers/mbimmodem/mbim-private.h 
b/drivers/mbimmodem/mbim-private.h
-index e159235..51693ea 100644
 a/drivers/mbimmodem/mbim-private.h
-+++ b/drivers/mbimmodem/mbim-private.h
-@@ -21,6 +21,15 @@
- 
- #define align_len(len, boundary) (((len)+(boundary)-1) & ~((boundary)-1))
- 
-+#ifndef TEMP_FAILURE_RETRY
-+#define TEMP_FAILURE_RETRY(expression) ({ \
-+  __typeof(expression) __result;  \
-+  do {\
-+__result = (expression);  \
-+  } while (__result == -1 && errno == EINTR); \
-+  __result; })
-+#endif
-+
- enum mbim_control_message {
-   MBIM_OPEN_MSG = 0x1,
-   MBIM_CLOSE_MSG = 0x2,
--- 
-2.21.0
-
diff --git 
a/meta/recipes-connectivity/ofono/ofono/0002-mbim-Fix-build-with-ell-0.39-by-restoring-unlikely-m.patch
 
b/meta/recipes-connectivity/ofono/ofono/0002-mbim-Fix-build-with-ell-0.39-by-restoring-unlikely-m.patch
deleted file mode 100644
index 3655b3fd66..00
--- 
a/meta/recipes-connectivity/ofono/ofono/0002-mbim-Fix-build-with-ell-0.39-by-restoring-unlikely-m.patch
+++ /dev/null
@@ -1,28 +0,0 @@
-From 76e4054801350ebd4a44057379431a33d460ad0f Mon Sep 17 00:00:00 2001
-From: Martin Jansa 
-Date: Wed, 21 Apr 2021 11:01:34 +
-Subject: [PATCH] mbim: Fix build with ell-0.39 by restoring unlikely macro
- from ell/util.h
-
-Upstream-Status: Pending
-
-Signed-off-by: Martin Jansa 

- drivers/mbimmodem/mbim-private.h | 4 
- 1 file changed, 4 insertions(+)
-
-diff --git a/drivers/mbimmodem/mbim-private.h 
b/drivers/mbimmodem/mbim-private.h
-index 51693eae..d917312c 100644
 a/drivers/mbimmodem/mbim-private.h
-+++ b/drivers/mbimmodem/mbim-private.h
-@@ -30,6 +30,10 @@
-   __result; })
- #endif
- 
-+/* used to be part of ell/util.h before 0.39:
-+   
https://git.kernel.org/pub/scm/libs/ell/ell.git/commit/?id=2a682421b06e41c45098217a686157f576847021
 */
-+#define unlikely(x) __builtin_expect(!!(x), 0)
-+
- enum mbim_control_message {
-   MBIM_OPEN_MSG = 0x1,
-   MBIM_CLOSE_MSG = 0x2,
diff --git a/meta/recipes-connectivity/ofono/ofono_2.4.bb 
b/meta/recipes-connectivity/ofono/ofono_2.7.bb
similarity index 82%
rename from meta/recipes-connectivity/ofono/ofono_2.4.bb
rename to meta/recipes-connectivity/ofono/ofono_2.7.bb
index 5e1e5f3b6a..bc1f3efd04 100644
--- a/meta/recipes-connectivity/ofono/ofono_2.4.bb
+++ b/meta/recipes-connectivity/ofono/ofono_2.7.bb
@@ -10,10 +10,8 @@ DEPENDS = "dbus glib-2.0 udev mobile-broadband-provider-info 
ell"
 SRC_URI = "\
 ${KERNELORG_MIRROR}/linux/network/${BPN}/${BP}.tar.xz \
 file://ofono \
-file://0001-mbim-add-an-optional-TEMP_FAILURE_RETRY-macro-copy.patch \
-file://0002-mbim-Fix-build-with-ell-0.39-by-restoring-unlikely-m.patch \
 "
-SRC_URI[sha256sum] = 
"93580adc1afd1890dc516efb069de0c5cdfef014415256ddfb28ab172df2d11d"
+SRC_URI[sha256sum] = 
"dabf6ef06b94beaad65253200abe3887046a4e722f4fe373c4264f357ae47ad3"
 
 inherit autotools pkgconfig update-rc.d systemd

Re: [OE-core] [PATCH 3/5] ninja: build modified version with GNU Make jobserver support

2024-05-18 Thread Martin Hundeb?ll

Hi Randy,

On Mon, 2024-04-08 at 17:39 -0400, Randy MacLeod wrote:
>  On 2024-04-03 3:08 p.m., Martin Hundebøll wrote:
> > On Wed, 2024-04-03 at 17:58 +0200, Alexander Kanavin wrote:
> > > We can't carry this mass of patches. It has to land upstream
> > > first.
> > > 
> > Understandable.
> > > 
> > > Are you able/willing to drive the upstream pull request to
> > > completion?
> > > 
> > No, I don't think so. I'm not familiar with the ninja codebase, and
> > the
> > pull requests has been lingering for too long.
> > 
> > Another option might be to propose adding the fifo feature to the
> > kitware fork of ninja:
> > https://github.com/Kitware/ninja
> > 
> > They have forked ninja to get the jobserver feature in, but haven't
> > taken the fifo part so far.
> > 
> > The kitware fork is also used by buildroot:
> > https://gitlab.com/buildroot.org/buildroot/-/blob/master/package/ninja/ninja.mk
> > 
> I see that v2 has switched to kitware, which is fine in the short
> run.
> 
>  I'm just back from vacation. I plan to test this patchset out.
> 
>  I can spend some time convincing upstream ninja that a solution is
> needed for our use-case and 
>  that perhaps the jobsever approach and the kitware fork is a good
> starting point. 
> 
>  Of course, I can't promise to "drive it to completion" but I'll
> hound the upstream devs for months or even years!
> 
> Thanks for re-submitting the patchset Martin!
>  

I got hit by a strike of motivation and submitted a new attempt on
jobserver support to ninja:
https://github.com/ninja-build/ninja/pull/2450

First review from the ninja maintainer seems positive. I'll resubmit
the my patchset to OE once the ninja PR is (hopefully) merged.

// Martin

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199530): 
https://lists.openembedded.org/g/openembedded-core/message/199530
Mute This Topic: https://lists.openembedded.org/mt/105304230/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 05/10] base: Switch UNPACKDIR to a subdir of WORKDIR

2024-05-16 Thread Martin Hundeb?ll

On Wed, 2024-05-15 at 12:56 +0100, Richard Purdie wrote:
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index b2c500d8739..75c850760f6 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -405,7 +405,7 @@ STAMP =
> "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/${PV}"
>  STAMPCLEAN = "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/*-*"
>  BASE_WORKDIR ?= "${TMPDIR}/work"
>  WORKDIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/${PV}"
> -UNPACKDIR ??= "${WORKDIR}"
> +UNPACKDIR ??= "${WORKDIR}/sources-unpack"
>  T = "${WORKDIR}/temp"
>  D = "${WORKDIR}/image"
>  S = "${WORKDIR}/${BP}"

Why not use

  UNPACKDIR ??= "${WORKDIR}/sources"

like it's done in the individual recipes?


And shouldn't we do 

  S = ?? "${UNPACKDIR}/${BP}"

also?

// Martin

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199439): 
https://lists.openembedded.org/g/openembedded-core/message/199439
Mute This Topic: https://lists.openembedded.org/mt/106112374/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATH][master][scarthgap] classes: image_types: apply EXTRA_IMAGECMD:squashfs* in oe_mksquashfs()

2024-05-06 Thread Martin Hundeb?ll

Since commit c991f9d6031 ("image_types: Set SOURCE_DATE_EPOCH for squashfs"),
I assume, the EXTRA_IMAGECMD:squashfs* variable(s) has been ignored.
This is due to the override magic, which isn't applied to functions
called by IMAGE_CMD:, but only to the IMAGE_CMD: itself.

Other image types (e.g. ext*) works around this by passing the
EXTRA_IMAGECMD variable as an argument to the called function.

To do the same for oe_mksquashfs(), the number of mandatory arguments is
fixed to one (with a little logic to handle the zstd filename). This
allows passing ${EXTRA_IMAGECMD} as an argument to oe_mksquashfs(),
which makes the variable functional again.

Signed-off-by: Martin Hundebøll 
---
 meta/classes-recipe/image_types.bbclass | 20 
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/meta/classes-recipe/image_types.bbclass 
b/meta/classes-recipe/image_types.bbclass
index 913cb8788c..b4a83ae284 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -112,18 +112,22 @@ IMAGE_CMD:btrfs () {
 }
 
 oe_mksquashfs () {
-local comp=$1
-local suffix=$2
+local comp=$1; shift
+local extra_imagecmd=$@
+
+if [ "$comp" = "zstd" ]; then
+suffix="zst"
+fi
 
 # Use the bitbake reproducible timestamp instead of the hardcoded squashfs 
one
 export SOURCE_DATE_EPOCH=$(stat -c '%Y' ${IMAGE_ROOTFS})
-mksquashfs ${IMAGE_ROOTFS} 
${IMGDEPLOYDIR}/${IMAGE_NAME}.squashfs${comp:+-}${suffix:-$comp} 
${EXTRA_IMAGECMD} -noappend ${comp:+-comp }$comp
+mksquashfs ${IMAGE_ROOTFS} 
${IMGDEPLOYDIR}/${IMAGE_NAME}.squashfs${comp:+-}${suffix:-$comp} -noappend 
${comp:+-comp }$comp $extra_imagecmd
 }
-IMAGE_CMD:squashfs = "oe_mksquashfs"
-IMAGE_CMD:squashfs-xz = "oe_mksquashfs xz"
-IMAGE_CMD:squashfs-lzo = "oe_mksquashfs lzo"
-IMAGE_CMD:squashfs-lz4 = "oe_mksquashfs lz4"
-IMAGE_CMD:squashfs-zst = "oe_mksquashfs zstd zst"
+IMAGE_CMD:squashfs = "oe_mksquashfs '' ${EXTRA_IMAGECMD}"
+IMAGE_CMD:squashfs-xz = "oe_mksquashfs xz ${EXTRA_IMAGECMD}"
+IMAGE_CMD:squashfs-lzo = "oe_mksquashfs lzo ${EXTRA_IMAGECMD}"
+IMAGE_CMD:squashfs-lz4 = "oe_mksquashfs lz4 ${EXTRA_IMAGECMD}"
+IMAGE_CMD:squashfs-zst = "oe_mksquashfs zstd ${EXTRA_IMAGECMD}"
 
 IMAGE_CMD:erofs = "mkfs.erofs ${EXTRA_IMAGECMD} 
${IMGDEPLOYDIR}/${IMAGE_NAME}.erofs ${IMAGE_ROOTFS}"
 IMAGE_CMD:erofs-lz4 = "mkfs.erofs -zlz4 ${EXTRA_IMAGECMD} 
${IMGDEPLOYDIR}/${IMAGE_NAME}.erofs-lz4 ${IMAGE_ROOTFS}"
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199041): 
https://lists.openembedded.org/g/openembedded-core/message/199041
Mute This Topic: https://lists.openembedded.org/mt/105938011/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [scarthgap v2] systemd: sed ROOT_HOME only if sysusers PACKAGECONFIG is set

2024-05-02 Thread Martin Hundeb?ll

From: Christian Bräuner Sørensen 

Fixes a bug introducted in ebafe46379 systemd: upgrade to 255.1.

Besides updating systemd, that commit also made other changes. One of them
being when to perform the replacement in order to fix ROOT_HOME.

Previously, that happened on a configure prefunc and on
${S}/sysusers.d/basic.conf.in.
Now it happens in install and on image/usr/lib/sysusers.d/basic.conf.

However, that file is not present if sysusers is not in PACKAGECONFIG,
since that file in that case is not installed hence resulting in:
sed: can't read /image/usr/lib/sysusers.d/basic.conf: No such 
file or directory

Previously, in the case of sysusers not being in PACKAGECONFIG, that was a
"silent error" since the replacement was done but the file was not really
used since the file was not installed.

Signed-off-by: Christian Bräuner Sørensen 
Signed-off-by: Richard Purdie 
Signed-off-by: Martin Hundebøll 
---

Added Signed-off-by.

 meta/recipes-core/systemd/systemd_255.4.bb | 18 ++
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_255.4.bb 
b/meta/recipes-core/systemd/systemd_255.4.bb
index e7498c802d..f58a1bc2b6 100644
--- a/meta/recipes-core/systemd/systemd_255.4.bb
+++ b/meta/recipes-core/systemd/systemd_255.4.bb
@@ -271,14 +271,16 @@ WATCHDOG_TIMEOUT ??= "60"
 
 do_install() {
meson_do_install
-   # Change the root user's home directory in /lib/sysusers.d/basic.conf.
-   # This is done merely for backward compatibility with previous systemd 
recipes.
-   # systemd hardcodes root user's HOME to be "/root". Changing to use 
other values
-   # may have unexpected runtime behaviors.
-   if [ "${ROOT_HOME}" != "/root" ]; then
-   bbwarn "Using ${ROOT_HOME} as root user's home directory is not 
fully supported by systemd"
-   sed -i -e 's#/root#${ROOT_HOME}#g' 
${D}${exec_prefix}/lib/sysusers.d/basic.conf
-   fi
+   if ${@bb.utils.contains('PACKAGECONFIG', 'sysusers', 'true', 'false', 
d)}; then
+# Change the root user's home directory in 
/lib/sysusers.d/basic.conf.
+# This is done merely for backward compatibility with previous 
systemd recipes.
+# systemd hardcodes root user's HOME to be "/root". Changing to 
use other values
+# may have unexpected runtime behaviors.
+if [ "${ROOT_HOME}" != "/root" ]; then
+bbwarn "Using ${ROOT_HOME} as root user's home directory 
is not fully supported by systemd"
+sed -i -e 's#/root#${ROOT_HOME}#g' 
${D}${exec_prefix}/lib/sysusers.d/basic.conf
+fi
+fi
install -d ${D}/${base_sbindir}
if ${@bb.utils.contains('PACKAGECONFIG', 'serial-getty-generator', 
'false', 'true', d)}; then
# Provided by a separate recipe
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198909): 
https://lists.openembedded.org/g/openembedded-core/message/198909
Mute This Topic: https://lists.openembedded.org/mt/105859601/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [scarthgap] systemd: sed ROOT_HOME only if sysusers PACKAGECONFIG is set

2024-05-02 Thread Martin Hundeb?ll

From: Christian Bräuner Sørensen 

Fixes a bug introducted in ebafe46379 systemd: upgrade to 255.1.

Besides updating systemd, that commit also made other changes. One of them
being when to perform the replacement in order to fix ROOT_HOME.

Previously, that happened on a configure prefunc and on
${S}/sysusers.d/basic.conf.in.
Now it happens in install and on image/usr/lib/sysusers.d/basic.conf.

However, that file is not present if sysusers is not in PACKAGECONFIG,
since that file in that case is not installed hence resulting in:
sed: can't read /image/usr/lib/sysusers.d/basic.conf: No such 
file or directory

Previously, in the case of sysusers not being in PACKAGECONFIG, that was a
"silent error" since the replacement was done but the file was not really
used since the file was not installed.

Signed-off-by: Christian Bräuner Sørensen 
Signed-off-by: Richard Purdie 
---
 meta/recipes-core/systemd/systemd_255.4.bb | 18 ++
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_255.4.bb 
b/meta/recipes-core/systemd/systemd_255.4.bb
index e7498c802d..f58a1bc2b6 100644
--- a/meta/recipes-core/systemd/systemd_255.4.bb
+++ b/meta/recipes-core/systemd/systemd_255.4.bb
@@ -271,14 +271,16 @@ WATCHDOG_TIMEOUT ??= "60"
 
 do_install() {
meson_do_install
-   # Change the root user's home directory in /lib/sysusers.d/basic.conf.
-   # This is done merely for backward compatibility with previous systemd 
recipes.
-   # systemd hardcodes root user's HOME to be "/root". Changing to use 
other values
-   # may have unexpected runtime behaviors.
-   if [ "${ROOT_HOME}" != "/root" ]; then
-   bbwarn "Using ${ROOT_HOME} as root user's home directory is not 
fully supported by systemd"
-   sed -i -e 's#/root#${ROOT_HOME}#g' 
${D}${exec_prefix}/lib/sysusers.d/basic.conf
-   fi
+   if ${@bb.utils.contains('PACKAGECONFIG', 'sysusers', 'true', 'false', 
d)}; then
+# Change the root user's home directory in 
/lib/sysusers.d/basic.conf.
+# This is done merely for backward compatibility with previous 
systemd recipes.
+# systemd hardcodes root user's HOME to be "/root". Changing to 
use other values
+# may have unexpected runtime behaviors.
+if [ "${ROOT_HOME}" != "/root" ]; then
+bbwarn "Using ${ROOT_HOME} as root user's home directory 
is not fully supported by systemd"
+sed -i -e 's#/root#${ROOT_HOME}#g' 
${D}${exec_prefix}/lib/sysusers.d/basic.conf
+fi
+fi
install -d ${D}/${base_sbindir}
if ${@bb.utils.contains('PACKAGECONFIG', 'serial-getty-generator', 
'false', 'true', d)}; then
# Provided by a separate recipe
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198908): 
https://lists.openembedded.org/g/openembedded-core/message/198908
Mute This Topic: https://lists.openembedded.org/mt/105859569/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 4/5] qemu: enable parallel builds when using the jobserver class

2024-04-04 Thread Martin Hundeb?ll

If the jobserver class is enabled, the PARALLEL_MAKE variable is unset in
favor of configuring a shared jobserver in the MAKEFLAGS variable. However,
the qemu makefile translates the missing `-j` argument to `-j1` when
calling into meson / ninja. Add a patch to make the qemu makefile
consider the --jobserver-auth option too.

Signed-off-by: Martin Hundebøll 
---
 meta/recipes-devtools/qemu/qemu.inc   |  1 +
 ...e-jobserver-auth-argument-when-calli.patch | 37 +++
 2 files changed, 38 insertions(+)
 create mode 100644 
meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch

diff --git a/meta/recipes-devtools/qemu/qemu.inc 
b/meta/recipes-devtools/qemu/qemu.inc
index 4501f84c2b..1f86bf5a44 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -39,6 +39,7 @@ SRC_URI = "https://download.qemu.org/${BPN}-${PV}.tar.xz \
file://0003-linux-user-Add-strace-for-shmat.patch \
file://0004-linux-user-Rewrite-target_shmat.patch \

file://0005-tests-tcg-Check-that-shmat-does-not-break-proc-self-.patch \
+   
file://0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch \
file://CVE-2023-6683.patch \
file://qemu-guest-agent.init \
file://qemu-guest-agent.udev \
diff --git 
a/meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch
 
b/meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch
new file mode 100644
index 00..0dbc32eed0
--- /dev/null
+++ 
b/meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch
@@ -0,0 +1,37 @@
+From 730530c5f01e00cdc3754ebb8f3d7ff995f3376e Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Martin=20Hundeb=C3=B8ll?= 
+Date: Thu, 21 Sep 2023 10:57:45 +0200
+Subject: [PATCH] Makefile: preserve --jobserver-auth argument when calling
+ ninja
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Qemu wraps its call to ninja in a Makefile. Since ninja, as opposed to
+make, utilizes all CPU cores by default, the qemu Makefile translates
+the absense of a `-jN` argument into `-j1`. This breaks jobserver
+functionality, so update the -jN mangling to take the --jobserver-auth
+argument into considerationa too.
+
+Signed-off-by: Martin Hundebøll 
+Upstream-Status: Submitted 
[https://lists.nongnu.org/archive/html/qemu-devel/2024-04/msg00172.html]
+---
+ Makefile | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Makefile b/Makefile
+index 8f36990335..183756018f 100644
+--- a/Makefile
 b/Makefile
+@@ -142,7 +142,7 @@ MAKE.k = $(findstring k,$(firstword $(filter-out 
--%,$(MAKEFLAGS
+ MAKE.q = $(findstring q,$(firstword $(filter-out --%,$(MAKEFLAGS
+ MAKE.nq = $(if $(word 2, $(MAKE.n) $(MAKE.q)),nq)
+ NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \
+-$(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS \
++$(or $(filter -l% -j%, $(MAKEFLAGS)), $(if $(filter 
--jobserver-auth=%, $(MAKEFLAGS)),, -j1)) \
+ -d keepdepfile
+ ninja-cmd-goals = $(or $(MAKECMDGOALS), all)
+ ninja-cmd-goals += $(foreach g, $(MAKECMDGOALS), $(.ninja-goals.$g))
+-- 
+2.44.0
+
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197960): 
https://lists.openembedded.org/g/openembedded-core/message/197960
Mute This Topic: https://lists.openembedded.org/mt/105326972/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 5/5] contrib: add python service and systemd unit to run shared jobserver

2024-04-04 Thread Martin Hundeb?ll

For CI setups that might end up building multiple yocto builds in
parallel, a shared jobserver can reduce the total load of the system.
Setting up such a jobserver is simple, but it does require a process
hanging around to keep the jobserver fifo open (to avoid blocking token
requests).

Add a simple python script that creates such a jobserver fifo and waits
forever. Also add a systemd unit file to start the python service at
boot.

The systemd unit can be installed in $HOME/.config/systemd/user/, but
one might need to add a droplet config (i.e. `systemctl --user edit
jobserver.service`) to setup the PYTHONPATH variable to make the python
script loadable.

Signed-off-by: Martin Hundebøll 
---
 contrib/jobserver/jobserver.py  | 78 +
 contrib/jobserver/jobserver.service | 10 
 2 files changed, 88 insertions(+)
 create mode 100644 contrib/jobserver/jobserver.py
 create mode 100644 contrib/jobserver/jobserver.service

diff --git a/contrib/jobserver/jobserver.py b/contrib/jobserver/jobserver.py
new file mode 100644
index 00..52f9711277
--- /dev/null
+++ b/contrib/jobserver/jobserver.py
@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+
+from pathlib import Path
+from threading import Event
+import argparse
+import os
+import shutil
+import signal
+
+resumed = Event()
+runtime_dir = os.environ.get("XDG_RUNTIME_DIR", "/run")
+
+def signal_handler(signum, _frame):
+"""Wait for an external signal to exit the process gracefully."""
+resumed.set()
+
+
+def main(path, user, group, mode, jobs):
+"""Setup a fifo to use as jobserver shared between builds."""
+try:
+path.unlink(missing_ok=True)
+os.mkfifo(path)
+shutil.chown(path, user, group)
+os.chmod(path, mode)
+except (FileNotFoundError, PermissionError) as exc:
+raise SystemExit(f"failed to create fifo: {path}: {exc.strerror}")
+
+print(f"jobserver: {path}: {jobs} jobs")
+fifo = os.open(path, os.O_RDWR)
+os.write(fifo, b"+" * jobs)
+
+print("jobserver: ready; waiting indefinitely")
+signal.signal(signal.SIGTERM, signal_handler)
+signal.signal(signal.SIGINT, signal_handler)
+resumed.wait()
+
+print("jobserver: exiting")
+path.unlink()
+os.close(fifo)
+
+
+if __name__ == "__main__":
+parser = argparse.ArgumentParser(
+prog='Make jobserver',
+description='Simple application to instantiate a jobserver fifo and 
hang around',
+)
+parser.add_argument(
+"--mode",
+help="Permission to apply to jobserver fifo",
+type=lambda v: int(v, 8),
+default=0o0666,
+)
+parser.add_argument(
+"--user",
+help="Username or id to assign ownership of fifo to",
+default=os.getuid(),
+)
+parser.add_argument(
+"--group",
+help="Groupname or id to assign ownership of fifo to",
+default=os.getgid(),
+)
+parser.add_argument(
+"path",
+help="Path to jobserver fifo",
+type=Path,
+nargs='?',
+default=f"{runtime_dir}/jobserver",
+)
+parser.add_argument(
+"jobs",
+help="Number of tokens to load jobserver with",
+type=int,
+nargs='?',
+default=os.cpu_count(),
+)
+args = parser.parse_args()
+main(args.path, args.user, args.group, args.mode, args.jobs)
diff --git a/contrib/jobserver/jobserver.service 
b/contrib/jobserver/jobserver.service
new file mode 100644
index 00..bbc7167ac0
--- /dev/null
+++ b/contrib/jobserver/jobserver.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=Shared jobserver fifo
+
+[Service]
+Type=simple
+Environment=PYTHONUNBUFFERED=1
+ExecStart=python jobserver.py
+
+[Install]
+WantedBy=multi-user.target
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197959): 
https://lists.openembedded.org/g/openembedded-core/message/197959
Mute This Topic: https://lists.openembedded.org/mt/105326971/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 2/5] scripts: build-env: allow passing JOBSERVER_FIFO from environment

2024-04-04 Thread Martin Hundeb?ll

Sharing a common jobserver fifo between multiple (containerized) builds
is much easier, if an administrator can configure said jobserver fifo
path in the environment. Append the JOBSERVER_FIFO variable name to the
list of variables configurable through the environment.

Signed-off-by: Martin Hundebøll 
---
 scripts/oe-buildenv-internal | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index 2fdb19565a..c8e67ffb8f 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -112,7 +112,7 @@ HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY 
ftps_proxy ALL_PROXY \
 all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \
 SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND \
 SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA 
BB_SETSCENE_ENFORCE \
-BB_LOGCONFIG"
+BB_LOGCONFIG JOBSERVER_FIFO"
 
 BB_ENV_PASSTHROUGH_ADDITIONS="$(echo $BB_ENV_PASSTHROUGH_ADDITIONS 
$BB_ENV_PASSTHROUGH_ADDITIONS_OE | tr ' ' '\n' | LC_ALL=C sort --unique | tr 
'\n' ' ')"
 
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197958): 
https://lists.openembedded.org/g/openembedded-core/message/197958
Mute This Topic: https://lists.openembedded.org/mt/105326970/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 1/5] classes: jobserver: support gnu make fifo jobserver

2024-04-04 Thread Martin Hundeb?ll

Add a class to implement the gnu make fifo style jobserver. The class
can be activated by simply adding an `INHERIT += "jobserver"` to the
local configuration.

Furthermore, one can configure an external jobserver (i.e. a server
shared between multiple builds), by configuring the `JOBSERVER_FIFO`
variable to point at an existing jobserver fifo.

The jobserver class uses the fifo style jobserver, which doesn't require
passing open file descriptors around. It does, however, require
make-4.4, which isn't available in common distro yet. To work around
this, the class makes all recipes (except make and its dependencies
itself) depend on `virtual/make-native`.

Signed-off-by: Martin Hundebøll 
---
 meta/classes-global/jobserver.bbclass | 87 +++
 meta/conf/bitbake.conf|  2 +-
 2 files changed, 88 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes-global/jobserver.bbclass

diff --git a/meta/classes-global/jobserver.bbclass 
b/meta/classes-global/jobserver.bbclass
new file mode 100644
index 00..3866b473e6
--- /dev/null
+++ b/meta/classes-global/jobserver.bbclass
@@ -0,0 +1,87 @@
+JOBSERVER_FIFO ?= ""
+JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of 
creating a per-build server."
+
+JOBSERVER_IGNORE ?= ""
+JOBSERVER_IGNORE[doc] = "Space separated list of packages that shouldn't be 
configured to use the jobserver feature."
+
+addhandler jobserver_setup_fifo
+jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
+
+python jobserver_setup_fifo() {
+# don't setup a per-build fifo, if an external one is configured
+if d.getVar("JOBSERVER_FIFO"):
+return
+
+# don't use a job-server if no parallelism is configured
+jobs = oe.utils.parallel_make(d)
+if jobs in (None, 1):
+return
+
+# reduce jobs by one as a token has implicitly been handed to the
+# process requesting tokens
+jobs -= 1
+
+fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
+
+# an old fifo might be lingering; remove it
+if os.path.exists(fifo):
+os.remove(fifo)
+
+# create a new fifo to use for communicating tokens
+os.mkfifo(fifo)
+
+# fill the fifo with the number of tokens to hand out
+wfd = os.open(fifo, os.O_RDWR)
+written = os.write(wfd, b"+" * jobs)
+if written != (jobs):
+bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
+
+# configure the per-build fifo path to use
+d.setVar("JOBSERVER_FIFO", fifo)
+}
+
+python () {
+# don't configure the fifo if none is defined
+fifo = d.getVar("JOBSERVER_FIFO")
+if not fifo:
+return
+
+# don't configure the fifo if the package wants to ignore it
+if d.getVar("PN") in (d.getVar("JOBSERVER_IGNORE") or "").split():
+return
+
+# avoid making make-native or its dependencies depend on make-native itself
+if d.getVar("PN") in (
+"make-native",
+"libtool-native",
+"pkgconfig-native",
+"automake-native",
+"autoconf-native",
+"m4-native",
+"texinfo-dummy-native",
+"gettext-minimal-native",
+"quilt-native",
+"gnu-config-native",
+):
+return
+
+# don't make unwilling recipes depend on make-native
+if d.getVar('INHIBIT_DEFAULT_DEPS', False):
+return
+
+# make other recipes depend on make-native to make sure it is new enough to
+# support the --jobserver-auth=fifo: syntax (from make-4.4 and 
onwards)
+d.appendVar("DEPENDS", " virtual/make-native")
+
+# disable the "-j " flag, as that overrides the jobserver fifo tokens
+d.setVar("PARALLEL_MAKE", "")
+d.setVar("PARALLEL_MAKEINST", "")
+
+# set and export the jobserver in the environment
+d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
+d.setVarFlag("MAKEFLAGS", "export", "1")
+
+# ignore the jobserver argument part of MAKEFLAGS in the hash, as that
+# shouldn't change the build output
+d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "| 
--jobserver-auth=fifo:" + fifo)
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6f180d18b0..23a016b31e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -960,7 +960,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH 
BBPATH BBSERVER DL_DI
 BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot 
DEPLOY_DIR \
 SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
 SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES 
\
-OMP_NUM_THREADS BB_CURRENTTASK"
+OMP_NUM_THREADS BB_CURRENTTASK JOBSERVER_FIFO"
 BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS 
BUILDHISTORY_DIR \
 SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS"
 BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME

[OE-core] [PATCH v2 3/5] ninja: build modified version with GNU Make jobserver support

2024-04-04 Thread Martin Hundeb?ll

Ninja doesn't (yet) support the GNU Make jobserver out of the box, but
there is a pull request adding that support[1]. Since that pull request
(and its derived three-part pull requests) seem to be ignored by
upstream, kitware (creator/maintainer of cmake) has created a fork[2]
only to carry the jobserver patches. Change the source uri to point at
the kitware fork of ninja, and add two patches from the original pull
request to also support the new-style fifo jobserver feature.

Note that the kitware fork of ninja is also used by buildroot[3].

[1] https://github.com/ninja-build/ninja/pull/2263
[2] https://github.com/Kitware/ninja
[3] 
https://gitlab.com/buildroot.org/buildroot/-/blob/master/package/ninja/ninja.mk

Signed-off-by: Martin Hundebøll 
---
 ...ename-TokenPool-Setup-to-SetupClient.patch | 113 
 ...-jobserver-fifo-style-client-support.patch | 271 ++
 meta/recipes-devtools/ninja/ninja_1.11.1.bb   |   8 +-
 3 files changed, 390 insertions(+), 2 deletions(-)
 create mode 100644 
meta/recipes-devtools/ninja/files/0001-Rename-TokenPool-Setup-to-SetupClient.patch
 create mode 100644 
meta/recipes-devtools/ninja/files/0002-Add-GNU-make-jobserver-fifo-style-client-support.patch

diff --git 
a/meta/recipes-devtools/ninja/files/0001-Rename-TokenPool-Setup-to-SetupClient.patch
 
b/meta/recipes-devtools/ninja/files/0001-Rename-TokenPool-Setup-to-SetupClient.patch
new file mode 100644
index 00..a503f8c75f
--- /dev/null
+++ 
b/meta/recipes-devtools/ninja/files/0001-Rename-TokenPool-Setup-to-SetupClient.patch
@@ -0,0 +1,113 @@
+From f5642d8b49688dfc84679451b531d92f3b6e7cb0 Mon Sep 17 00:00:00 2001
+From: Stefan Becker 
+Date: Sat, 15 Dec 2018 19:29:42 +0200
+Subject: [PATCH 1/2] Rename TokenPool::Setup() to SetupClient()
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Make space to add new API to set up token pool master.
+
+Signed-off-by: Martin Hundebøll 
+Upstream-Status: Submitted [https://github.com/Kitware/ninja/pull/2]
+---
+ src/build.cc  | 6 +++---
+ src/subprocess_test.cc| 3 ++-
+ src/tokenpool-gnu-make.cc | 6 +++---
+ src/tokenpool-gnu-make.h  | 3 ++-
+ src/tokenpool.h   | 3 ++-
+ src/tokenpool_test.cc | 2 +-
+ 6 files changed, 13 insertions(+), 10 deletions(-)
+
+diff --git a/src/build.cc b/src/build.cc
+index 53e4405..d8a6dff 100644
+--- a/src/build.cc
 b/src/build.cc
+@@ -474,9 +474,9 @@ struct RealCommandRunner : public CommandRunner {
+ RealCommandRunner::RealCommandRunner(const BuildConfig& config) : 
config_(config) {
+   max_load_average_ = config.max_load_average;
+   if ((tokens_ = TokenPool::Get()) != NULL) {
+-if (!tokens_->Setup(config_.parallelism_from_cmdline,
+-config_.verbosity == BuildConfig::VERBOSE,
+-max_load_average_)) {
++if (!tokens_->SetupClient(config_.parallelism_from_cmdline,
++  config_.verbosity == BuildConfig::VERBOSE,
++  max_load_average_)) {
+   delete tokens_;
+   tokens_ = NULL;
+ }
+diff --git a/src/subprocess_test.cc b/src/subprocess_test.cc
+index eddc110..222b59b 100644
+--- a/src/subprocess_test.cc
 b/src/subprocess_test.cc
+@@ -40,7 +40,8 @@ struct TestTokenPool : public TokenPool {
+   void Reserve() {}
+   void Release() {}
+   void Clear()   {}
+-  bool Setup(bool ignore_unused, bool verbose, double& max_load_average) { 
return false; }
++  bool SetupClient(bool ignore_unused, bool verbose,
++   double& max_load_average) { return false; }
+ 
+ #ifdef _WIN32
+   bool _token_available;
+diff --git a/src/tokenpool-gnu-make.cc b/src/tokenpool-gnu-make.cc
+index 60e0552..6fb72a6 100644
+--- a/src/tokenpool-gnu-make.cc
 b/src/tokenpool-gnu-make.cc
+@@ -28,9 +28,9 @@ GNUmakeTokenPool::GNUmakeTokenPool() : available_(1), 
used_(0) {
+ GNUmakeTokenPool::~GNUmakeTokenPool() {
+ }
+ 
+-bool GNUmakeTokenPool::Setup(bool ignore,
+- bool verbose,
+- double& max_load_average) {
++bool GNUmakeTokenPool::SetupClient(bool ignore,
++   bool verbose,
++   double& max_load_average) {
+   const char* value = GetEnv("MAKEFLAGS");
+   if (!value)
+ return false;
+diff --git a/src/tokenpool-gnu-make.h b/src/tokenpool-gnu-make.h
+index c94cca5..f4ab8d7 100644
+--- a/src/tokenpool-gnu-make.h
 b/src/tokenpool-gnu-make.h
+@@ -24,7 +24,8 @@ struct GNUmakeTokenPool : public TokenPool {
+   virtual void Reserve();
+   virtual void Release();
+   virtual void Clear();
+-  virtual bool Setup(bool ignore, bool verbose, double& max_load_average);
++  virtual bool SetupClient(bool ignore, bool verbose,
++   double& max_load_average);
+ 
+   // platform specific implementation
+   virtual const char* GetEnv(const char* name) = 0;
+diff --git a/src/tokenpool.h b/src/tokenpool.h

[OE-core] [PATCH v2 0/5] Jobserver support

2024-04-04 Thread Martin Hundeb?ll

The parallelism of bitbake easily uses every available core on the build
host. But since every task is run with the same number of parallel
threads/processes, multiple tasks might load the CPU excessively, which
in turn slows down the build due to scheduling overhead.

This patch series adds a class that creates a fifo filled with
PARALLEL_MAKE tokens. The path to the created fifo is then configured in
the MAKEFLAGS environment variable, which is read by make and a patched
ninja (and gcc if doing lto).

The benefits from using the jobserver depends on the set of executed
tasks: running multiple large do_compile tasks simultaneously benefits
more than multiple do_fetch tasks. A simple test building the following
tasks (and all their dependencies) yields a ~5% improvement in build
time (20:20 -> 19:20):

  nodejs-native
  rust-llvm-native
  rust-native
  linux-yocto
  qemu-native

On build machines shared by multiple users, a single jobserver can be
shared between multiple builds (using the JOBSERVER_FIFO variable).
Running the above build in two different build directories at the same
time gives a ~12% improvement (43:17 -> 37:55).

Finally, the memory pressure from e.g. compiling multiple c++ based
projects is also reduced. In our case, a cloud based build machine (with
32 cores and 32GB RAM) fails to compile llvm-rust-native (in parallel to
nodejs) without the jobserver due to a lack of memory.

This patch set is roughly based on previous work by Richard[1]. That
patch lists three TODO items, which are all addressed by these patches:
 * The fifo path defaults to TMPDIR/jobserver_fifo, but can be
   configured using JOBSERVER_FIFO.
 * The number of make threads defaults to the value from PARALLEL_MAKE
   (which is then redundant).
 * If PARALLEL_MAKE is unset, the jobserver functionality is skipped.

Further work in addition to this patch set could be to make bitbake
tasks jobserver aware.

Changes since v1[2]:
 * Fixed typos in various places
 * Added JOBSERVER_IGNORE variable to jobserver.bbclass in patch 1
 * Changed ninja to use kitware fork with only two patches added in
   patch 3
 * Updated upstream patch submission link in qemu in patch 4

Changes since the RFC[3]:
 * The ninja src uri change in patch 3 is converted to a set of patches
 * The qemu fix in patch 4 is converted to a submitted patch

[1] 
https://lore.kernel.org/openembedded-core/1423223184.20217.15.ca...@linuxfoundation.org/
[2] 
https://lore.kernel.org/openembedded-core/20240403070204.367470-1-mar...@geanix.com/
[3] 
https://lore.kernel.org/openembedded-core/20230828124834.376779-1-mar...@geanix.com/

Martin Hundebøll (5):
  classes: jobserver: support gnu make fifo jobserver
  scripts: build-env: allow passing JOBSERVER_FIFO from environment
  ninja: build modified version with GNU Make jobserver support
  qemu: enable parallel builds when using the jobserver class
  contrib: add python service and systemd unit to run shared jobserver

 contrib/jobserver/jobserver.py|  78 +
 contrib/jobserver/jobserver.service   |  10 +
 meta/classes-global/jobserver.bbclass |  87 ++
 meta/conf/bitbake.conf|   2 +-
 ...ename-TokenPool-Setup-to-SetupClient.patch | 113 
 ...-jobserver-fifo-style-client-support.patch | 271 ++
 meta/recipes-devtools/ninja/ninja_1.11.1.bb   |   8 +-
 meta/recipes-devtools/qemu/qemu.inc   |   1 +
 ...e-jobserver-auth-argument-when-calli.patch |  37 +++
 scripts/oe-buildenv-internal  |   2 +-
 10 files changed, 605 insertions(+), 4 deletions(-)
 create mode 100644 contrib/jobserver/jobserver.py
 create mode 100644 contrib/jobserver/jobserver.service
 create mode 100644 meta/classes-global/jobserver.bbclass
 create mode 100644 
meta/recipes-devtools/ninja/files/0001-Rename-TokenPool-Setup-to-SetupClient.patch
 create mode 100644 
meta/recipes-devtools/ninja/files/0002-Add-GNU-make-jobserver-fifo-style-client-support.patch
 create mode 100644 
meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch

-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197955): 
https://lists.openembedded.org/g/openembedded-core/message/197955
Mute This Topic: https://lists.openembedded.org/mt/105326966/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/5] classes: jobserver: support gnu make fifo jobserver

2024-04-04 Thread Martin Hundeb?ll

Hi Andreas,

On Thu, 2024-04-04 at 09:56 +0200, Andreas Helbech Kleist wrote:
> > +    # avoid making make-native or its dependencies depend on make-
> > native itself
> > +    if d.getVar("PN") in (
> > +    "make-native",
> > +    "libtool-native",
> > +    "pkgconfig-native",
> > +    "automake-native",
> > +    "autoconf-native",
> > +    "m4-native",
> > +    "texinfo-dummy-native",
> > +    "gettext-minimal-native",
> > +    "quilt-native",
> > +    "gnu-config-native",
> 
> This seems a bit brittle, but I guess make's dependencies doesn't
> change often. So it is probably not a big issue.
> 
> How does it break if a dependency is missing?

Bitbake errors out with a dependency loop.

// Martin

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197948): 
https://lists.openembedded.org/g/openembedded-core/message/197948
Mute This Topic: https://lists.openembedded.org/mt/105304233/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 3/5] ninja: build modified version with GNU Make jobserver support

2024-04-03 Thread Martin Hundeb?ll

On Wed, 2024-04-03 at 17:58 +0200, Alexander Kanavin wrote:
> We can't carry this mass of patches. It has to land upstream first.

Understandable.

> Are you able/willing to drive the upstream pull request to
> completion?

No, I don't think so. I'm not familiar with the ninja codebase, and the
pull requests has been lingering for too long.

Another option might be to propose adding the fifo feature to the
kitware fork of ninja:
https://github.com/Kitware/ninja

They have forked ninja to get the jobserver feature in, but haven't
taken the fifo part so far.

The kitware fork is also used by buildroot:
https://gitlab.com/buildroot.org/buildroot/-/blob/master/package/ninja/ninja.mk

// Martin

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197940): 
https://lists.openembedded.org/g/openembedded-core/message/197940
Mute This Topic: https://lists.openembedded.org/mt/105304230/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 1/5] classes: jobserver: support gnu make fifo jobserver

2024-04-03 Thread Martin Hundeb?ll

Add a class to implement the gnu make fifo style jobserver. The class
can be activated by symply adding an `INHERIT += "jobserver"` to the
local configuration.

Furthermore, one can configure an external jobserver (i.e. a server
shared between multiple builds), by configured the `JOBSERVER_FIFO`
variable to point at an existing jobserver fifo.

The jobserver class uses the fifo style jobserver, which doesn't require
passing open file descriptors around. It does, however, require
make-4.4, which isn't available in common distro yet. To work around
this, the class make all recipes (except make and its dependencies
itself) depend on `virtual/make-native`.

Signed-off-by: Martin Hundebøll 
---
 meta/classes-global/jobserver.bbclass | 80 +++
 meta/conf/bitbake.conf|  2 +-
 2 files changed, 81 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes-global/jobserver.bbclass

diff --git a/meta/classes-global/jobserver.bbclass 
b/meta/classes-global/jobserver.bbclass
new file mode 100644
index 00..c76909fe50
--- /dev/null
+++ b/meta/classes-global/jobserver.bbclass
@@ -0,0 +1,80 @@
+JOBSERVER_FIFO ?= ""
+JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of 
creating a per-build server."
+
+addhandler jobserver_setup_fifo
+jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
+
+python jobserver_setup_fifo() {
+# don't setup a per-build fifo, if an external one is configured
+if d.getVar("JOBSERVER_FIFO"):
+return
+
+# don't use a job-server if no parallelism is configured
+jobs = oe.utils.parallel_make(d)
+if jobs in (None, 1):
+return
+
+# reduce jobs by one as a token has implicitly been handed to the
+# process requesting tokens
+jobs -= 1
+
+fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
+
+# and old fifo might be lingering; remove it
+if os.path.exists(fifo):
+os.remove(fifo)
+
+# create a new fifo to use for communicating tokens
+os.mkfifo(fifo)
+
+# fill the fifo with the number of tokens to hand out
+wfd = os.open(fifo, os.O_RDWR)
+written = os.write(wfd, b"+" * jobs)
+if written != (jobs):
+bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
+
+# configure the per-build fifo path to use
+d.setVar("JOBSERVER_FIFO", fifo)
+}
+
+python () {
+# don't configure the fifo if none is defined
+fifo = d.getVar("JOBSERVER_FIFO")
+if not fifo:
+return
+
+# avoid making make-native or its dependencies depend on make-native itself
+if d.getVar("PN") in (
+"make-native",
+"libtool-native",
+"pkgconfig-native",
+"automake-native",
+"autoconf-native",
+"m4-native",
+"texinfo-dummy-native",
+"gettext-minimal-native",
+"quilt-native",
+"gnu-config-native",
+):
+return
+
+# don't make unwilling recipes depend on make-native
+if d.getVar('INHIBIT_DEFAULT_DEPS', False):
+return
+
+# make other recipes depend on make-native to make sure it is new enough to
+# support the --jobserver-auth=fifo: syntax (from make-4.4 and 
onwards)
+d.appendVar("DEPENDS", " virtual/make-native")
+
+# disable the "-j " flag, as that overrides the jobserver fifo tokens
+d.setVar("PARALLEL_MAKE", "")
+d.setVar("PARALLEL_MAKEINST", "")
+
+# set and export the jobserver in the environment
+d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
+d.setVarFlag("MAKEFLAGS", "export", "1")
+
+# ignore the joberserver argument part of MAKEFLAGS in the hash, as that
+# shouldn't change the build output
+d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "| 
--jobserver-auth=fifo:" + fifo)
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6f180d18b0..23a016b31e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -960,7 +960,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH 
BBPATH BBSERVER DL_DI
 BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot 
DEPLOY_DIR \
 SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
 SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES 
\
-OMP_NUM_THREADS BB_CURRENTTASK"
+OMP_NUM_THREADS BB_CURRENTTASK JOBSERVER_FIFO"
 BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS 
BUILDHISTORY_DIR \
 SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS"
 BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID 
\
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197910): 
https://lists.openembedded.org/g/openembedded-core/message/197910
Mute This Topic: https://lists.openembedded.org/mt/105304233/21656
Group Owner:

[OE-core] [PATCH 0/5] Jobserver support

2024-04-03 Thread Martin Hundeb?ll

The parallelism of bitbake easily uses every available core on the build
host. But since every task is run with the same number of parallel
threads/processes, multiple tasks might load the CPU excessively, which
in turn slows down the build due to scheduling overhead.

This patch series adds a class that creates a fifo filled with
PARALLEL_MAKE tokens. The path to the created fifo is then configured in
the MAKEFLAGS environment variable, which is read by make and a patched
ninja (and gcc if doing lto).

The benefits from using the jobserver depends on the set of executed
tasks: running multiple large do_compile tasks simultaneously benefits
more than multiple do_fetch tasks. A simple test building the following
tasks (and all their dependencies) yields a ~5% improvement in build
time (20:20 -> 19:20):

  nodejs-native
  rust-llvm-native
  rust-native
  linux-yocto
  qemu-native

On build machines shared by multiple users, a single jobserver can be
shared between multiple builds (using the JOBSERVER_FIFO variable).
Running the above build in two different build directories at the same
time gives a ~12% improvement (43:17 -> 37:55).

Finally, the memory pressure from e.g. compiling multiple c++ based
projects is also reduced. In our case, a cloud based build machine (with
32 cores and 32GB RAM) fails to compile llvm-rust-native (in parallel to
nodejs) without the jobserver due to a lack of memory.

This patch set is roughly based on previous work by Richard[1]. That
patch lists three TODO items, which are all addressed by these patches:
 * The fifo path defaults to TMPDIR/jobserver_fifo, but can be
   configured using JOBSERVER_FIFO.
 * The number of make threads defaults to the value from PARALLEL_MAKE
   (which is then redundant).
 * If PARALLEL_MAKE is unset, the jobserver functionality is skipped.

Further work in addition to this patch set could be to make bitbake
tasks jobserver aware.

Changes since the RFC[2]:
 * The ninja src uri change in patch 3 is converted to a set of patches
 * The qemu fix in patch 4 is converted to a submitted patch

[1] 
https://lore.kernel.org/openembedded-core/1423223184.20217.15.ca...@linuxfoundation.org/
[2] 
https://lore.kernel.org/openembedded-core/20230828124834.376779-1-mar...@geanix.com/

Martin Hundebøll (5):
  classes: jobserver: support gnu make fifo jobserver
  scripts: build-env: allow passing JOBSERVER_FIFO from environment
  ninja: build modified version with GNU Make jobserver support
  qemu: enable parallel builds when using the jobserver class
  contrib: add python service and systemd unit to run shared jobserver

 contrib/jobserver/jobserver.py|   78 +
 contrib/jobserver/jobserver.service   |   10 +
 meta/classes-global/jobserver.bbclass |   80 +
 meta/conf/bitbake.conf|2 +-
 ...dd-GNU-make-jobserver-client-support.patch |  494 +++
 ...l-monitoring-to-SubprocessSet-DoWork.patch |  560 +++
 ...er-when-jN-is-forced-on-command-line.patch |  198 +++
 .../files/0004-Honor-lN-from-MAKEFLAGS.patch  |  134 ++
 ...e-LinePrinter-for-TokenPool-messages.patch |  128 ++
 .../files/0006-Prepare-PR-for-merging.patch   |  156 ++
 .../files/0007-Add-tests-for-TokenPool.patch  |  237 +++
 ...0008-Add-tests-for-subprocess-module.patch |  121 ++
 .../0009-Add-tests-for-build-module.patch |  397 +
 ...-implementation-for-GNUmakeTokenPool.patch | 1283 +
 .../0011-Prepare-PR-for-merging-part-II.patch |  744 ++
 ...ename-TokenPool-Setup-to-SetupClient.patch |  109 ++
 ...013-Add-TokenPool-SetupMaster-method.patch |   78 +
 ...mmand-line-option-m-tokenpool-master.patch |  101 ++
 ...plement-GNUmakeTokenPool-SetupMaster.patch |  152 ++
 ...mplement-GNUmakeTokenPool-CreatePool.patch |   87 ++
 ...-Add-tests-for-TokenPool-SetupMaster.patch |  145 ++
 ...-jobserver-fifo-style-client-support.patch |  265 
 ...ional-argument-to-m-tokenpool-master.patch |  368 +
 ...-jobserver-fifo-style-master-support.patch |  287 
 meta/recipes-devtools/ninja/ninja_1.11.1.bb   |   33 -
 meta/recipes-devtools/ninja/ninja_1.12.0.bb   |   55 +
 meta/recipes-devtools/qemu/qemu.inc   |1 +
 ...e-jobserver-auth-argument-when-calli.patch |   37 +
 scripts/oe-buildenv-internal  |2 +-
 29 files changed, 6307 insertions(+), 35 deletions(-)
 create mode 100644 contrib/jobserver/jobserver.py
 create mode 100644 contrib/jobserver/jobserver.service
 create mode 100644 meta/classes-global/jobserver.bbclass
 create mode 100644 
meta/recipes-devtools/ninja/files/0001-Add-GNU-make-jobserver-client-support.patch
 create mode 100644 
meta/recipes-devtools/ninja/files/0002-Add-TokenPool-monitoring-to-SubprocessSet-DoWork.patch
 create mode 100644 
meta/recipes-devtools/ninja/files/0003-Ignore-jobserver-when-jN-is-forced-on-command-line.patch
 create mode 100644 
meta/recipes-devtools/ninja/files/0004-Honor-lN-from-MAKEFLAGS.patch
 create mode 100644

[OE-core] [PATCH 2/5] scripts: build-env: allow passing JOBSERVER_FIFO from environment

2024-04-03 Thread Martin Hundeb?ll

Sharing a common jobserver fifo between multiple (containerized) builds
is much easier, if an administrator can configure said jobserver fifo
path in the environment. Append the JOBSERVER_FIFO variable name to the
list of variables configurable through the environment.

Signed-off-by: Martin Hundebøll 
---
 scripts/oe-buildenv-internal | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index 2fdb19565a..c8e67ffb8f 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -112,7 +112,7 @@ HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY 
ftps_proxy ALL_PROXY \
 all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \
 SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND \
 SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA 
BB_SETSCENE_ENFORCE \
-BB_LOGCONFIG"
+BB_LOGCONFIG JOBSERVER_FIFO"
 
 BB_ENV_PASSTHROUGH_ADDITIONS="$(echo $BB_ENV_PASSTHROUGH_ADDITIONS 
$BB_ENV_PASSTHROUGH_ADDITIONS_OE | tr ' ' '\n' | LC_ALL=C sort --unique | tr 
'\n' ' ')"
 
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197908): 
https://lists.openembedded.org/g/openembedded-core/message/197908
Mute This Topic: https://lists.openembedded.org/mt/105304231/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 5/5] contrib: add python service and systemd unit to run shared jobserver

2024-04-03 Thread Martin Hundeb?ll

For CI setups that might end up building multiple yocto builds in
parallel, a shared jobserver can reduce the total load of the system.
Setting up such a jobserver is simple, but it does require a process
hanging around to keep the jobserver fifo open (to avoid blocking token
requests).

Add a simple python script that creates such a jobserver fifo and waits
forever. Also add a systemd unit file to start the python service at
boot.

The systemd unit can be installed in $HOME/.config/systemd/user/, but
one might need to add a droplet config (i.e. `systemctl --user edit
jobserver.service`) to setup the PYTHONPATH variable to make the python
script loadable.

Signed-off-by: Martin Hundebøll 
---
 contrib/jobserver/jobserver.py  | 78 +
 contrib/jobserver/jobserver.service | 10 
 2 files changed, 88 insertions(+)
 create mode 100644 contrib/jobserver/jobserver.py
 create mode 100644 contrib/jobserver/jobserver.service

diff --git a/contrib/jobserver/jobserver.py b/contrib/jobserver/jobserver.py
new file mode 100644
index 00..41b085f47f
--- /dev/null
+++ b/contrib/jobserver/jobserver.py
@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+
+from pathlib import Path
+from threading import Event
+import argparse
+import os
+import shutil
+import signal
+
+resumed = Event()
+runtime_dir = os.environ.get("XDG_RUNTIME_DIR", "/run")
+
+def signal_handler(signum, _frame):
+"""Wait for an external signal exit the process gracefully."""
+resumed.set()
+
+
+def main(path, user, group, mode, jobs):
+"""Setup a fifo to used as jobserver shared between builds."""
+try:
+path.unlink(missing_ok=True)
+os.mkfifo(path)
+shutil.chown(path, user, group)
+os.chmod(path, mode)
+except (FileNotFoundError, PermissionError) as exc:
+raise SystemExit(f"failed to create fifo: {path}: {exc.strerror}")
+
+print(f"jobserver: {path}: {jobs} jobs")
+fifo = os.open(path, os.O_RDWR)
+os.write(fifo, b"+" * jobs)
+
+print("jobserver: ready; waiting indefinitely")
+signal.signal(signal.SIGTERM, signal_handler)
+signal.signal(signal.SIGINT, signal_handler)
+resumed.wait()
+
+print("jobserver: exiting")
+path.unlink()
+os.close(fifo)
+
+
+if __name__ == "__main__":
+parser = argparse.ArgumentParser(
+prog='Make jobserver',
+description='Simple application to instantiate a jobserver fifo and 
hang around',
+)
+parser.add_argument(
+"--mode",
+help="Permission to apply to jobserver fifo",
+type=lambda v: int(v, 8),
+default=0o0666,
+)
+parser.add_argument(
+"--user",
+help="Username or id to assign ownership of fifo to",
+default=os.getuid(),
+)
+parser.add_argument(
+"--group",
+help="Groupname of id to assign ownership of fifo to",
+default=os.getgid(),
+)
+parser.add_argument(
+"path",
+help="Path to jobserver fifo path",
+type=Path,
+nargs='?',
+default=f"{runtime_dir}/jobserver",
+)
+parser.add_argument(
+"jobs",
+help="Number of tokens to load jobserver with",
+type=int,
+nargs='?',
+default=os.cpu_count(),
+)
+args = parser.parse_args()
+main(args.path, args.user, args.group, args.mode, args.jobs)
diff --git a/contrib/jobserver/jobserver.service 
b/contrib/jobserver/jobserver.service
new file mode 100644
index 00..bbc7167ac0
--- /dev/null
+++ b/contrib/jobserver/jobserver.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=Shared jobserver fifo
+
+[Service]
+Type=simple
+Environment=PYTHONUNBUFFERED=1
+ExecStart=python jobserver.py
+
+[Install]
+WantedBy=multi-user.target
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197909): 
https://lists.openembedded.org/g/openembedded-core/message/197909
Mute This Topic: https://lists.openembedded.org/mt/105304232/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 4/5] qemu: enable parallel builds when using the jobserver class

2024-04-03 Thread Martin Hundeb?ll

If the jobserver class is enabled, the PARALLEL_MAKE variable is unset in
favor of configuring a shared jobserver in the MAKEFLAGS variable. However,
the qemu makefile translates the missing `-j` argument to `-j1` when
calling into meson / ninja. Add a patch to make the qemu makefile
consider the --jobserver-auth option too.

Signed-off-by: Martin Hundebøll 
---
 meta/recipes-devtools/qemu/qemu.inc   |  1 +
 ...e-jobserver-auth-argument-when-calli.patch | 37 +++
 2 files changed, 38 insertions(+)
 create mode 100644 
meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch

diff --git a/meta/recipes-devtools/qemu/qemu.inc 
b/meta/recipes-devtools/qemu/qemu.inc
index 4501f84c2b..1f86bf5a44 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -39,6 +39,7 @@ SRC_URI = "https://download.qemu.org/${BPN}-${PV}.tar.xz \
file://0003-linux-user-Add-strace-for-shmat.patch \
file://0004-linux-user-Rewrite-target_shmat.patch \

file://0005-tests-tcg-Check-that-shmat-does-not-break-proc-self-.patch \
+   
file://0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch \
file://CVE-2023-6683.patch \
file://qemu-guest-agent.init \
file://qemu-guest-agent.udev \
diff --git 
a/meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch
 
b/meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch
new file mode 100644
index 00..33db8b7ddc
--- /dev/null
+++ 
b/meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch
@@ -0,0 +1,37 @@
+From 730530c5f01e00cdc3754ebb8f3d7ff995f3376e Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Martin=20Hundeb=C3=B8ll?= 
+Date: Thu, 21 Sep 2023 10:57:45 +0200
+Subject: [PATCH] Makefile: preserve --jobserver-auth argument when calling
+ ninja
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Qemu wraps its call to ninja in a Makefile. Since ninja, as opposed to
+make, utilizes all CPU cores by default, the qemu Makefile translates
+the absense of a `-jN` argument into `-j1`. This breaks jobserver
+functionality, so update the -jN mangling to take the --jobserver-auth
+argument into considerationa too.
+
+Signed-off-by: Martin Hundebøll 
+Upstream-Status: Submitted [https://gitlab.com/qemu-project/qemu/-/issues/1898]
+---
+ Makefile | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Makefile b/Makefile
+index 8f36990335..183756018f 100644
+--- a/Makefile
 b/Makefile
+@@ -142,7 +142,7 @@ MAKE.k = $(findstring k,$(firstword $(filter-out 
--%,$(MAKEFLAGS
+ MAKE.q = $(findstring q,$(firstword $(filter-out --%,$(MAKEFLAGS
+ MAKE.nq = $(if $(word 2, $(MAKE.n) $(MAKE.q)),nq)
+ NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \
+-$(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS \
++$(or $(filter -l% -j%, $(MAKEFLAGS)), $(if $(filter 
--jobserver-auth=%, $(MAKEFLAGS)),, -j1)) \
+ -d keepdepfile
+ ninja-cmd-goals = $(or $(MAKECMDGOALS), all)
+ ninja-cmd-goals += $(foreach g, $(MAKECMDGOALS), $(.ninja-goals.$g))
+-- 
+2.44.0
+
-- 
2.44.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#197906): 
https://lists.openembedded.org/g/openembedded-core/message/197906
Mute This Topic: https://lists.openembedded.org/mt/105304229/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2] libpam: split /etc/environment into pam-plugin-env package

2023-11-27 Thread Martin Hundeb?ll

Since systemd-v255, pam-plugin-umask is pulled in by by the logind
package config for systemd. This causes /etc/environment to be installed
as part of libpam-runtime. In our case, this broke do_rootfs for our
image, because /etc/environment is already provided by another (custom)
recipe.

Fix this by making the /etc/environment file part of the pam-plugin-env
package, which isn't automatically pulled in by systemd-logind. It also
happens to be the where it should be, as the file is installed as part
of the pam_env plugin.

Signed-off-by: Martin Hundebøll 
---

Changes in v2:
 * fixed typos in commit message
 * replaced changes to FILES:${PN}-runtime with prepend=True in
   do_split_packages()

 meta/recipes-extended/pam/libpam_1.5.3.bb | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-extended/pam/libpam_1.5.3.bb 
b/meta/recipes-extended/pam/libpam_1.5.3.bb
index 1aa307af4d..3183bb018f 100644
--- a/meta/recipes-extended/pam/libpam_1.5.3.bb
+++ b/meta/recipes-extended/pam/libpam_1.5.3.bb
@@ -54,6 +54,11 @@ FILES:${PN}-dev += "${base_libdir}/security/*.la 
${base_libdir}/*.la ${base_libd
 FILES:${PN}-runtime = "${sysconfdir} ${sbindir} ${systemd_system_unitdir}"
 FILES:${PN}-xtests = "${datadir}/Linux-PAM/xtests"
 
+# libpam installs /etc/environment for use with the pam_env plugin. Make sure 
it is
+# packaged with the pam-plugin-env package to avoid breaking installations 
which
+# install that file via other packages
+FILES:pam-plugin-env = "${sysconfdir}/environment"
+
 PACKAGES_DYNAMIC += "^${MLPREFIX}pam-plugin-.*"
 
 def get_multilib_bit(d):
@@ -110,7 +115,7 @@ python populate_packages:prepend () {
 pam_pkgname = mlprefix + 'pam-plugin%s'
 
 do_split_packages(d, pam_libdir, r'^pam(.*)\.so$', pam_pkgname,
-  'PAM plugin for %s', hook=pam_plugin_hook, 
extra_depends='')
+  'PAM plugin for %s', hook=pam_plugin_hook, 
extra_depends='', prepend=True)
 do_split_packages(d, pam_filterdir, r'^(.*)$', 'pam-filter-%s', 'PAM 
filter for %s', extra_depends='')
 }
 
-- 
2.42.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191297): 
https://lists.openembedded.org/g/openembedded-core/message/191297
Mute This Topic: https://lists.openembedded.org/mt/102827883/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] libpam: split /etc/environment into pam-plugin-env package

2023-11-24 Thread Martin Hundeb?ll

Since systemd-v255, pam-plugin-umask is pulled in by by the logind
package config for systemd. This causes /etc/environment to by installed
as part of libpam-runtime. In our case, this broke do_rootfs for our
image, because /etc/environment is already provided another (custom)
recipe.

Fix this by making the /etc/environment file part of the pam-plugin-env
package, which isn't automatically pulled in by systemd-logind. It also
happens to be the where it should be, as the file is installed as by the
pam_env plugin.

Signed-off-by: Martin Hundebøll 
---
 meta/recipes-extended/pam/libpam_1.5.3.bb | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-extended/pam/libpam_1.5.3.bb 
b/meta/recipes-extended/pam/libpam_1.5.3.bb
index 1aa307af4d..c2ba1e6899 100644
--- a/meta/recipes-extended/pam/libpam_1.5.3.bb
+++ b/meta/recipes-extended/pam/libpam_1.5.3.bb
@@ -51,9 +51,14 @@ PACKAGECONFIG[userdb] = "--enable-db=db,--enable-db=no,db,"
 PACKAGES += "${PN}-runtime ${PN}-xtests"
 FILES:${PN} = "${base_libdir}/lib*${SOLIBS}"
 FILES:${PN}-dev += "${base_libdir}/security/*.la ${base_libdir}/*.la 
${base_libdir}/lib*${SOLIBSDEV}"
-FILES:${PN}-runtime = "${sysconfdir} ${sbindir} ${systemd_system_unitdir}"
+FILES:${PN}-runtime = "${sysconfdir}/pam.d ${sysconfdir}/security 
${sysconfdir}/tmpfiles.d ${sbindir} ${systemd_system_unitdir}"
 FILES:${PN}-xtests = "${datadir}/Linux-PAM/xtests"
 
+# libpam installs /etc/environment for use with the pam_env plugin. Make sure 
it is
+# packaged with the pam-plugin-env package to avoid breaking installations 
which
+# install that file via other packages
+FILES:pam-plugin-env = "/etc/environment"
+
 PACKAGES_DYNAMIC += "^${MLPREFIX}pam-plugin-.*"
 
 def get_multilib_bit(d):
-- 
2.42.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191201): 
https://lists.openembedded.org/g/openembedded-core/message/191201
Mute This Topic: https://lists.openembedded.org/mt/102780994/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH 3/5] ninja: build modified version with GNU Make jobserver support

2023-09-05 Thread Martin Hundeb?ll

On Mon, 2023-08-28 at 19:31 +0200, Martin Hundeb?ll wrote:
>  
>  Aug 28, 2023 19:16:28 Khem Raj : 
>  
>  > On Mon, Aug 28, 2023 at 5:48 AM Martin Hundeb?ll
>  wrote: 
>  >> 
>  >> Ninja doesn't (yet) support the GNU Make jobserver out of the
> box, but 
>  >> there is a pull request adding that support[1]. Switch the
> SRC_URI and 
>  >> SRCREV to point at the source of that pull request, to make ninja
> play 
>  >> nicely together with the recently added jobserver class. 
>  >> 
>  >> Signed-off-by: Martin Hundebøll  
>  >> --- 
>  >> .../ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb}    | 4
> ++-- 
>  >> 1 file changed, 2 insertions(+), 2 deletions(-) 
>  >> rename meta/recipes-devtools/ninja/{ninja_1.11.1.bb =>
> ninja_1.12.0.bb} (84%) 
>  >> 
>  >> diff --git a/meta/recipes-devtools/ninja/ninja_1.11.1.bb
> b/meta/recipes-devtools/ninja/ninja_1.12.0.bb 
>  >> similarity index 84% 
>  >> rename from meta/recipes-devtools/ninja/ninja_1.11.1.bb 
>  >> rename to meta/recipes-devtools/ninja/ninja_1.12.0.bb 
>  >> index 8e297ec4d4..9abdd40a92 100644 
>  >> --- a/meta/recipes-devtools/ninja/ninja_1.11.1.bb 
>  >> +++ b/meta/recipes-devtools/ninja/ninja_1.12.0.bb 
>  >> @@ -6,9 +6,9 @@ LIC_FILES_CHKSUM =
> "file://COPYING;md5=a81586a64ad4e476c791cda7e2f2c52e" 
>  >> 
>  >> DEPENDS = "re2c-native ninja-native" 
>  >> 
>  >> -SRCREV = "a524bf3f6bacd1b4ad85d719eed2737d8562f27a" 
>  >> +SRCREV = "c9e21dbbc4c746ba397c0f9bec5f65c99f783c08" 
>  >> 
>  >> -SRC_URI = "git://github.com/ninja-
> build/ninja.git;branch=release;protocol=https" 
>  >> +SRC_URI = "git://github.com/stefanb2/ninja.git;branch=topic-
> issue-1139-part-3-jobserver-fifo;protocol=https" 
>  > 
>  > This is a little concerning, as we are pointing to a fork here and
>  > quite a lot depend on ninja now 
>  > a days. so I wonder whats the status of the patches and likelyhood
> of 
>  > them going upstream 
>  > in anycase, we should pick the patches instead of switching
> SRC_URI 
>  
>  Yes, I agree that we shouldn't change uri to a fork; hence the
> RFC... Maybe I should try to put some pressure on ninja upstream to
> consider the pull request... 

One option could be to use the kitware fork of ninja, which supports
the fifo makeserver:
https://github.com/Kitware/ninja

This is what buildroot does:
https://gitlab.com/buildroot.org/buildroot/-/blob/master/package/ninja/ninja.mk

// Martin

>  >> UPSTREAM_CHECK_GITTAGREGEX = "v(?P.*)" 
>  >> 
>  >> S = "${WORKDIR}/git" 
>  >> -- 
>  >> 2.41.0 
>  >> 
>  >> 
>  >>  
>  >> 
>  
>  
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#187205): 
https://lists.openembedded.org/g/openembedded-core/message/187205
Mute This Topic: https://lists.openembedded.org/mt/101009093/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH 1/5] classes: jobserver: support gnu make fifo jobserver

2023-08-28 Thread Martin Hundeb?ll


Aug 28, 2023 14:48:46 Martin Hundebøll :

> Add a class to implement the gnu make fifo style jobserver. The class
> can be activated by symply adding an `INHERIT += "jobserver"` to the
> local configuration.
>
> Furthermore, one can configure an external jobserver (i.e. a server
> shared between multiple builds), by configured the `JOBSERVER_FIFO`
> variable to point at an existing jobserver fifo.
>
> The jobserver class uses the fifo style jobserver, which doesn't require
> passing open file descriptors around. It does, however, require
> make-4.4, which isn't available in common distro yet. To work around
> this, the class make all recipes (except make and its dependencies
> itself) depend on `virtual/make-native`.
>
> Signed-off-by: Martin Hundebøll 
> ---
> meta/classes-global/jobserver.bbclass | 80 +++
> meta/conf/bitbake.conf    |  2 +-
> 2 files changed, 81 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes-global/jobserver.bbclass
>
> diff --git a/meta/classes-global/jobserver.bbclass 
> b/meta/classes-global/jobserver.bbclass
> new file mode 100644
> index 00..c76909fe50
> --- /dev/null
> +++ b/meta/classes-global/jobserver.bbclass
> @@ -0,0 +1,80 @@
> +JOBSERVER_FIFO ?= ""
> +JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of 
> creating a per-build server."
> +
> +addhandler jobserver_setup_fifo
> +jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
> +
> +python jobserver_setup_fifo() {
> +    # don't setup a per-build fifo, if an external one is configured
> +    if d.getVar("JOBSERVER_FIFO"):
> +    return
> +
> +    # don't use a job-server if no parallelism is configured
> +    jobs = oe.utils.parallel_make(d)
> +    if jobs in (None, 1):
> +    return
> +
> +    # reduce jobs by one as a token has implicitly been handed to the
> +    # process requesting tokens
> +    jobs -= 1
> +
> +    fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
> +
> +    # and old fifo might be lingering; remove it
> +    if os.path.exists(fifo):
> +    os.remove(fifo)
> +
> +    # create a new fifo to use for communicating tokens
> +    os.mkfifo(fifo)
> +
> +    # fill the fifo with the number of tokens to hand out
> +    wfd = os.open(fifo, os.O_RDWR)
> +    written = os.write(wfd, b"+" * jobs)
> +    if written != (jobs):
> +    bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
> +
> +    # configure the per-build fifo path to use
> +    d.setVar("JOBSERVER_FIFO", fifo)
> +}
> +
> +python () {
> +    # don't configure the fifo if none is defined
> +    fifo = d.getVar("JOBSERVER_FIFO")
> +    if not fifo:
> +    return
> +
> +    # avoid making make-native or its dependencies depend on make-native 
> itself
> +    if d.getVar("PN") in (
> +    "make-native",
> +    "libtool-native",
> +    "pkgconfig-native",
> +    "automake-native",
> +    "autoconf-native",
> +    "m4-native",
> +    "texinfo-dummy-native",
> +    "gettext-minimal-native",
> +    "quilt-native",
> +    "gnu-config-native",
> +    ):
> +    return
> +
> +    # don't make unwilling recipes depend on make-native
> +    if d.getVar('INHIBIT_DEFAULT_DEPS', False):
> +    return
> +
> +    # make other recipes depend on make-native to make sure it is new enough 
> to
> +    # support the --jobserver-auth=fifo: syntax (from make-4.4 and 
> onwards)
> +    d.appendVar("DEPENDS", " virtual/make-native")

I would like some feedback on this part, i.e. changing package dependencies 
depending on a pure build-configuration like this. I would prefer if the build 
didn't change at all when enabling the jobserver class, which would require 
adding virtual/make-native to the base dependencies regardless of this class. 
Would that be acceptable?

> +    # disable the "-j " flag, as that overrides the jobserver fifo 
> tokens
> +    d.setVar("PARALLEL_MAKE", "")
> +    d.setVar("PARALLEL_MAKEINST", "")
> +
> +    # set and export the jobserver in the environment
> +    d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
> +    d.setVarFlag("MAKEFLAGS", "export", "1")
> +
> +    # ignore the joberserver argument part of MAKEFLAGS in the hash, as that
> +    # shouldn't change the build output
> +    d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "| 
> --jobserver-auth=fifo:" + fifo)
> +}
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index cf7ff3328c..8cf188270b 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -946,7 +946,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD 
> BB_TASKHASH BBPATH BBSERVER DL_DI
>  BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot 
> DEPLOY_DIR \
>  SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
>  SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE 
> GIT_CEILING_DIRECTORIES \
> -

Re: [OE-core] [RFC PATCH 3/5] ninja: build modified version with GNU Make jobserver support

2023-08-28 Thread Martin Hundeb?ll


Aug 28, 2023 19:16:28 Khem Raj :

> On Mon, Aug 28, 2023 at 5:48 AM Martin Hundeb?ll  wrote:
>>
>> Ninja doesn't (yet) support the GNU Make jobserver out of the box, but
>> there is a pull request adding that support[1]. Switch the SRC_URI and
>> SRCREV to point at the source of that pull request, to make ninja play
>> nicely together with the recently added jobserver class.
>>
>> Signed-off-by: Martin Hundebøll 
>> ---
>> .../ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb}    | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>> rename meta/recipes-devtools/ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb} (84%)
>>
>> diff --git a/meta/recipes-devtools/ninja/ninja_1.11.1.bb 
>> b/meta/recipes-devtools/ninja/ninja_1.12.0.bb
>> similarity index 84%
>> rename from meta/recipes-devtools/ninja/ninja_1.11.1.bb
>> rename to meta/recipes-devtools/ninja/ninja_1.12.0.bb
>> index 8e297ec4d4..9abdd40a92 100644
>> --- a/meta/recipes-devtools/ninja/ninja_1.11.1.bb
>> +++ b/meta/recipes-devtools/ninja/ninja_1.12.0.bb
>> @@ -6,9 +6,9 @@ LIC_FILES_CHKSUM = 
>> "file://COPYING;md5=a81586a64ad4e476c791cda7e2f2c52e"
>>
>> DEPENDS = "re2c-native ninja-native"
>>
>> -SRCREV = "a524bf3f6bacd1b4ad85d719eed2737d8562f27a"
>> +SRCREV = "c9e21dbbc4c746ba397c0f9bec5f65c99f783c08"
>>
>> -SRC_URI = 
>> "git://github.com/ninja-build/ninja.git;branch=release;protocol=https"
>> +SRC_URI = 
>> "git://github.com/stefanb2/ninja.git;branch=topic-issue-1139-part-3-jobserver-fifo;protocol=https"
>
> This is a little concerning, as we are pointing to a fork here and
> quite a lot depend on ninja now
> a days. so I wonder whats the status of the patches and likelyhood of
> them going upstream
> in anycase, we should pick the patches instead of switching SRC_URI

Yes, I agree that we shouldn't change uri to a fork; hence the RFC... Maybe I 
should try to put some pressure on ninja upstream to consider the pull 
request...

I can look into whether a separate patch is feasible...

>> UPSTREAM_CHECK_GITTAGREGEX = "v(?P.*)"
>>
>> S = "${WORKDIR}/git"
>> --
>> 2.41.0
>>
>>
>> 
>>


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186841): 
https://lists.openembedded.org/g/openembedded-core/message/186841
Mute This Topic: https://lists.openembedded.org/mt/101009093/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [RFC PATCH 3/5] ninja: build modified version with GNU Make jobserver support

2023-08-28 Thread Martin Hundeb?ll

On Mon, 2023-08-28 at 14:57 +0200, Alexander Kanavin wrote:
> Thanks, make/ninja jobserver is something we've been talking about
> since forever, and it's great to see actual code.
> 
> I suppose the biggest obstacle is that ninja support hasn't yet
> landed
> upstream, and I'd like to ensure it does. Also, a link to the pull
> request is missing? :)

Indeed, upstream seems to ignore the pull request :(

Link references are like attachments: easy to forget... Here it is:
https://github.com/ninja-build/ninja/issues/1139

// Martin

> On Mon, 28 Aug 2023 at 14:48, Martin Hundeb?ll 
> wrote:
> > 
> > Ninja doesn't (yet) support the GNU Make jobserver out of the box,
> > but
> > there is a pull request adding that support[1]. Switch the SRC_URI
> > and
> > SRCREV to point at the source of that pull request, to make ninja
> > play
> > nicely together with the recently added jobserver class.
> > 
> > Signed-off-by: Martin Hundebøll 
> > ---
> >  .../ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb}    | 4
> > ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >  rename meta/recipes-devtools/ninja/{ninja_1.11.1.bb =>
> > ninja_1.12.0.bb} (84%)
> > 
> > diff --git a/meta/recipes-devtools/ninja/ninja_1.11.1.bb
> > b/meta/recipes-devtools/ninja/ninja_1.12.0.bb
> > similarity index 84%
> > rename from meta/recipes-devtools/ninja/ninja_1.11.1.bb
> > rename to meta/recipes-devtools/ninja/ninja_1.12.0.bb
> > index 8e297ec4d4..9abdd40a92 100644
> > --- a/meta/recipes-devtools/ninja/ninja_1.11.1.bb
> > +++ b/meta/recipes-devtools/ninja/ninja_1.12.0.bb
> > @@ -6,9 +6,9 @@ LIC_FILES_CHKSUM =
> > "file://COPYING;md5=a81586a64ad4e476c791cda7e2f2c52e"
> > 
> >  DEPENDS = "re2c-native ninja-native"
> > 
> > -SRCREV = "a524bf3f6bacd1b4ad85d719eed2737d8562f27a"
> > +SRCREV = "c9e21dbbc4c746ba397c0f9bec5f65c99f783c08"
> > 
> > -SRC_URI = "git://github.com/ninja-
> > build/ninja.git;branch=release;protocol=https"
> > +SRC_URI = "git://github.com/stefanb2/ninja.git;branch=topic-issue-
> > 1139-part-3-jobserver-fifo;protocol=https"
> >  UPSTREAM_CHECK_GITTAGREGEX = "v(?P.*)"
> > 
> >  S = "${WORKDIR}/git"
> > --
> > 2.41.0
> > 
> > 
> > 
> > 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186829): 
https://lists.openembedded.org/g/openembedded-core/message/186829
Mute This Topic: https://lists.openembedded.org/mt/101009093/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [RFC PATCH 5/5] contrib: add python service and systemd unit to run shared jobserver

2023-08-28 Thread Martin Hundeb?ll

For CI setups that might end up building multiple yocto builds in
parallel, a shared jobserver can reduce the total load of the system.
Setting up such a jobserver is simple, but it does require a process
hanging around to keep the jobserver fifo open (to avoid blocking token
requests).

Add a simple python script that creates such a jobserver fifo and waits
forever. Also add a systemd unit file to start the python service at
boot.

The systemd unit can be installed in $HOME/.config/systemd/user/, but
one might need to add a droplet config (i.e. `systemctl --user edit
jobserver.service`) to setup the PYTHONPATH variable to make the python
script loadable.

Signed-off-by: Martin Hundebøll 
---
 contrib/jobserver/jobserver.py  | 78 +
 contrib/jobserver/jobserver.service | 10 
 2 files changed, 88 insertions(+)
 create mode 100644 contrib/jobserver/jobserver.py
 create mode 100644 contrib/jobserver/jobserver.service

diff --git a/contrib/jobserver/jobserver.py b/contrib/jobserver/jobserver.py
new file mode 100644
index 00..41b085f47f
--- /dev/null
+++ b/contrib/jobserver/jobserver.py
@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+
+from pathlib import Path
+from threading import Event
+import argparse
+import os
+import shutil
+import signal
+
+resumed = Event()
+runtime_dir = os.environ.get("XDG_RUNTIME_DIR", "/run")
+
+def signal_handler(signum, _frame):
+"""Wait for an external signal exit the process gracefully."""
+resumed.set()
+
+
+def main(path, user, group, mode, jobs):
+"""Setup a fifo to used as jobserver shared between builds."""
+try:
+path.unlink(missing_ok=True)
+os.mkfifo(path)
+shutil.chown(path, user, group)
+os.chmod(path, mode)
+except (FileNotFoundError, PermissionError) as exc:
+raise SystemExit(f"failed to create fifo: {path}: {exc.strerror}")
+
+print(f"jobserver: {path}: {jobs} jobs")
+fifo = os.open(path, os.O_RDWR)
+os.write(fifo, b"+" * jobs)
+
+print("jobserver: ready; waiting indefinitely")
+signal.signal(signal.SIGTERM, signal_handler)
+signal.signal(signal.SIGINT, signal_handler)
+resumed.wait()
+
+print("jobserver: exiting")
+path.unlink()
+os.close(fifo)
+
+
+if __name__ == "__main__":
+parser = argparse.ArgumentParser(
+prog='Make jobserver',
+description='Simple application to instantiate a jobserver fifo and 
hang around',
+)
+parser.add_argument(
+"--mode",
+help="Permission to apply to jobserver fifo",
+type=lambda v: int(v, 8),
+default=0o0666,
+)
+parser.add_argument(
+"--user",
+help="Username or id to assign ownership of fifo to",
+default=os.getuid(),
+)
+parser.add_argument(
+"--group",
+help="Groupname of id to assign ownership of fifo to",
+default=os.getgid(),
+)
+parser.add_argument(
+"path",
+help="Path to jobserver fifo path",
+type=Path,
+nargs='?',
+default=f"{runtime_dir}/jobserver",
+)
+parser.add_argument(
+"jobs",
+help="Number of tokens to load jobserver with",
+type=int,
+nargs='?',
+default=os.cpu_count(),
+)
+args = parser.parse_args()
+main(args.path, args.user, args.group, args.mode, args.jobs)
diff --git a/contrib/jobserver/jobserver.service 
b/contrib/jobserver/jobserver.service
new file mode 100644
index 00..bbc7167ac0
--- /dev/null
+++ b/contrib/jobserver/jobserver.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=Shared jobserver fifo
+
+[Service]
+Type=simple
+Environment=PYTHONUNBUFFERED=1
+ExecStart=python jobserver.py
+
+[Install]
+WantedBy=multi-user.target
-- 
2.41.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186826): 
https://lists.openembedded.org/g/openembedded-core/message/186826
Mute This Topic: https://lists.openembedded.org/mt/101009094/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [RFC PATCH 1/5] classes: jobserver: support gnu make fifo jobserver

2023-08-28 Thread Martin Hundeb?ll

Add a class to implement the gnu make fifo style jobserver. The class
can be activated by symply adding an `INHERIT += "jobserver"` to the
local configuration.

Furthermore, one can configure an external jobserver (i.e. a server
shared between multiple builds), by configured the `JOBSERVER_FIFO`
variable to point at an existing jobserver fifo.

The jobserver class uses the fifo style jobserver, which doesn't require
passing open file descriptors around. It does, however, require
make-4.4, which isn't available in common distro yet. To work around
this, the class make all recipes (except make and its dependencies
itself) depend on `virtual/make-native`.

Signed-off-by: Martin Hundebøll 
---
 meta/classes-global/jobserver.bbclass | 80 +++
 meta/conf/bitbake.conf|  2 +-
 2 files changed, 81 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes-global/jobserver.bbclass

diff --git a/meta/classes-global/jobserver.bbclass 
b/meta/classes-global/jobserver.bbclass
new file mode 100644
index 00..c76909fe50
--- /dev/null
+++ b/meta/classes-global/jobserver.bbclass
@@ -0,0 +1,80 @@
+JOBSERVER_FIFO ?= ""
+JOBSERVER_FIFO[doc] = "Path to external jobserver fifo to use instead of 
creating a per-build server."
+
+addhandler jobserver_setup_fifo
+jobserver_setup_fifo[eventmask] = "bb.event.ConfigParsed"
+
+python jobserver_setup_fifo() {
+# don't setup a per-build fifo, if an external one is configured
+if d.getVar("JOBSERVER_FIFO"):
+return
+
+# don't use a job-server if no parallelism is configured
+jobs = oe.utils.parallel_make(d)
+if jobs in (None, 1):
+return
+
+# reduce jobs by one as a token has implicitly been handed to the
+# process requesting tokens
+jobs -= 1
+
+fifo = d.getVar("TMPDIR") + "/jobserver_fifo"
+
+# and old fifo might be lingering; remove it
+if os.path.exists(fifo):
+os.remove(fifo)
+
+# create a new fifo to use for communicating tokens
+os.mkfifo(fifo)
+
+# fill the fifo with the number of tokens to hand out
+wfd = os.open(fifo, os.O_RDWR)
+written = os.write(wfd, b"+" * jobs)
+if written != (jobs):
+bb.error("Failed to fil make fifo: {} != {}".format(written, jobs))
+
+# configure the per-build fifo path to use
+d.setVar("JOBSERVER_FIFO", fifo)
+}
+
+python () {
+# don't configure the fifo if none is defined
+fifo = d.getVar("JOBSERVER_FIFO")
+if not fifo:
+return
+
+# avoid making make-native or its dependencies depend on make-native itself
+if d.getVar("PN") in (
+"make-native",
+"libtool-native",
+"pkgconfig-native",
+"automake-native",
+"autoconf-native",
+"m4-native",
+"texinfo-dummy-native",
+"gettext-minimal-native",
+"quilt-native",
+"gnu-config-native",
+):
+return
+
+# don't make unwilling recipes depend on make-native
+if d.getVar('INHIBIT_DEFAULT_DEPS', False):
+return
+
+# make other recipes depend on make-native to make sure it is new enough to
+# support the --jobserver-auth=fifo: syntax (from make-4.4 and 
onwards)
+d.appendVar("DEPENDS", " virtual/make-native")
+
+# disable the "-j " flag, as that overrides the jobserver fifo tokens
+d.setVar("PARALLEL_MAKE", "")
+d.setVar("PARALLEL_MAKEINST", "")
+
+# set and export the jobserver in the environment
+d.appendVar("MAKEFLAGS", " --jobserver-auth=fifo:" + fifo)
+d.setVarFlag("MAKEFLAGS", "export", "1")
+
+# ignore the joberserver argument part of MAKEFLAGS in the hash, as that
+# shouldn't change the build output
+d.appendVarFlag("MAKEFLAGS", "vardepvalueexclude", "| 
--jobserver-auth=fifo:" + fifo)
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cf7ff3328c..8cf188270b 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -946,7 +946,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH 
BBPATH BBSERVER DL_DI
 BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot 
DEPLOY_DIR \
 SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
 SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES 
\
-OMP_NUM_THREADS BB_CURRENTTASK"
+OMP_NUM_THREADS BB_CURRENTTASK JOBSERVER_FIFO"
 BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS 
BUILDHISTORY_DIR \
 SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS"
 BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID 
\
-- 
2.41.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186824): 
https://lists.openembedded.org/g/openembedded-core/message/186824
Mute This Topic: https://lists.openembedded.org/mt/101009092/21656
Group Owner:

[OE-core] [RFC PATCH 2/5] scripts: build-env: allow passing JOBSERVER_FIFO from environment

2023-08-28 Thread Martin Hundeb?ll

Sharing a common jobserver fifo between multiple (containerized) builds
is much easier, if an administrator can configure said jobserver fifo
path in the environment. Append the JOBSERVER_FIFO variable name to the
list of variables configurable through the environment.

Signed-off-by: Martin Hundebøll 
---
 scripts/oe-buildenv-internal | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index 2fdb19565a..c8e67ffb8f 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -112,7 +112,7 @@ HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY 
ftps_proxy ALL_PROXY \
 all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \
 SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND \
 SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA 
BB_SETSCENE_ENFORCE \
-BB_LOGCONFIG"
+BB_LOGCONFIG JOBSERVER_FIFO"
 
 BB_ENV_PASSTHROUGH_ADDITIONS="$(echo $BB_ENV_PASSTHROUGH_ADDITIONS 
$BB_ENV_PASSTHROUGH_ADDITIONS_OE | tr ' ' '\n' | LC_ALL=C sort --unique | tr 
'\n' ' ')"
 
-- 
2.41.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186823): 
https://lists.openembedded.org/g/openembedded-core/message/186823
Mute This Topic: https://lists.openembedded.org/mt/101009091/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [RFC PATCH 4/5] qemu: enable parallel builds when using the jobserver class

2023-08-28 Thread Martin Hundeb?ll

If the jobserver class is enabled, the PARALLEL_MAKE variable is unset in
favor of configuring a shared jobserver in the MAKEFLAGS variable. However,
the qemu makefile translates the missing `-j` argument to `-j1` when
calling into meson / ninja. Avoid this by setting `-j` without a value.
For normal/GNU make, this can result in a fork bomb, but for ninja, it
simply makes it use the jobserver fifa instead.

Signed-off-by: Martin Hundebøll 
---
 meta/recipes-devtools/qemu/qemu.inc | 5 +
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-devtools/qemu/qemu.inc 
b/meta/recipes-devtools/qemu/qemu.inc
index ea02bf0c73..7da05fcbf4 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -110,6 +110,11 @@ EXTRA_OECONF = " \
 ${PACKAGECONFIG_CONFARGS} \
 "
 
+# Avoid the qemu makefile treating a missing `-j` argument as `-j1` when
+# calling meson / ninja. This happens when the `jobserver` class is used, since
+# it manages parallelism in the MAKEFLAGS variable instead of PARALLEL_MAKE.
+EXTRA_OEMAKE:append = "${@' -j' if d.getVar('JOBSERVER_FIFO') else ''}"
+
 B = "${WORKDIR}/build"
 
 #EXTRA_OECONF:append = " --python=${HOSTTOOLS_DIR}/python3"
-- 
2.41.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186827): 
https://lists.openembedded.org/g/openembedded-core/message/186827
Mute This Topic: https://lists.openembedded.org/mt/101009095/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [RFC PATCH 3/5] ninja: build modified version with GNU Make jobserver support

2023-08-28 Thread Martin Hundeb?ll

Ninja doesn't (yet) support the GNU Make jobserver out of the box, but
there is a pull request adding that support[1]. Switch the SRC_URI and
SRCREV to point at the source of that pull request, to make ninja play
nicely together with the recently added jobserver class.

Signed-off-by: Martin Hundebøll 
---
 .../ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb}| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
 rename meta/recipes-devtools/ninja/{ninja_1.11.1.bb => ninja_1.12.0.bb} (84%)

diff --git a/meta/recipes-devtools/ninja/ninja_1.11.1.bb 
b/meta/recipes-devtools/ninja/ninja_1.12.0.bb
similarity index 84%
rename from meta/recipes-devtools/ninja/ninja_1.11.1.bb
rename to meta/recipes-devtools/ninja/ninja_1.12.0.bb
index 8e297ec4d4..9abdd40a92 100644
--- a/meta/recipes-devtools/ninja/ninja_1.11.1.bb
+++ b/meta/recipes-devtools/ninja/ninja_1.12.0.bb
@@ -6,9 +6,9 @@ LIC_FILES_CHKSUM = 
"file://COPYING;md5=a81586a64ad4e476c791cda7e2f2c52e"
 
 DEPENDS = "re2c-native ninja-native"
 
-SRCREV = "a524bf3f6bacd1b4ad85d719eed2737d8562f27a"
+SRCREV = "c9e21dbbc4c746ba397c0f9bec5f65c99f783c08"
 
-SRC_URI = 
"git://github.com/ninja-build/ninja.git;branch=release;protocol=https"
+SRC_URI = 
"git://github.com/stefanb2/ninja.git;branch=topic-issue-1139-part-3-jobserver-fifo;protocol=https"
 UPSTREAM_CHECK_GITTAGREGEX = "v(?P.*)"
 
 S = "${WORKDIR}/git"
-- 
2.41.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#186825): 
https://lists.openembedded.org/g/openembedded-core/message/186825
Mute This Topic: https://lists.openembedded.org/mt/101009093/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

37 matches

Mail list logo