Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
hat was to set LC_ALL in opkg with: setlocale(LC_ALL, "en_US.UTF-8"); to check if it fixes the issue and yes it fixes but this is no go. Question is probably now to Alex does opkg can run fork instances for depends? Because I have found that it uses this vfork, however I dunno if it uses parent's process env variables or it's own. I think that this needs further investigations... BR Piotr Od: Richard Purdie Wysłane: środa, 19 lipca 2023 10:37 Do: Piotr Łobacz ; Alexandre Belloni DW: Alex Stewart ; openembedded-core@lists.openembedded.org Temat: Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support On Wed, 2023-07-19 at 07:53 +, Piotr Łobacz wrote: > I'm running this on docker with ubuntu 22.04 LTS and I have patched > tar 1.34 with the patch I have given to you. I know that it contains > these parameters, but they are faulty - meaning the ACLs do not > preserve uid/gid in tar archive. Are you saying we need to patch tar in order for these patches to work? > Nevertheless this concerns me that opkg-build command should not > fail. Additionally I have tested yesterday running my build without > acl and xattr in DISTRO_FEATURES and it also worked for me - without > applaying acls and xattrs to tar archive. > Generally I need to reproduce your error and it seems to me that > there is no other option as to re-create exactly the same environment > on my machine locally, so I could investigate it. > > My question is how can we achive it in the simplest way? Does > autobuilder use a docker for building?Because if so, I could take > these docker scripts, run them and check what's happening. The autobuilders are standard installs of various distros. We keep them matching upstream and the list of installed software minimal. There are more issues in this patch series. For example there is this reproducibility issue: https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F117%2Fbuilds%2F3219%2Fsteps%2F13%2Flogs%2Fstdio=05%7C01%7Cp.lobacz%40welotec.com%7C4ee44b94eb4d4bf9d55d08db885d34ef%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638253706138546929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=9jOjTjLagKdq%2B%2BQAHJ%2FM%2BEiYMUQ7nIuLILV9Zuoppwk%3D=0<https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3219/steps/13/logs/stdio> which is saying there were two sets of different ipk packages produced. These are available here: https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fautobuilder.yocto.io%2Fpub%2Frepro-fail%2Foe-reproducible-20230718-bn_npkdc%2Fpackages%2F=05%7C01%7Cp.lobacz%40welotec.com%7C4ee44b94eb4d4bf9d55d08db885d34ef%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638253706138546929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=ZJfNYwp%2FbyGN30IVyPJNIBGA2C5SV%2F1p7QHhSsIxqK8%3D=0<http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230718-bn_npkdc/packages/> sadly the automatic diffoscope output wasn't generated. Worryingly this report suggests the opkg-build change is not deterministic. FWIW this is just with the opkg-build change and the bitbake.conf change, not the other patches. These two changes alone also cause these warnings: https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F76%2Fbuilds%2F7436%2Fsteps%2F12%2Flogs%2Fstdio=05%7C01%7Cp.lobacz%40welotec.com%7C4ee44b94eb4d4bf9d55d08db885d34ef%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638253706138546929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=k1qJppjBWBpF0brnmTWfrzTSTTRCdaXTaTRaOagRfug%3D=0<https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/7436/steps/12/logs/stdio> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F76%2Fbuilds%2F7436%2Fsteps%2F18%2Flogs%2Fstdio=05%7C01%7Cp.lobacz%40welotec.com%7C4ee44b94eb4d4bf9d55d08db885d34ef%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638253706138546929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=nY9kd6CQBMfM4olhd9TRpD%2BWpIYVWDDHjJhvNQGh8Mo%3D=0<https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/7436/steps/18/logs/stdio> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F44%2Fbuilds%2F7509%2Fsteps%2F21%2Flogs%2Fstdio=05%7C01%7Cp.lobacz%40welotec.com%7C4ee44b94eb4d4bf9d55d08db885d34ef%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638253706138546929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=JZjun2owGzx%2FsCQ1sXtTsnJRWnAeohwo2e
Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
On Wed, 2023-07-19 at 07:53 +, Piotr Łobacz wrote: > I'm running this on docker with ubuntu 22.04 LTS and I have patched > tar 1.34 with the patch I have given to you. I know that it contains > these parameters, but they are faulty - meaning the ACLs do not > preserve uid/gid in tar archive. Are you saying we need to patch tar in order for these patches to work? > Nevertheless this concerns me that opkg-build command should not > fail. Additionally I have tested yesterday running my build without > acl and xattr in DISTRO_FEATURES and it also worked for me - without > applaying acls and xattrs to tar archive. > Generally I need to reproduce your error and it seems to me that > there is no other option as to re-create exactly the same environment > on my machine locally, so I could investigate it. > > My question is how can we achive it in the simplest way? Does > autobuilder use a docker for building?Because if so, I could take > these docker scripts, run them and check what's happening. The autobuilders are standard installs of various distros. We keep them matching upstream and the list of installed software minimal. There are more issues in this patch series. For example there is this reproducibility issue: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3219/steps/13/logs/stdio which is saying there were two sets of different ipk packages produced. These are available here: http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230718-bn_npkdc/packages/ sadly the automatic diffoscope output wasn't generated. Worryingly this report suggests the opkg-build change is not deterministic. FWIW this is just with the opkg-build change and the bitbake.conf change, not the other patches. These two changes alone also cause these warnings: https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/7436/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/7436/steps/18/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/7509/steps/21/logs/stdio e.g. things like: WARNING: core-image-sato-1.0-r0 do_rootfs: [log_check] core-image-sato: found 4 warning messages in the logfile: [log_check] Warning when reading ar archive header: Pathname can't be converted from UTF-8 to current locale. (errno=84) [log_check] Warning when reading ar archive header: Pathname can't be converted from UTF-8 to current locale. (errno=84) [log_check] Warning when reading ar archive header: Pathname can't be converted from UTF-8 to current locale. (errno=84) [log_check] Warning when reading ar archive header: Pathname can't be converted from UTF-8 to current locale. (errno=84) so there is a further issue to investigate there. All in all, there are a lot of issues with this patch series :(. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184569): https://lists.openembedded.org/g/openembedded-core/message/184569 Mute This Topic: https://lists.openembedded.org/mt/100231396/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
Hi Alexandre, I'm running this on docker with ubuntu 22.04 LTS and I have patched tar 1.34 with the patch I have given to you. I know that it contains these parameters, but they are faulty - meaning the ACLs do not preserve uid/gid in tar archive. Nevertheless this concerns me that opkg-build command should not fail. Additionally I have tested yesterday running my build without acl and xattr in DISTRO_FEATURES and it also worked for me - without applaying acls and xattrs to tar archive. Generally I need to reproduce your error and it seems to me that there is no other option as to re-create exactly the same environment on my machine locally, so I could investigate it. My question is how can we achive it in the simplest way? Does autobuilder use a docker for building? Because if so, I could take these docker scripts, run them and check what's happening. BR Piotr Od: Alexandre Belloni Wysłane: Tuesday, July 18, 2023 11:05:15 PM Do: Piotr Łobacz DW: Alex Stewart ; openembedded-core@lists.openembedded.org Temat: Re: ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support On 18/07/2023 08:05:12+, Piotr Łobacz wrote: > Alexander, this message: > > > Alex, > > from what I'm seeing the issue touches opkg-build command: > > > > opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" > > nativesdk-xcb-proto-dbg > > /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' > > returned non-zero exit status 1. > > > > which causes you an error. This may happen with bad tar hosttools command. > > Can you please post me which version is on yocto autobuilder? > > > BR > > Piotr > > was meant for you, sorry for the confusion. Can you please verify/check what > version of `tar` is being used on the autobuilder? This is really important > for me, if we're going to move forward with it, because I have suspicions > that it may not support posix or it may not be patched with --acls and > --xattrs attributes > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fbug-tar%40gnu.org%2Fmsg06198.html=05%7C01%7Cp.lobacz%40welotec.com%7Cc1ceda9eea4c4969c45408db87d2b6a1%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638253111303120469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=a2OBzWVj%2B2hsqeRdq3s2S9F59GX002Swf%2FyYlO%2B7bZg%3D=0<https://www.mail-archive.com/bug-tar@gnu.org/msg06198.html> > This fails at least on: fedora38-ty-4: tar (GNU tar) 1.34, has --acls and --xattrs stream8-ty-1: tar (GNU tar) 1.30, has --acls and --xattrs rocky9-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs debian12-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2204-ty-3: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2004-arm-1: tar (GNU tar) 1.30, has --acls and --xattrs opensuse154-ty-3: tar (GNU tar) 1.34, has --acls and --xattrs alma9-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2210-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2004-ty-1: tar (GNU tar) 1.30, has --acls and --xattrs Really, the question is more on which host this is working. -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbootlin.com%2F=05%7C01%7Cp.lobacz%40welotec.com%7Cc1ceda9eea4c4969c45408db87d2b6a1%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638253111303120469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=yO%2BFc0BrkTVuC4MNNDuNVErK81guyI0J%2FNxkFT6P1so%3D=0<https://bootlin.com/> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184568): https://lists.openembedded.org/g/openembedded-core/message/184568 Mute This Topic: https://lists.openembedded.org/mt/100231396/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
Ok, this is odd. Can you tell me how can I reproduce this, step by step on my local machine? Because I have to admit that it is not happening for me… BR Piotr Wysyłane z aplikacji Outlook dla systemu iOS<https://aka.ms/o0ukef> Od: Alexandre Belloni Wysłane: Sunday, July 16, 2023 11:38:57 PM Do: Piotr Łobacz DW: Alex Stewart ; openembedded-core@lists.openembedded.org Temat: Re: ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support Hello, On 14/07/2023 10:14:50+, Piotr Łobacz wrote: > OK so, does any one have any thoughts regarding this patchset? This still fails on the autobuilder, most of the builds failed: https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F37%2Fbuilds%2F7477%2Fsteps%2F11%2Flogs%2Fstdio=05%7C01%7Cp.lobacz%40welotec.com%7C10c1276b3fd543cc67b708db8645103e%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638251403412289373%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=zXRK85lPPyJLxmXEhO%2BgnfIiGFrOgxG5caBoqkHILO8%3D=0<https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/7477/steps/11/logs/stdio> ERROR: nativesdk-xcb-proto-1.15.2-r0 do_package_write_ipk: Fatal errors occurred in subprocesses: Command 'PATH="/home/pokybuild/yocto-worker/genericx86-64/build/scripts/nativesdk-intercept:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/chrpath-native:/home/pokybuild/yocto-worker/genericx86-64/build/scripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/i686-pokysdk-linux:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/usr/bin/crossscripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/genericx86-64/build/bitbake/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/hosttools" opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" nativesdk-python-xcbgen /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' returned non-zero exit status 1. Subprocess output:Usage: /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/opkg-build [-A] [-X] [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] [] Command 'PATH="/home/pokybuild/yocto-worker/genericx86-64/build/scripts/nativesdk-intercept:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/chrpath-native:/home/pokybuild/yocto-worker/genericx86-64/build/scripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/i686-pokysdk-linux:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/usr/bin/crossscripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-work
Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
ACK this patchset, assuming the other maintainers are happy with how
we're handling the DISTO_FEATURES.
I chose not to include Piotr's opkg and opkg-utils ACL/xattr changes
into the 0.6.2 release, which I have just pushed. They've been merged
afterward, and can still be applied atop 0.6.2, and they'll be merged
into the December opkg release.
I have a branch ready to upgrade the opkg and opkg-utils recipes, but
I'd like to wait for this patchset to clear, because they will otherwise
conflict.
On 7/12/23 15:04, Piotr Łobacz via lists.openembedded.org wrote:
Include support for ACLs and extended file attributes for native
builds, by default.
Signed-off-by: Piotr Łobacz
---
meta/conf/bitbake.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9625a6fef4..8dd615 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -904,7 +904,7 @@ IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"
# Native distro features (will always be used for -native, even if they
# are not enabled for target)
-DISTRO_FEATURES_NATIVE ?= "x11 ipv6 xattr"
+DISTRO_FEATURES_NATIVE ?= "acl x11 ipv6 xattr"
DISTRO_FEATURES_NATIVESDK ?= "x11"
# Normally target distro features will not be applied to native builds:
--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)
alex.stew...@ni.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184228):
https://lists.openembedded.org/g/openembedded-core/message/184228
Mute This Topic: https://lists.openembedded.org/mt/100106217/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
