ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
HI all, Hi Richard thx for quick response. Generally this patch for tar has been already applied to the upstream http://git.savannah.gnu.org/cgit/tar.git/commit/?id=5461025569c2d946fb31b79f16f60e923bbd79f9 Additionally a new version 1.35 has been released which has this fix applied as well. > Are you saying we need to patch tar in order for these patches to work? Unfortunately yes, but still this error, which occurs on autobuilder should not happen, because the patch for ACLs and xattrs only changes the writing algorithm to the tar file - meaning, when --numeric-owner parameter is being used all uid(s)/gid(s) are written with numbers instead of names. > WARNING: core-image-sato-1.0-r0 do_rootfs: [log_check] core-image-sato: found > 4 warning messages in the logfile: > [log_check] Warning when reading ar archive header: Pathname can't be > converted from UTF-8 to current locale. (errno=84) > [log_check] Warning when reading ar archive header: Pathname can't be > converted from UTF-8 to current locale. (errno=84) > [log_check] Warning when reading ar archive header: Pathname can't be > converted from UTF-8 to current locale. (errno=84) > [log_check] Warning when reading ar archive header: Pathname can't be > converted from UTF-8 to current locale. (errno=84) This one is also occurring on our side, but I thought that it is irrelevant as it's just a warning and everything is working fine. Generally I have investigated it and it occurs that this error occurs from opkg source code https://git.yoctoproject.org/opkg/tree/libopkg/opkg_archive.c#n272 I added a debug code in here like that: opkg_msg(NOTICE, "Warning when reading ar archive header: %s (errno=%d) LC_CTYPE=%s LC_ALL=%s\n", archive_error_string(ar), archive_errno(ar), setlocale(LC_CTYPE, NULL), setlocale(LC_ALL, NULL)); to actually see what are the LC values and it occurred that it is equal C. Afterwards my investigations were focused on whom is calling the opkg command, maybe it is changing somehow locales and this is being executed in here http://git.openembedded.org/openembedded-core/tree/meta/lib/oe/package_manager/ipk/__init__.py#n368 but to my surprise LC_ALL which is being passed through os.environ is proper and equals LC_ALL: 'en_US.UTF-8' My guess is that opkg runs fork process to actually install all depends, because in log.do_rootfs file from this http://git.openembedded.org/openembedded-core/tree/meta/lib/oe/package_manager/ipk/__init__.py#n366 line I don't see the whole list of all packages that are being installed. A stupid fix for that was to set LC_ALL in opkg with: setlocale(LC_ALL, "en_US.UTF-8"); to check if it fixes the issue and yes it fixes but this is no go. Question is probably now to Alex does opkg can run fork instances for depends? Because I have found that it uses this vfork, however I dunno if it uses parent's process env variables or it's own. I think that this needs further investigations... BR Piotr Od: Richard Purdie Wysłane: środa, 19 lipca 2023 10:37 Do: Piotr Łobacz ; Alexandre Belloni DW: Alex Stewart ; openembedded-core@lists.openembedded.org Temat: Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support On Wed, 2023-07-19 at 07:53 +, Piotr Łobacz wrote: > I'm running this on docker with ubuntu 22.04 LTS and I have patched > tar 1.34 with the patch I have given to you. I know that it contains > these parameters, but they are faulty - meaning the ACLs do not > preserve uid/gid in tar archive. Are you saying we need to patch tar in order for these patches to work? > Nevertheless this concerns me that opkg-build command should not > fail. Additionally I have tested yesterday running my build without > acl and xattr in DISTRO_FEATURES and it also worked for me - without > applaying acls and xattrs to tar archive. > Generally I need to reproduce your error and it seems to me that > there is no other option as to re-create exactly the same environment > on my machine locally, so I could investigate it. > > My question is how can we achive it in the simplest way? Does > autobuilder use a docker for building?Because if so, I could take > these docker scripts, run them and check what's happening. The autobuilders are standard installs of various distros. We keep them matching upstream and the list of installed software minimal. There are more issues in this patch series. For example there is this reproducibility issue: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3219/steps/13/logs/stdio which is saying there were two sets of different ipk packages produced. These are available here: http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230718-bn_npkdc/packages/ sadly the automatic diffoscope output wasn't generated. Worryingly this report suggests the opkg-build change is not deterministic. FWIW this is just with the opkg-build change and the bitbake.conf change, not the other
Re: ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
On 18/07/2023 08:05:12+, Piotr Łobacz wrote: > Alexander, this message: > > > Alex, > > from what I'm seeing the issue touches opkg-build command: > > > > opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" > > nativesdk-xcb-proto-dbg > > /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' > > returned non-zero exit status 1. > > > > which causes you an error. This may happen with bad tar hosttools command. > > Can you please post me which version is on yocto autobuilder? > > > BR > > Piotr > > was meant for you, sorry for the confusion. Can you please verify/check what > version of `tar` is being used on the autobuilder? This is really important > for me, if we're going to move forward with it, because I have suspicions > that it may not support posix or it may not be patched with --acls and > --xattrs attributes https://www.mail-archive.com/bug-tar@gnu.org/msg06198.html > This fails at least on: fedora38-ty-4: tar (GNU tar) 1.34, has --acls and --xattrs stream8-ty-1: tar (GNU tar) 1.30, has --acls and --xattrs rocky9-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs debian12-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2204-ty-3: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2004-arm-1: tar (GNU tar) 1.30, has --acls and --xattrs opensuse154-ty-3: tar (GNU tar) 1.34, has --acls and --xattrs alma9-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2210-ty-1: tar (GNU tar) 1.34, has --acls and --xattrs ubuntu2004-ty-1: tar (GNU tar) 1.30, has --acls and --xattrs Really, the question is more on which host this is working. -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184559): https://lists.openembedded.org/g/openembedded-core/message/184559 Mute This Topic: https://lists.openembedded.org/mt/100138221/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
Alexander, this message: > Alex, > from what I'm seeing the issue touches opkg-build command: > > opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" nativesdk-xcb-proto-dbg > /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' > returned non-zero exit status 1. > > which causes you an error. This may happen with bad tar hosttools command. > Can you please post me which version is on yocto autobuilder? > BR > Piotr was meant for you, sorry for the confusion. Can you please verify/check what version of `tar` is being used on the autobuilder? This is really important for me, if we're going to move forward with it, because I have suspicions that it may not support posix or it may not be patched with --acls and --xattrs attributes https://www.mail-archive.com/bug-tar@gnu.org/msg06198.html BR Piotr -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184497): https://lists.openembedded.org/g/openembedded-core/message/184497 Mute This Topic: https://lists.openembedded.org/mt/100138221/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
Alex, from what I'm seeing the issue touches opkg-build command: opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" nativesdk-xcb-proto-dbg /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' returned non-zero exit status 1. which causes you an error. This may happen with bad tar hosttools command. Can you please post me which version is on yocto autobuilder? BR Piotr Od: openembedded-core@lists.openembedded.org w imieniu użytkownika Piotr Łobacz via lists.openembedded.org Wysłane: poniedziałek, 17 lipca 2023 07:07 Do: Alexandre Belloni DW: Alex Stewart ; openembedded-core@lists.openembedded.org Temat: Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support Ok, this is odd. Can you tell me how can I reproduce this, step by step on my local machine? Because I have to admit that it is not happening for me… BR Piotr Wysyłane z aplikacji Outlook dla systemu iOS<https://aka.ms/o0ukef> Od: Alexandre Belloni Wysłane: Sunday, July 16, 2023 11:38:57 PM Do: Piotr Łobacz DW: Alex Stewart ; openembedded-core@lists.openembedded.org Temat: Re: ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support Hello, On 14/07/2023 10:14:50+, Piotr Łobacz wrote: > OK so, does any one have any thoughts regarding this patchset? This still fails on the autobuilder, most of the builds failed: https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F37%2Fbuilds%2F7477%2Fsteps%2F11%2Flogs%2Fstdio=05%7C01%7Cp.lobacz%40welotec.com%7C10c1276b3fd543cc67b708db8645103e%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638251403412289373%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=zXRK85lPPyJLxmXEhO%2BgnfIiGFrOgxG5caBoqkHILO8%3D=0<https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/7477/steps/11/logs/stdio> ERROR: nativesdk-xcb-proto-1.15.2-r0 do_package_write_ipk: Fatal errors occurred in subprocesses: Command 'PATH="/home/pokybuild/yocto-worker/genericx86-64/build/scripts/nativesdk-intercept:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/chrpath-native:/home/pokybuild/yocto-worker/genericx86-64/build/scripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/i686-pokysdk-linux:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/usr/bin/crossscripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/genericx86-64/build/bitbake/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/hosttools" opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" nativesdk-python-xcbgen /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' returned non-zero exit status 1. Subprocess output:Usage: /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/opkg-build [-A] [-X] [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] [] Command 'PATH="/home/pokybuild/yocto-worker/genericx86-64/build/scripts/nativesdk-intercept:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesd
Re: ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
Hello, On 14/07/2023 10:14:50+, Piotr Łobacz wrote: > OK so, does any one have any thoughts regarding this patchset? This still fails on the autobuilder, most of the builds failed: https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/7477/steps/11/logs/stdio ERROR: nativesdk-xcb-proto-1.15.2-r0 do_package_write_ipk: Fatal errors occurred in subprocesses: Command 'PATH="/home/pokybuild/yocto-worker/genericx86-64/build/scripts/nativesdk-intercept:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/chrpath-native:/home/pokybuild/yocto-worker/genericx86-64/build/scripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/i686-pokysdk-linux:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/usr/bin/crossscripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/genericx86-64/build/bitbake/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/hosttools" opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" nativesdk-python-xcbgen /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' returned non-zero exit status 1. Subprocess output:Usage: /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/opkg-build [-A] [-X] [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] [] Command 'PATH="/home/pokybuild/yocto-worker/genericx86-64/build/scripts/nativesdk-intercept:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/chrpath-native:/home/pokybuild/yocto-worker/genericx86-64/build/scripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/i686-pokysdk-linux:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot/usr/local/oe-sdk-hardcoded-buildpath/sysroots/i686-pokysdk-linux/usr/bin/crossscripts:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/genericx86-64/build/bitbake/bin:/home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/hosttools" opkg-build -Z xz -a "--memlimit=5% --threads=8" "" "" nativesdk-xcb-proto-dev /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/deploy-ipks/i686-nativesdk' returned non-zero exit status 1. Subprocess output:Usage: /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/i686-nativesdk-pokysdk-linux/nativesdk-xcb-proto/1.15.2-r0/recipe-sysroot-native/usr/bin/opkg-build [-A] [-X] [-c] [-C] [-Z compressor] [-a compressor_args] [-O] [-o owner] [-g group] [] Command
ODP: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support
OK so, does any one have any thoughts regarding this patchset? Od: Alex Stewart Wysłane: czwartek, 13 lipca 2023 00:04 Do: Piotr Łobacz ; openembedded-core@lists.openembedded.org Temat: Re: [OE-Core][PATCH v5 1/5] bitbake.conf: add acl and xattr distro native features support ACK this patchset, assuming the other maintainers are happy with how we're handling the DISTO_FEATURES. I chose not to include Piotr's opkg and opkg-utils ACL/xattr changes into the 0.6.2 release, which I have just pushed. They've been merged afterward, and can still be applied atop 0.6.2, and they'll be merged into the December opkg release. I have a branch ready to upgrade the opkg and opkg-utils recipes, but I'd like to wait for this patchset to clear, because they will otherwise conflict. On 7/12/23 15:04, Piotr Łobacz via lists.openembedded.org wrote: > Include support for ACLs and extended file attributes for native > builds, by default. > > Signed-off-by: Piotr Łobacz > --- > meta/conf/bitbake.conf | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 9625a6fef4..8dd615 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -904,7 +904,7 @@ IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}" > > # Native distro features (will always be used for -native, even if they > # are not enabled for target) > -DISTRO_FEATURES_NATIVE ?= "x11 ipv6 xattr" > +DISTRO_FEATURES_NATIVE ?= "acl x11 ipv6 xattr" > DISTRO_FEATURES_NATIVESDK ?= "x11" > > # Normally target distro features will not be applied to native builds: > > > -- Alex Stewart Software Engineer - NI Real-Time OS NI (National Instruments) alex.stew...@ni.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#184256): https://lists.openembedded.org/g/openembedded-core/message/184256 Mute This Topic: https://lists.openembedded.org/mt/100138221/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-