Re: [OE-core] [PATCH 00/38] Morty Next pull request

2018-04-11 Thread Richard Purdie 
On Wed, 2018-04-11 at 17:07 +0200, Martin Jansa wrote:
> I think I've found the reason why it was failing for me.
> 
> glibc is now the only component in morty release which is using
> TMPDIR/sysroots-components (normally used only after RSS):
> oe-core$ git grep "\${STAGING_DIR}-components"
> meta/recipes-core/glibc/glibc-locale.inc:LOCALETREESRC =
> "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-locale"
> meta/recipes-core/glibc/glibc-mtrace.inc:SRC = "${STAGING_DIR}-
> components/${PACKAGE_ARCH}/glibc-stash-locale/scripts"
> meta/recipes-core/glibc/glibc-package.inc:do_stash_locale[sstate-
> outputdirs] = "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-
> locale"
> meta/recipes-core/glibc/glibc-package.inc:do_stash_locale[sstate-
> fixmedir] = "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-
> locale"
> meta/recipes-core/glibc/glibc-scripts.inc:SRC = "${STAGING_DIR}-
> components/${PACKAGE_ARCH}/glibc-stash-locale/scripts"
> 
> and the workspace of my jenkins jobs is still running test-
> dependencies jobs for morty, which does only selective pruning of
> TMPDIR:
> http://git.openembedded.org/openembedded-core/tree/scripts/test-depen
> dencies.sh#n165
> 
> which left TMPDIR/sysroots-components files from
> qlibc.do_stash_locale behind and when next build started in the same
> workspace it failed, because it was trying to overwrite files already
> there.
> 
> Every time I wiped whole TMPDIR and restarted the jobs it worked for
> a while and then it got broken again during first test-dependencies
> job in each workspace.
> 
> I can obviously change test-dependencies.sh to
> delete TMPDIR/sysroots-components as well and I'll send update for it
> after a bit more testing.
> 
> Ross's oe-core/scripts/wipe-sysroot and maybe some other
> scripts/tools need similar fix as well.

This starts to make more sense now, thanks for looking into it. We
could change this in morty to use sysroots, that should be safe...

Cheers,

Richard
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 00/38] Morty Next pull request

2018-04-11 Thread Martin Jansa 
I think I've found the reason why it was failing for me.

glibc is now the only component in morty release which is using
TMPDIR/sysroots-components (normally used only after RSS):
oe-core$ git grep "\${STAGING_DIR}-components"
meta/recipes-core/glibc/glibc-locale.inc:LOCALETREESRC =
"${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-locale"
meta/recipes-core/glibc/glibc-mtrace.inc:SRC =
"${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-locale/scripts"
meta/recipes-core/glibc/glibc-package.inc:do_stash_locale[sstate-outputdirs]
= "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-locale"
meta/recipes-core/glibc/glibc-package.inc:do_stash_locale[sstate-fixmedir]
= "${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-locale"
meta/recipes-core/glibc/glibc-scripts.inc:SRC =
"${STAGING_DIR}-components/${PACKAGE_ARCH}/glibc-stash-locale/scripts"

and the workspace of my jenkins jobs is still running test-dependencies
jobs for morty, which does only selective pruning of TMPDIR:
http://git.openembedded.org/openembedded-core/tree/scripts/test-dependencies.sh#n165

which left TMPDIR/sysroots-components files from qlibc.do_stash_locale
behind and when next build started in the same workspace it failed, because
it was trying to overwrite files already there.

Every time I wiped whole TMPDIR and restarted the jobs it worked for a
while and then it got broken again during first test-dependencies job in
each workspace.

I can obviously change test-dependencies.sh to
delete TMPDIR/sysroots-components as well and I'll send update for it after
a bit more testing.

Ross's oe-core/scripts/wipe-sysroot and maybe some other scripts/tools need
similar fix as well.

Regards,


On Fri, Apr 6, 2018 at 4:15 PM, Martin Jansa  wrote:

> In this build I was starting with empty TMPDIR, there shouldn't be any old
> files, but will retry manually in different builds.
>
> On Fri, Apr 6, 2018 at 4:11 PM, Richard Purdie  linuxfoundation.org> wrote:
>
>> On Fri, 2018-04-06 at 16:07 +0200, Martin Jansa wrote:
>> > > Is it safe to backport to morty, which doesn't have RSS?
>> >
>> > It doesn't work well for me, all the builds are now failing with:
>> >
>> > ERROR: glibc-2.24-r0 do_stash_locale_setscene: The recipe glibc is
>> > trying to install files into a shared area when those files already
>> > exist. Those files and their manifest location are:
>> >TMPDIR/sysroots-components/aarch64/glibc-stash-locale/SUPPORTED
>> >  Matched in b''
>> >TMPDIR/sysroots-components/aarch64/glibc-stash-
>> > locale/lib/libnss_nisplus.so.2
>> >  Matched in b''
>> >TMPDIR/sysroots-components/aarch64/glibc-stash-locale/lib/ld-
>> > 2.24.so
>> >  Matched in b''
>> > ...
>>
>> I suspect using its own sstate task is ok in morty even if it didn't
>> have RSS. What may have broken is the existing TMPDIR upgrade path
>> though :(
>>
>> If the files are removed, I suspect things will then work out ok. There
>> were manifest handling changes in later versions which dealt with this
>> kind of issue better (removals before installations).
>>
>> Cheers,
>>
>> Richard
>>
>
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 00/38] Morty Next pull request

2018-04-06 Thread Martin Jansa 
In this build I was starting with empty TMPDIR, there shouldn't be any old
files, but will retry manually in different builds.

On Fri, Apr 6, 2018 at 4:11 PM, Richard Purdie <
richard.pur...@linuxfoundation.org> wrote:

> On Fri, 2018-04-06 at 16:07 +0200, Martin Jansa wrote:
> > > Is it safe to backport to morty, which doesn't have RSS?
> >
> > It doesn't work well for me, all the builds are now failing with:
> >
> > ERROR: glibc-2.24-r0 do_stash_locale_setscene: The recipe glibc is
> > trying to install files into a shared area when those files already
> > exist. Those files and their manifest location are:
> >TMPDIR/sysroots-components/aarch64/glibc-stash-locale/SUPPORTED
> >  Matched in b''
> >TMPDIR/sysroots-components/aarch64/glibc-stash-
> > locale/lib/libnss_nisplus.so.2
> >  Matched in b''
> >TMPDIR/sysroots-components/aarch64/glibc-stash-locale/lib/ld-
> > 2.24.so
> >  Matched in b''
> > ...
>
> I suspect using its own sstate task is ok in morty even if it didn't
> have RSS. What may have broken is the existing TMPDIR upgrade path
> though :(
>
> If the files are removed, I suspect things will then work out ok. There
> were manifest handling changes in later versions which dealt with this
> kind of issue better (removals before installations).
>
> Cheers,
>
> Richard
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 00/38] Morty Next pull request

2018-04-06 Thread Richard Purdie 
On Fri, 2018-04-06 at 16:07 +0200, Martin Jansa wrote:
> > Is it safe to backport to morty, which doesn't have RSS?
> 
> It doesn't work well for me, all the builds are now failing with:
> 
> ERROR: glibc-2.24-r0 do_stash_locale_setscene: The recipe glibc is
> trying to install files into a shared area when those files already
> exist. Those files and their manifest location are:
>    TMPDIR/sysroots-components/aarch64/glibc-stash-locale/SUPPORTED
>  Matched in b''
>    TMPDIR/sysroots-components/aarch64/glibc-stash-
> locale/lib/libnss_nisplus.so.2
>  Matched in b''
>    TMPDIR/sysroots-components/aarch64/glibc-stash-locale/lib/ld-
> 2.24.so
>  Matched in b''
> ...

I suspect using its own sstate task is ok in morty even if it didn't
have RSS. What may have broken is the existing TMPDIR upgrade path
though :(

If the files are removed, I suspect things will then work out ok. There
were manifest handling changes in later versions which dealt with this
kind of issue better (removals before installations).

Cheers,

Richard
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 00/38] Morty Next pull request

2018-04-06 Thread Martin Jansa 
> Is it safe to backport to morty, which doesn't have RSS?

It doesn't work well for me, all the builds are now failing with:

ERROR: glibc-2.24-r0 do_stash_locale_setscene: The recipe glibc is trying
to install files into a shared area when those files already exist. Those
files and their manifest location are:
   TMPDIR/sysroots-components/aarch64/glibc-stash-locale/SUPPORTED
 Matched in b''
   TMPDIR/
sysroots-components/aarch64/glibc-stash-locale/lib/libnss_nisplus.so.2
 Matched in b''
   TMPDIR/sysroots-components/aarch64/glibc-stash-locale/lib/ld-2.24.so
 Matched in b''
...



On Thu, Apr 5, 2018 at 11:41 PM, Andre McCurdy  wrote:

> On Thu, Apr 5, 2018 at 6:53 AM, Armin Kuster  wrote:
> > Please consider these changes for Morty. Clean AB build
> >
> > This included sdk locale and host glibc 2.27 fixes.
> >
> > The following changes since commit 34b26ec18450a3bc72f7b8a0f2df0b
> faae1140dd:
> >
> >   linux-firmware: Add reference to iwlwifi-8000C firmware (2018-03-19
> 15:58:33 +)
> >
> > are available in the git repository at:
> >
> >   git://git.yoctoproject.org/poky-contrib stable/morty-next
> >   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=
> stable/morty-next
> >
> > Richard Purdie (4):
> >   lib/oe/package_manager/sdk: Ensure do_populate_sdk_ext and
> > do_populate_sdk repos don't conflict
> >   glibc: Separate locale files to their own sstate task
>
> This commit comes with the comment:
>
> Putting the locale and script files into the sysroot for use by their
> specific recipes used to be a simple way to share the files. With RSS,
> we don't want to copy these into many different recipes so put these
> files in their own sstate task.
>
> Is it safe to backport to morty, which doesn't have RSS?
>
> >   image/packagegroup/populate_sdk: Drop do_populate_sysroot task
> > properly
> >   glibc-package: Avoid race sstate races with do_stash_locale
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 00/38] Morty Next pull request

2018-04-05 Thread Andre McCurdy 
On Thu, Apr 5, 2018 at 6:53 AM, Armin Kuster  wrote:
> Please consider these changes for Morty. Clean AB build
>
> This included sdk locale and host glibc 2.27 fixes.
>
> The following changes since commit 34b26ec18450a3bc72f7b8a0f2df0bfaae1140dd:
>
>   linux-firmware: Add reference to iwlwifi-8000C firmware (2018-03-19 
> 15:58:33 +)
>
> are available in the git repository at:
>
>   git://git.yoctoproject.org/poky-contrib stable/morty-next
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stable/morty-next
>
> Richard Purdie (4):
>   lib/oe/package_manager/sdk: Ensure do_populate_sdk_ext and
> do_populate_sdk repos don't conflict
>   glibc: Separate locale files to their own sstate task

This commit comes with the comment:

Putting the locale and script files into the sysroot for use by their
specific recipes used to be a simple way to share the files. With RSS,
we don't want to copy these into many different recipes so put these
files in their own sstate task.

Is it safe to backport to morty, which doesn't have RSS?

>   image/packagegroup/populate_sdk: Drop do_populate_sysroot task
> properly
>   glibc-package: Avoid race sstate races with do_stash_locale
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 00/38] Morty Next pull request

2018-04-05 Thread akuster808 
Ignore this request.

- armin


On 04/05/2018 06:53 AM, Armin Kuster wrote:
> Please consider these changes for Morty. Clean AB build
>
> This included sdk locale and host glibc 2.27 fixes.

>
> The following changes since commit 34b26ec18450a3bc72f7b8a0f2df0bfaae1140dd:
>
>   linux-firmware: Add reference to iwlwifi-8000C firmware (2018-03-19 
> 15:58:33 +)
>
> are available in the git repository at:
>
>   git://git.yoctoproject.org/poky-contrib stable/morty-next
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stable/morty-next
>
> Alexander Kanavin (2):
>   selftest/signing: add --batch to gpg invocation when importing keys
>   populate_sdk_base.bbclass: add documentation packages to SDKs if
> api-documentation is in DISTRO_FEATURES
>
> Andreas Oberritter (1):
>   libc-package.bbclass: split binary localedata even more if asked to
>
> Armin Kuster (3):
>   lib/oe/terminal.py: use an absolute path to execute
> oe-gnome-terminal-phonehome
>   distcc: Change SRC_URI
>   package_manager: fix install_glob to use morty sdk pkgdir
>
> Christopher Larson (1):
>   buildhistory: fix latest_srcrev in the common case
>
> Ed Bartosh (1):
>   toolchain-shar-extract: compare SDK and host gcc versions
>
> Gianfranco Costamagna (1):
>   glibc-locale.inc: fix typo in comment
>
> Juro Bystricky (1):
>   glibc-locale.inc: avoid duplicate packages
>
> Linus Wallgren (1):
>   lib/oe/package_manager: .deb pre/postinst args
>
> Martin Jansa (1):
>   package_manager: flush installed_pkgs file before oe-pkgdata-util uses
> it
>
> Maxin B. John (1):
>   grub-efi: fix build failure
>
> Richard Purdie (4):
>   lib/oe/package_manager/sdk: Ensure do_populate_sdk_ext and
> do_populate_sdk repos don't conflict
>   glibc: Separate locale files to their own sstate task
>   image/packagegroup/populate_sdk: Drop do_populate_sysroot task
> properly
>   glibc-package: Avoid race sstate races with do_stash_locale
>
> Ross Burton (15):
>   default-distrovars: don't rename locales for nativesdk
>   glibc: don't use host locales in nativesdk
>   glibc: relocate locale paths in nativesdk
>   lib/oe/package_manager: bail if createrepo can't be found
>   package_manager: don't race on a file when installing complementary
> packages
>   package_manager: improve install_complementary
>   package-manager: add install_glob()
>   sdk: install specified locales into SDK
>   sdk: only install locales if we're using glibc
>   populate_sdk: install UTF-8 locales in SDKs
>   populate_sdk_base: depend on nativesdk-glibc-locale
>   cross-localedef-native: add way to specify which locale archive to
> write
>   qemu: fix memfd_create with glibc 2.27
>   sdk: generate locale archive and remove packages
>   sdk: streamline locale removal
>
> Stephano Cetola (1):
>   package_manager: remove strings and migrate to direct arrays
>
> Tanu Kaskinen (4):
>   e2fsprogs: fix compatibility with glibc 2.27
>   libvorbis: CVE-2017-14633
>   libvorbis: CVE-2017-14632
>   libvorbis: CVE-2018-5146
>
> Zhenbo Gao (1):
>   archiver: avoid archiving source for glibc-locale
>
>  meta/classes/archiver.bbclass  |   6 +
>  meta/classes/buildhistory.bbclass  |   2 +-
>  meta/classes/image.bbclass |   2 +-
>  meta/classes/libc-package.bbclass  |  39 ++-
>  meta/classes/packagegroup.bbclass  |   2 +-
>  meta/classes/populate_sdk_base.bbclass |  11 +-
>  meta/classes/populate_sdk_ext.bbclass  |   3 +-
>  meta/conf/distro/include/default-distrovars.inc|   1 +
>  meta/files/toolchain-shar-extract.sh   |   8 +
>  meta/lib/oe/package.py |  13 +-
>  meta/lib/oe/package_manager.py | 328 
> +++--
>  meta/lib/oe/sdk.py |  85 +-
>  meta/lib/oe/terminal.py|   2 +-
>  meta/lib/oeqa/selftest/signing.py  |   2 +-
>  ...ix-check-for-sys-sysmacros.h-under-glibc-.patch |  48 +++
>  meta/recipes-bsp/grub/grub-efi_2.00.bb |   1 +
>  .../glibc/cross-localedef-native_2.24.bb   |   1 +
>  meta/recipes-core/glibc/glibc-collateral.inc   |   2 +-
>  meta/recipes-core/glibc/glibc-initial.inc  |   2 +-
>  meta/recipes-core/glibc/glibc-locale.inc   |   5 +-
>  meta/recipes-core/glibc/glibc-mtrace.inc   |   2 +-
>  meta/recipes-core/glibc/glibc-package.inc  |  45 +--
>  meta/recipes-core/glibc/glibc-scripts.inc  |   2 +-
>  meta/recipes-core/glibc/glibc/archive-path.patch   |  39 +++
>  .../glibc/glibc/relocate-locales.patch |  33 +++
>  meta/recipes-core/glibc/glibc_2.24.bb  |   7 +-
>  meta/recipes-core/meta/buildtools-tarball.bb   |   1 -
>  meta/recipes-devtools/distcc/distcc_3.2.bb |   2 +-
>  ...rename-copy_file_range-to-copy_file_chunk.patch |  62 
>