Re: [OE-core] [PATCH] [zeus] aspell: CVE-2019-20433

[Mikko.Rapeli]

Yes, you are correct. White listing isn't right either.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] [zeus] aspell: CVE-2019-20433

[Mikko.Rapeli]

On Thu, Mar 12, 2020 at 12:25:21PM +, Mittal, Anuj wrote:
> It looks like this is changing the API. I wonder if this would need any
> other change or break something elsewhere in OE-core, meta-oe?
> 
> http://aspell.net/buffer-overread-ucs.txt

Debian classified issues as minor and fixed only by updating
to 0.60.8:

https://security-tracker.debian.org/tracker/CVE-2019-20433

https://metadata.ftp-master.debian.org/changelogs//main/a/aspell/aspell_0.60.8-1_changelog

Maybe whitelist for stable branches and update to new version on master?

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] how to cleanly centralize a zillion BBCLASSEXTEND += "nativesdk" settings?

[Mikko.Rapeli]

On Mon, Mar 02, 2020 at 07:40:35AM -0500, rpj...@crashcourse.ca wrote:
> layer i'm currently perusing has many, many bbappend files, of which
> quite a number are nothing more than the single line:
> 
>   BBCLASSEXTEND += "nativesdk"
> 
> literally, at least a hundred append files are like that. so rather
> than clutter up the layer with all those trivial .bbappend files,
> can one cram all that into a single .conf file? as i read it, can i
> do something like:
> 
>   BBCLASSEXTEND_append_pn-pkg1 = " nativesdk"
>   BBCLASSEXTEND_append_pn-pkg2 = " nativesdk"
>   ... and on and on ...
> 
> and toss all that into a distro.conf file or something?
> 
>   and even if that works, is it considered good coding style?

FWIW, I have the same problem. Solution until now is large number of
bbappends for adding native and nativesdk support.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [yocto] Change RO rootfs failed RF Kill Switch Status and Failed to start Run pending postinsts

[Mikko.Rapeli]

On Tue, Feb 18, 2020 at 08:43:01PM +1100, JH wrote:
> Hi Mikko,
> 
> On 2/18/20, mikko.rap...@bmw.de  wrote:
> > I think you may be missing volatile-binds package and service from your
> > image.
> > See poky/meta/recipes-core/volatile-binds/volatile-binds.bb
> 
> I got the source in my build system, it is zeus:
> 
> oe-core/meta/recipes-core/volatile-binds/volatile-binds.bb
> 
> ./all-oe-linux/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/packages-split/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/sysroot-destdir/sysroot-providers/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/pkgdata-pdata-input/runtime/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/pkgdata-pdata-input/runtime-reverse/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/pkgdata-pdata-input/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/pkgdata/runtime/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/pkgdata/runtime-reverse/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/pkgdata/volatile-binds
> ./all-oe-linux/volatile-binds/1.0-r0/license-destdir/volatile-binds
> 
> Are there correct?
> 
> > It may be missing /var/log but with systemd there should not be needs to
> > write
> > to that location after image builds...
> 
> The volatile did not have the log, so /var/log -> volatile/log was an
> invalid link, should I manually create it?
> 
> Thanks Mikko,

Well I have zeus and am using read-only rootfs with volatile binds and
I did not need anything extra. I would dig into this /var/log thing
and patch it away. I use systemd journal so no need for syslogs.

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [yocto] Change RO rootfs failed RF Kill Switch Status and Failed to start Run pending postinsts

[Mikko.Rapeli]

(trimming lists to yocto only)

Hi,

I think you may be missing volatile-binds package and service from your image.
See poky/meta/recipes-core/volatile-binds/volatile-binds.bb

It may be missing /var/log but with systemd there should not be needs to write
to that location after image builds...

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2] sqlite3: upgrade 3.30.1 -> 3.31.1

[Mikko.Rapeli]

Hi,

slightly off topic but I was checking CVEs for sqlite3 and noticed
this recipe uses the merged source tree format. This makes it very
hard to cherry-pick CVE and other patches from Debian, Ubuntu,
OpenSUSE etc.

Why use sqlite sources in "amalgamation" format?

https://sqlite.org/amalgamation.html

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] util-linux: upgrade 2.34 -> 2.35

[Mikko.Rapeli]

On Tue, Jan 21, 2020 at 07:28:44PM +0100, Pierre-Jean Texier via 
Openembedded-core wrote:
> License-Update: add GPLv3 text

Which parts of util-linux are now licensed with GPLv3?

-Mikko

> Drop upstreamed patch
> 
> See full changelog 
> https://lore.kernel.org/util-linux/20200121105711.zzeeolydlivqn...@ws.net.home/T/#u
> 
> Signed-off-by: Pierre-Jean Texier 
> ---
>  meta/recipes-core/util-linux/util-linux.inc|  5 +--
>  ...lsblk-force-to-print-PKNAME-for-partition.patch | 36 
> --
>  .../{util-linux_2.34.bb => util-linux_2.35.bb} |  5 ++-
>  3 files changed, 5 insertions(+), 41 deletions(-)
>  delete mode 100644 
> meta/recipes-core/util-linux/util-linux/0001-lsblk-force-to-print-PKNAME-for-partition.patch
>  rename meta/recipes-core/util-linux/{util-linux_2.34.bb => 
> util-linux_2.35.bb} (58%)
> 
> diff --git a/meta/recipes-core/util-linux/util-linux.inc 
> b/meta/recipes-core/util-linux/util-linux.inc
> index 179cb3d..0f8a6c2 100644
> --- a/meta/recipes-core/util-linux/util-linux.inc
> +++ b/meta/recipes-core/util-linux/util-linux.inc
> @@ -6,11 +6,12 @@ disk partitioning, kernel message management, filesystem 
> creation, and system lo
>  
>  SECTION = "base"
>  
> -LICENSE = "GPLv2+ & LGPLv2.1+ & BSD-3-Clause & BSD-4-Clause"
> +LICENSE = "GPLv2+ & GPLv3+ & LGPLv2.1+ & BSD-3-Clause & BSD-4-Clause"
>  
> -LIC_FILES_CHKSUM = 
> "file://README.licensing;md5=972a134f1e14b2b060e365df2fab0099 \
> +LIC_FILES_CHKSUM = 
> "file://README.licensing;md5=0fd5c050c6187d2bf0a4492b7f4e33da \
>  file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
>  
> file://Documentation/licenses/COPYING.GPL-2.0-or-later;md5=b234ee4d69f5fce4486a80fdaf4a4263
>  \
> +
> file://Documentation/licenses/COPYING.GPL-3.0-or-later;md5=1ebbd3e34237af26da5dc08a4e440464
>  \
>  
> file://Documentation/licenses/COPYING.LGPL-2.1-or-later;md5=4fbd65380cdd255951079008b364516c
>  \
>  
> file://Documentation/licenses/COPYING.BSD-3-Clause;md5=58dcd8452651fc8b07d1f65ce07ca8af
>  \
>  
> file://Documentation/licenses/COPYING.BSD-4-Clause-UC;md5=263860f8968d8bafa5392cab74285262
>  \
> diff --git 
> a/meta/recipes-core/util-linux/util-linux/0001-lsblk-force-to-print-PKNAME-for-partition.patch
>  
> b/meta/recipes-core/util-linux/util-linux/0001-lsblk-force-to-print-PKNAME-for-partition.patch
> deleted file mode 100644
> index 5d4c148..000
> --- 
> a/meta/recipes-core/util-linux/util-linux/0001-lsblk-force-to-print-PKNAME-for-partition.patch
> +++ /dev/null
> @@ -1,36 +0,0 @@
> -From e3bb9bfb76c17b1d05814436ced62c05c4011f48 Mon Sep 17 00:00:00 2001
> -From: Karel Zak 
> -Date: Thu, 27 Jun 2019 09:22:18 +0200
> -Subject: [PATCH] lsblk: force to print PKNAME for partition
> -
> -PKNAME (parent kernel device name) is based on printed tree according
> -to parent -> child relationship. The tree is optional and not printed
> -if partition specified (.e.g "lsblk -o+PKNAME /dev/sda1"), but old
> -versions print the PKNAME also in this case.
> -
> -Upstream-Status: Backport 
> [https://github.com/karelzak/util-linux/commit/e3bb9bfb76c17b1d05814436ced62c05c4011f48]
> -
> -Addresses: https://github.com/karelzak/util-linux/issues/813
> -Signed-off-by: Karel Zak 
> -Signed-off-by: Liwei Song 
> 
> - misc-utils/lsblk.c | 3 +++
> - 1 file changed, 3 insertions(+)
> -
> -diff --git a/misc-utils/lsblk.c b/misc-utils/lsblk.c
> -index e95af7af0256..3ce6da730264 100644
>  a/misc-utils/lsblk.c
> -+++ b/misc-utils/lsblk.c
> -@@ -1019,6 +1019,9 @@ static void device_to_scols(
> - DBG(DEV, ul_debugobj(dev, "add '%s' to scols", dev->name));
> - ON_DBG(DEV, if (ul_path_isopen_dirfd(dev->sysfs)) ul_debugobj(dev, " %s 
> ---> is open!", dev->name));
> - 
> -+if (!parent && dev->wholedisk)
> -+parent = dev->wholedisk;
> -+
> - /* Do not print device more than one in --list mode */
> - if (!(lsblk->flags & LSBLK_TREE) && dev->is_printed)
> - return;
> --- 
> -2.17.1
> -
> diff --git a/meta/recipes-core/util-linux/util-linux_2.34.bb 
> b/meta/recipes-core/util-linux/util-linux_2.35.bb
> similarity index 58%
> rename from meta/recipes-core/util-linux/util-linux_2.34.bb
> rename to meta/recipes-core/util-linux/util-linux_2.35.bb
> index e9c2d80..71f7cd1 100644
> --- a/meta/recipes-core/util-linux/util-linux_2.34.bb
> +++ b/meta/recipes-core/util-linux/util-linux_2.35.bb
> @@ -7,7 +7,6 @@ SRC_URI += "file://configure-sbindir.patch \
>  file://run-ptest \
>  file://display_testname_for_subtest.patch \
>  file://avoid_parallel_tests.patch \
> -file://0001-lsblk-force-to-print-PKNAME-for-partition.patch \
>  "
> -SRC_URI[md5sum] = "a78cbeaed9c39094b96a48ba8f891d50"
> -SRC_URI[sha256sum] = 
> "743f9d0c7252b6db246b659c1e1ce0bd45d8d4508b4dfa427bbb4a3e9b9f62b5"
> +SRC_URI[md5sum] = "479488469f057ddeed0d3ca7d6ab5f16"
> +SRC_URI[sha256sum] 

Re: [OE-core] [PATCH 1/5] opkg: Add upstream fixes for empty packages

[Mikko.Rapeli]

This fixes the crash I was seeing too.

While debugging it, I noticed that host valgrind was not able to work with
yocto host tools and I had to build native valgrind which needed
some patches and workarounds. Will try to submit them...

Thanks for fixing this!

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Mikko.Rapeli]

On Wed, Nov 20, 2019 at 03:53:14PM -0800, Andre McCurdy wrote:
> But anyway, in all cases, the way to debug what's going on isn't to
> try random recipe changes and then rebuild the final image. Instead
> you should build your chosen version of openssl, look in the
> packages-split directory to see which package includes the openssl
> command line tool and then add that package to your image.

Or enable buildhistory, build openssl and/or image(s), cd build/buildhistory
and git grep for the binaries needed to find out which binary package
they belong to.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Mikko.Rapeli]

On Thu, Nov 21, 2019 at 01:05:55AM +0200, Adrian Bunk wrote:
> On Wed, Nov 20, 2019 at 09:39:51PM +, mikko.rap...@bmw.de wrote:
> >...
> > I could submit these too if someone wants to setup a communit maintenance 
> > branch for sumo.
> 
> I would not consider this appropriate for a stable branch. With such 
> invasive changes it would no longer be reasonably safe for users to 
> follow the branch to receive security updates for other recipes.
> 
> In Ubuntu 18.04 security support for OpenSSL 1.0.2 is provided until at 
> least April 2023. Similar schedules exist for other LTS distributions.
> This provides sources for piggy-backing security support for a few years
> after upstream support ends.

Yes, I agree to this. The reasons for the large intrusive backport are:

 * openssl version 1.1.0 in sumo is no longer supported by upstream
   developers, see https://www.openssl.org/policies/releasestrat.html
   "Version 1.1.0 will be supported until 2019-09-11." but 1.1.1
   is an LTS with support unit 2023-09-11

 * many recipes like openssh in sumo do not support openssl 1.1.x and an
   update is needed to cover the API breakage. The backported pathes
   fixes most of the issues in poky and meta-openembedded and I've been
   able to use the set in multiple projects with different BSP stacks.

So in sumo, openssl 1.0.2 could still be maintainable with Ubuntu etc
help even when upstream openssl.org support has now ended. Same could
apply to openssl 1.1.0 there, but if one suffers and fixes the API
changes, then it is maybe better for users to jump directly to the next
openssl 1.1.1x LTS version. The patches I mentioned achieve this,
but I agree they are intrucive and not following stable policies.

In my case, openssl 1.1.x transition is one of the major blockers
for doing more yocto updates and running closer to master. The backport
has helped there and a following jump to zeus was really straight
forward (ignoring lots of issues in BSP layers but that's life).

Then a note on openssl 1.1.x impact to various BSP layers, some scripting and
bbclasses related to signing etc may need to be updated but also
those changes are simple. I wish there was more open source community
approach so share changes like these among users of various BSPs.

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Mikko.Rapeli]

On Wed, Nov 20, 2019 at 06:18:05PM +, Ryan Harkin wrote:
> I'm struggling with backporting OpenSSL to my Sumo build [1], so wondered
> if anyone else had done something similar with success.

I've done it by backporting following changes to poky (sorry for subject only):

openssh: upgrade 7.6p1 -> 7.7p1
openssh: drop sshd support for DSA host keys
openssh: stop adding -D__FILE_OFFSET_BITS=64 to CFLAGS
openssh: drop RCONFLICTS for openssh-keygen
openssh: minor indent cleanup for sshd init script
openssh: sync local ssh_config + sshd_config files with upstream 7.7p1
openssh: only create sshd host keys which have been enabled
openssh: update from 7.7p1 to 7.8p1
openssh: upgrade 7.8p1 -> 7.8p1+git to support openssl 1.1.x
openssl-1.1: rework packaging
openssl-1.1: /etc/ssl location compatibility
openssl: minor reformatting to align the 1.0 and 1.1 recipes
openssl: move the libdir openssl.cnf symlink into the openssl package
openssl: fix path in nativesdk environment-setup script
openssl: drop obsolete no-afalgeng workaround for aarch64
openssl: fix hardcoded paths in native for openssl 1.1
openssl: remove dependency on relative_symlinks class
openssl: rename openssl 1.0.x to openssl10 and make openssl 1.1.x the default 
version
openssl: update to 1.1.1
openssl: do not tweak so names, use PRIVATE_LIBS instead
openssl: Handle -conf package file conflicts
openssl: rename PV to 1.1.1~pre9 to avoid future versions from going backwards
openssl_1.1.1: Fix Musl build by disabling async during configure
openssl: update to 1.1.1 final
openssl10: fix compile error for debian-mips64
openssl: skip ptest case `test_symbol_presence'
openssl: use deterministic perl Text::Template module bundled by openssl source
openssl: correct license comment
openssl: fix ptest
openssl: do an out-of-tree build
openssl: fix CVE-2018-0734 for both 1.0.2p and 1.1.1
openssl: fix CVE-2018-0735 for 1.1.1
openssl-1.1.1: remove build path from version info
openssl: update to 1.1.1a
openssl: correct bad path on package preprocess
python3{,-native}: backport openssl 1.1.1 compatibility changes
python3: fix openssl 1.1.1 changes
cryptodev-tests: port to openssl 1.1

Plus a patch to allow overriding openssl version in default-versions.inc,
and one hack to drop perl RDEPENDS from openssl-bin. This is still missing
the latest CVEs and letter releases.

Then meta-openembedded needed at least:

asio: Upgrade to 1.12.1
mailx: support openssl 1.1.x
cyrus-sasl: add UPSTREAM_CHECK_REGEX
cyrus-sasl: CLEANBROKEN = "1"
cyrus-sasl: Update to 2.1.27-rc7
cyrus-sasl: do not set CLEANBROKEN
cyrus-sasl: fix build out of source tree failed while configuring with 
`--enable-ldapdb'
cyrus-sasl: fix parallel build issue

I could submit these too if someone wants to setup a communit maintenance 
branch for sumo.

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Oe-core: python and BBCLASSEXTEND = "native nativesdk"

[Mikko.Rapeli]

Hi,

On Tue, Nov 19, 2019 at 03:21:22PM +, nick83ola wrote:
> Dear Openembedded developer,
> 
> a lot of python recipes need to add the
> 
> BBCLASSEXTEND = "native nativesdk"
> 
> To the recipe to build the native version of the package.
> Wouldn't be better to add it to the pythonnative.bbclass by default?

I agree to this. I have lots of bbappends which only add BBCLASSEXTEND = 
"native nativesdk".

This would not mean that python recipes would all compile or work on native or 
nativesdk
target, but quite a few of them would work out of the box.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to add build date? (was: ... basehash value changed...)

[Mikko.Rapeli]

On Sun, Nov 17, 2019 at 10:19:32PM +0100, Alexander Kanavin wrote:
> I'd write the date into the file at image creation, via
> ROOTFS_POSTPROCESS_COMMAND.
> 
> Having it in a recipe means that you either force the recipe to not be a
> part of sstate cache, always rebuilding it and its dependencies (bad idea),
> or accept that the date comes from a previously built cache object, which
> means it will mismatch the actual image creation date.

This is the approach I use. Also, the need of build time stamp is usually not 
what
users need. They likely need the latest time stamp in meta layer commits,
or similar. I use git submodules for them so a "git describe --tags --always 
--dirty" will
generate a unique tag and hash combination for all builds. The commit time
can be queried with 'git log HEAD~1.. --format="%cD"'.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [oe] OEDeM minutes clarification regarding stable branch update frequency

[Mikko.Rapeli]

On Wed, Nov 13, 2019 at 07:48:34AM -0800, akuster808 wrote:
> Hello,
> 
> Reading through the 2019 OEDeM minutes, I saw a statement regarding a
> more regular update frequency on the stable branches. Based on Richard's
> response I could not tell if it is regarding merge frequency  to
> mainline or more dot releases? Since I was not present at the meeting,
> can someone clarify which one that question was pertaining to or may both?

AFAIK, the comment meant more frequent dot releases.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] maintaining sumo (was Re: [PATCH][thud] cve-check: backport rewrite from master)

[Mikko.Rapeli]

Hi,

On Thu, Nov 07, 2019 at 10:41:42AM +, Ryan Harkin wrote:

> Not sure if it helps, but I have a Jenkins job that tests sumo on a trigger
> (there is one for Warrior also):
> 
> https://ci.linaro.org/job/warp7-openembedded-sumo/
> 
> eg. it was triggered when Armin's patch was merged yesterday.
> 
> This builds Sumo, based on Linaro's OE-RPB distro for NXP WaRP7
> (imx7s-warp). It then runs the build in our LAVA lab (although the boards
> have gone down recently, they're normally up). Once the boards are up
> again, I'll add ptest to the job, to give it a more thorough workout. I'll
> also add the sumo-next branch to the list of build configurations.

Could it make sense to also test sumo-next using this job? Also, I would not 
mind
seeing test trigger and result emails on this list as long as access to logs 
and details
is open to everyone.

I checked the jenkins build job config and looks quite straight forward. Did 
not see
the actual target testing logs though.

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH][zeus][master] harfbuzz: split libharfbuzz-subset.so to its own binary package

[Mikko.Rapeli]

On Thu, Nov 07, 2019 at 10:54:23AM +, Ross Burton wrote:
> On 07/11/2019 08:13, Mikko Rapeli wrote:
> > harfbuzz binary package size increased from 624608 bytes in yocto 2.5 to
> > 1365431 bytes in yocto 3.0. Most of the size increase is in the new
> > libharfbuzz-subset.so* library, which based on the name seems to
> > duplicate parts of the libharfbuzz main library, though it
> > RDEPENDS on harfbuzz which is odd. Split it to its own binary
> > package which will be installed if anyone needs it. Effect to harfbuzz
> > binary package size is:
> 
> 
> The subset piece is a library to subset fonts, not a subset of the library
> itself.
> 
> https://harfbuzz.github.io/utilities.html#utilities-command-line-hbsubset
> 
> Can you update the message?

Thanks, v2 coming.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH RFC CFH][sumo 00/47] CVE check backport

[Mikko.Rapeli]

Hi,

On Thu, Nov 07, 2019 at 01:13:32PM +0200, Adrian Bunk wrote:
> On Wed, Nov 06, 2019 at 05:37:15PM +0200, Mikko Rapeli wrote:
> > Hi,
> 
> Hi Mikko,
> 
> >...
> > I use sumo and due to various reasons like BSP layers, binary
> > compatibility, contracts etc can't update to newer release
> > or to master branch. I suspect I'm not alone.
> 
> I might end up with similar reasons, but for warrior.
> And might end up doing similar longer term updates for warrior.
> (not yet 100% certain)

I'm skipping warrior but going to zeus in addition to sumo. After
insipiration from Yocto Project Summit I hope to run master branch
in some projects with regular updates, and eventually aligning to
some stable release again. Hopefully an LTS one :)

> >...
> > The tooling will expose that sumo is severely lacking in security
> > patches, but the tooling is a start for anyone interested, like me,
> > to fill the gaps and publish patches for bitbake recipes we care
> > about.
> >...
> 
> Thud is officially still community maintained, as long as this is true
> the point could be made that everything that gets fixed in sumo should
> also get fixed in thud.

So to keep sumo alive, we should the also keep zeus, warrior and thud, and
of course master branch first. For some issues this actually works when
the exact same CVE patch applies, but the open question then is testing.

How should a developer test a patch before submitting it, or multiple versions
of it?

I'm testing in project tree with CI and target tests, then compile testing on
master. qemu ptest runs would be nice but not sure how to get a stable or useful
test set for various branches.

To make things more complicated, the project trees sadly contain more 
backports, fixes
and workarounds which are not suitable for upstreaming into stable or even 
master
branches.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH RFC CFH][sumo 00/47] CVE check backport

[Mikko.Rapeli]

Hi,

On Wed, Nov 06, 2019 at 01:46:09PM -0800, akuster808 wrote:
> Hello Mikko;
 
> I have collected other patches for sumo and built them locally but I
> have no way to inform Richard they pass an AB  builds or automated
> testing for them to get  into mainline sumo.
> 
> I am placing them into
> https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/sumo-community

Wow, I had no idea of this tree. Looks like the content is quite liberal
with version updates which may be necessary to maintain decent CVE fix status
with the resources at hand.

What is the relationship to thud?

Comparing oe-core sumo branch to contrib/stable/sumo-community shows for 
example:

--- a/meta-selftest/conf/layer.conf
+++ b/meta-selftest/conf/layer.conf
@@ -9,4 +9,4 @@ BBFILE_COLLECTIONS += "selftest"
 BBFILE_PATTERN_selftest = "^${LAYERDIR}/"
 BBFILE_PRIORITY_selftest = "5"
 
-LAYERSERIES_COMPAT_selftest = "sumo"
+LAYERSERIES_COMPAT_selftest = "thud"

--- a/meta-skeleton/conf/layer.conf
+++ b/meta-skeleton/conf/layer.conf
@@ -14,4 +14,4 @@ LAYERVERSION_skeleton = "1"
 
 LAYERDEPENDS_skeleton = "core"
 
-LAYERSERIES_COMPAT_skeleton = "sumo"
+LAYERSERIES_COMPAT_skeleton = "thud"

--- a/meta/conf/layer.conf
+++ b/meta/conf/layer.conf
@@ -7,12 +7,12 @@ BBFILE_COLLECTIONS += "core"
 BBFILE_PATTERN_core = "^${LAYERDIR}/"
 BBFILE_PRIORITY_core = "5"
 
-LAYERSERIES_CORENAMES = "sumo"
+LAYERSERIES_CORENAMES = "thud"
 
 # This should only be incremented on significant changes that will
 # cause compatibility issues with other layers
 LAYERVERSION_core = "11"
-LAYERSERIES_COMPAT_core = "sumo"
+LAYERSERIES_COMPAT_core = "thud"
 
...

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] maintaining sumo (was Re: [PATCH][thud] cve-check: backport rewrite from master)

[Mikko.Rapeli]

Hi,

On Wed, Nov 06, 2019 at 05:53:27PM +, Richard Purdie wrote:
> On Wed, 2019-11-06 at 16:06 +, mikko.rap...@bmw.de wrote:
> > Hi,
> > 
> > On Wed, Nov 06, 2019 at 02:59:16PM +, Ryan Harkin wrote:
> > > Hi Ross/Richard,
> > > 
> > > I'd like this applied to Sumo also. Should I create a new patch and
> > > send it
> > > to the list, or is there a process for requesting this is cherry-
> > > picked
> > > across?
> > 
> > I just posted the port of this and all other CVE scan related changes
> > to sumo
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-November/288817.html
> > 
> > But the question is valid :)
> 
> Support for sumo officially ended. I can see a case that the broken CVE
> tools there are a good reason we could consider merging the patch
> series but we do need to be able to test it to merge it to the main
> branch. If we can't test, we're merging blind and the quality the
> project tries to deliver could be compromised.
> 
> I have made some tweaks to the autobuilder which bring us closer to
> being able to test sumo using the workers still around from that
> release.
> 
> The things that make me nervous are questions like:
> 
> Which releases do we "open" for such patches? How far back do we go?
> Which kinds of patches are acceptable?
> 
> Note that sumo (and earlier) doesn't have much of the QA automation
> which we've now built our processes around so we don't get test
> reports.
> 
> You mention wanting to change gcc. That means we really do need a full
> retest of it to merge that (which is why it never happened originally
> from what I remember).
> 
> Also, the LTS proposal stated we needed someone to handle this work. We
> have no such person, even if we do somehow find them, they can't be
> expected to cover all the old releases and effectively turn all of them
> into LTS releases. How can we get the funding to try and get some help
> with handling this workload?
> 
> I am probably going to try and make a case for sorting the CVE tooling
> on sumo as I agree its bad and we should do something. Where do we draw
> the line though.
> 
> Basically, this looks like it could create a lot of extra work without
> helping the core project under-resourcing we currently struggle with.
> You can therefore see why I might be nervous :/.

All this is understood.

I need to maintain sumo in a project for a while longer so I can publish that 
work.
The CVE checker patches are just a start.

Providing funding for Yocto Project LTS work is possible but a lot harder for 
me to do.
Testing and publishing patches is much easier.

Could you clarify Yocto Project side answers to these questions:

If I continue to publish patches for sumo, can I continue doing so on oe-core 
mailing list?

If I continue to collect patches for sumo, can I do so using Yocto Project 
infrastructure, e.g.
a sumo-contrib-lts or similar branch in poky git tree?

If I continue to test patches, what would be the patch acceptance criteria and 
required testing?
I would assume same as stable release rules, but maybe these need to be even 
stricter, e.g.
only support building on Debian stable, following the LTS proposal. I'm testing 
in my own project
trees and CI with target HW, and doing world builds on pure poky with qemu 
target. I could some
kind of ptest execution to plain poky as well.

Would any testing of patches be possible in Yocto Project infrastructure?

All of these things I can do also completely outside of Yocto Project, e.g. 
publish a sumo
git tree on github, and rely only on my own testing. But I'd like to see
some co-operation here from other users who are stuck with sumo.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH][thud] cve-check: backport rewrite from master

[Mikko.Rapeli]

Hi,

On Wed, Nov 06, 2019 at 02:59:16PM +, Ryan Harkin wrote:
> Hi Ross/Richard,
> 
> I'd like this applied to Sumo also. Should I create a new patch and send it
> to the list, or is there a process for requesting this is cherry-picked
> across?

I just posted the port of this and all other CVE scan related changes to sumo
http://lists.openembedded.org/pipermail/openembedded-core/2019-November/288817.html

But the question is valid :)

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemd.bbclass: enable all services specified in ${SYSTEMD_SERVICE}

[Mikko.Rapeli]

On Wed, Oct 16, 2019 at 12:45:20PM +, Peter Kjellerstedt wrote:
> > -Original Message-
> > From: openembedded-core-boun...@lists.openembedded.org  > core-boun...@lists.openembedded.org> On Behalf Of Mikko Rapeli
> > Sent: den 16 oktober 2019 14:32
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH] systemd.bbclass: enable all services
> > specified in ${SYSTEMD_SERVICE}
> > 
> > This has been the traditional way of enabling systemd services.
> > It may conflict with presets feature, but other layers, image classes
> > and recipes add services to be enabled using SYSTEMD_SERVICE
> > variable also with read-only rootfs, e.g. IMAGE_FEATURES has
> > stateless-rootfs and systemd_preset_all task is not executed.
> > 
> > Fixes startup of custom services from our recipes using custom
> > image classes with various BSP layers. In the worst case even
> > serial console getty service wasn't starting due to dependency
> > no not enabled services.
> > 
> > Signed-off-by: Mikko Rapeli 
> > ---
> >  meta/classes/systemd.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/classes/systemd.bbclass
> > b/meta/classes/systemd.bbclass
> > index 1dca099..ae03c6f 100644
> > --- a/meta/classes/systemd.bbclass
> > +++ b/meta/classes/systemd.bbclass
> > @@ -33,7 +33,7 @@ if type systemctl >/dev/null 2>/dev/null; then
> > if [ "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then
> > for service in ${SYSTEMD_SERVICE_ESCAPED}; do
> > case "${service}" in
> > -   *@*)
> > +   *)
> > systemctl ${OPTS} enable "${service}"
> > ;;
> > esac
> 
> Not much point in leaving the case statement if it only has a 
> capture all case. I.e., the above simplifies to:
> 
>   if [ "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then
>   for service in ${SYSTEMD_SERVICE_ESCAPED}; do
>   systemctl ${OPTS} enable "$service"
> 
> (also note the change of "${service}" to "$service" to avoid using 
> ${...} for shell variables where not necessary as this causes them 
> to unnecessarily end up in the bitbake hash for the function.)

Thanks, I'll send a v2.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [RFC PATCH 2/2] license_image.bbclass: check and reject packages which have incompatible licenses

[Mikko.Rapeli]

On Wed, Oct 09, 2019 at 09:41:28PM +0200, Alexander Kanavin wrote:
> It wouldn't be too hard to add a condition that checks the (image-specific)
> whitelist, I just wanted to gather a bit of feedback for the overall idea :)

I like the idea. I'm building images with e.g. GPLv3 forbidden but I enable 
building
lots of GPLv3 components because they are needed in e.g. ptests. Resulting 
distro
config snippet is large with lots of lines like:

WHITELIST_GPL-3.0 += "autoconf"
PACKAGE_EXCLUDE += "autoconf-dbg autoconf-staticdev autoconf-dev autoconf-doc 
autoconf-locale autoconf"
WHITELIST_GPL-3.0 += "ccache"
PACKAGE_EXCLUDE += "ccache-sdktests-dbg ccache-sdktests ccache-dbg 
ccache-staticdev ccache-dev ccache-doc ccache-locale ccache"

In testing then I start with the pure product image without GPLv3 components 
and extend it
with extra packages which are needed for the test execution. I have separate 
product
feature and SW component test phases. Only in the latter case target SW is 
changed before
test execution.

I want to avoid building separate test images with GPLv3, because in the past
test vs. product images resulted in only the test images being actually tested.

But I can see that having separate product and test images from a single build 
could
be useful.

Thanks for proposing this patch!

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] openssl: Enable os option for with-rand-seed as well

[Mikko.Rapeli]

On Fri, Sep 20, 2019 at 03:13:44PM +0200, Andrey Zhizhikin wrote:
> Hello Raj,
> 
> On Tue, Sep 17, 2019 at 8:50 PM Khem Raj  wrote:
> >
> > with openSSL 1.1.1d we start seeing errors like
> >
> > Error Generating Key
> > 139979727451584:error:2406C06E:random number 
> > generator:RAND_DRBG_instantiate:error retrieving 
> > entropy:../openssl-1.1.1d/crypto/rand/drbg_lib.c:342:
> >
> > when using openssl from openssl-native on build hosts, this is due to
> > limiting the random seed to devrandom, to support older hosts, since the
> > option allows to have a comma separated list of methods to try, we can
> > try the default first and if that fails then fallback to devrandom, this
> > will ensure that it keeps working with build systems which dont support
> > getrandom()
> >
> > Signed-off-by: Khem Raj 
> > Cc: Adrian Bunk 
> > Cc: Alexander Kanavin 
> > ---
> 
> Just as a test report for this patch:
> 
> I've tested this patch on the HW (i.MX8M Mini EVK) and unfortunately
> my sshd given up with a message: PRNG is not seeded
> 
> Reverting commits (effectively rolling back to openssl 1.1.1c) made
> sshd operable again.:
> 53b5654d6e openssl: Enable os option for with-rand-seed as well
> 2c6b9b918c openssl: Upgrade 1.1.1c -> 1.1.1d

Do you have rng-tools on the image? That helped me with the kernel random pool
initialization for sshd in iMX8 and openssl 1.1.1x.

I don't see how 53b5654d6e could change this behavior for target openssl.
2c6b9b918c could change the behavior and would be suprise. Maybe also
target recipe needs --with-rand-seed=os,devrandom on iMX8 or similar platforms.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [thud][PATCH v4] gcc: CVE fix for gcc

[Mikko.Rapeli]

Looks ok, though I would consider updating to point releases or latest
versions of the gcc-8 stable tree, where these patches are already backported
by upstream developers. The 8.3 update does not have this CVE patch, but
a lot of bug fixes which should be useful for gcc 8 users.

Reviewed-by: Mikko Rapeli 

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [thud][PATCH v2] gcc: CVE fix for gcc

[Mikko.Rapeli]

On Mon, Sep 16, 2019 at 08:37:28PM +, Muminul Islam wrote:
> Signed-off-by: Muminul Islam 
> ---
>  meta/recipes-devtools/gcc/gcc-8.2.inc |   2 +
>  .../gcc/gcc/0042-CVE-2019-15847_1.patch   | 570 
>  .../gcc/gcc/0043-CVE-2019-15847_2.patch   | 640 ++
>  3 files changed, 1212 insertions(+)
>  create mode 100644 meta/recipes-devtools/gcc/gcc/0042-CVE-2019-15847_1.patch
>  create mode 100644 meta/recipes-devtools/gcc/gcc/0043-CVE-2019-15847_2.patch
> 
> diff --git a/meta/recipes-devtools/gcc/gcc-8.2.inc 
> b/meta/recipes-devtools/gcc/gcc-8.2.inc
> index 866a77558b..cab494989e 100644
> --- a/meta/recipes-devtools/gcc/gcc-8.2.inc
> +++ b/meta/recipes-devtools/gcc/gcc-8.2.inc
> @@ -70,6 +70,8 @@ SRC_URI = "\
> file://0039-Fix-for-testsuite-failure.patch \
> file://0040-Re-introduce-spe-commandline-options.patch \
> file://0041-ARC-fix-spec-gen.patch \
> +   file://0042-CVE-2019-15847_1.patch \
> +   file://0043-CVE-2019-15847_2.patch \
> ${BACKPORTS} \
>  "
>  BACKPORTS = "\
> diff --git a/meta/recipes-devtools/gcc/gcc/0042-CVE-2019-15847_1.patch 
> b/meta/recipes-devtools/gcc/gcc/0042-CVE-2019-15847_1.patch
> new file mode 100644
> index 00..edebf2fb41
> --- /dev/null
> +++ b/meta/recipes-devtools/gcc/gcc/0042-CVE-2019-15847_1.patch
> @@ -0,0 +1,570 @@
> +From 3efdb8c4afcbc5e07d33b05ab8c2bf88f42f4890 Mon Sep 17 00:00:00 2001
> +From: segher 
> +Date: Thu, 22 Aug 2019 19:36:21 +
> +Subject: [PATCH] rs6000: Use unspec_volatile for darn (PR91481)
> +Reply-To: muis...@microsoft.com
> +
> +Every call to darn should deliver a *new* random number; such calls
> +should not be CSEd together.  So they should be unspec_volatile, not
> +plain unspec.
> +
> + PR target/91481
> + * config/rs6000/rs6000.md (unspec): Delete UNSPEC_DARN, UNSPEC_DARN_32,
> + and UNSPEC_DARN_RAW.
> + (unspecv): New enumerator values UNSPECV_DARN, UNSPECV_DARN_32, and
> + UNSPECV_DARN_RAW.
> + (darn_32): Use an unspec_volatile, and UNSPECV_DARN_32.
> + (darn_raw): Use an unspec_volatile, and UNSPECV_DARN_RAW.
> + (darn): Use an unspec_volatile, and UNSPECV_DARN.
> +
> +git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@274835 
> 138bc75d-0d04-0410-961f-82ee72b054a4
> +Signed-off-by: Muminul Islam 
> +
> +CVE: CVE-2019-15847
> +Upstream-Status: Backport
> +---
> + gcc/ChangeLog   | 336 +++-
> + gcc/config/rs6000/rs6000.md | 169 +-
> + 2 files changed, 503 insertions(+), 2 deletions(-)
> +
> +diff --git a/gcc/ChangeLog b/gcc/ChangeLog
> +index b93dae5dfb0..dc22d7e43b7 100644
> +--- a/gcc/ChangeLog
>  b/gcc/ChangeLog

This changelog is not correct for only fixing PR 91481.

Because every upstream commit basically adds to the changelog and makes
all cherry-picks and backports fail, I would be fine in omitting
any changes to it in patches like this.

-Mikko

> +@@ -1,4 +1,338 @@
> +-2018-07-26  Release Manager
> ++2019-08-22  Segher Boessenkool  
> ++
> ++PR target/91481
> ++* config/rs6000/rs6000.md (unspec): Delete UNSPEC_DARN, UNSPEC_DARN_32,
> ++and UNSPEC_DARN_RAW.
> ++(unspecv): New enumerator values UNSPECV_DARN, UNSPECV_DARN_32, and
> ++UNSPECV_DARN_RAW.
> ++(darn_32): Use an unspec_volatile, and UNSPECV_DARN_32.
> ++(darn_raw): Use an unspec_volatile, and UNSPECV_DARN_RAW.
> ++(darn): Use an unspec_volatile, and UNSPECV_DARN.
> ++
> ++2019-08-22  Segher Boessenkool  
> ++
> ++* config/rs6000/altivec.md (unspec): Delete UNSPEC_DARN, UNSPEC_DARN_32,
> ++UNSPEC_DARN_RAW, UNSPEC_CMPRB, UNSPEC_CMPRB2, UNSPEC_CMPEQB; move to...
> ++* config/rs6000/rs6000.md (unspec): ... here.
> ++* config/rs6000/altivec.md (darn_32, darn_raw, darn, cmprb,
> ++*cmprb_internal, setb_signed, setb_unsigned, cmprb2, *cmprb2_internal,
> ++cmpeqb, *cmpeqb_internal): Delete, move to...
> ++* config/rs6000/rs6000.md (darn_32, darn_raw, darn, cmprb,
> ++*cmprb_internal, setb_signed, setb_unsigned, cmprb2, *cmprb2_internal,
> ++cmpeqb, *cmpeqb_internal): ... here.
> ++
> ++2019-08-22  Kyrylo Tkachov 
> ++
> ++* config/arm/arm_acle.h: Use arch=armv8-a+crc+simd pragma for CRC32
> ++intrinsics if __ARM_FP.
> ++Use __ARM_FEATURE_CRC32 ifdef guard.
> ++
> ++2019-08-22  Wilco Dijkstra  
> ++
> ++* config/arm/arm.md (neon_for_64bits): Remove.
> ++(avoid_neon_for_64bits): Remove.
> ++(arm_adddi3): Always split early.
> ++(arm_subdi3): Always split early.
> ++(negdi2): Remove Neon expansion.
> ++(split zero_extend): Split before reload.
> ++(split sign_extend): Split before reload.
> ++
> ++2019-08-22  Wilco Dijkstra  
> ++
> ++* config/arm/iterators.md (qhs_extenddi_cstr): Update.
> ++(qhs_extenddi_cstr): Likewise.
> ++* config/arm/arm.md (ashldi3): Always expand early.
> ++(ashlsi3): Likewise.
> ++(ashrsi3): Likewise.
> ++

Re: [OE-core] [PATCH 1/2] busybox.inc: handle empty DEBUG_PREFIX_MAP

[Mikko.Rapeli]

Hi,

On Mon, Sep 16, 2019 at 01:37:25PM +0100, Ross Burton wrote:
> Hi Mikko,
> 
> This is 1/2 but 2/2 never arrived on the list.  Can you repost it?

Sorry, 2/2 from my poky tree was the bitbake svn fetcher patch in
http://lists.openembedded.org/pipermail/bitbake-devel/2019-September/020328.html

Can you ignore the numberings or shall I send again without them?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemd: ensure reproducible builds by clearly exposing the time epoch support

[Mikko.Rapeli]

On Fri, Sep 06, 2019 at 10:03:14AM +0100, Ross Burton wrote:
> On 06/09/2019 09:02, mikko.rap...@bmw.de wrote:
> > Could this be enabled automatically when local.conf has INHERIT += 
> > "reproducible_build" ?
> 
> The default behaviour is to respect that if enabled, otherwise use the
> upstream default of the build time.  If the recipe followed the
> reproducble_build inherit then if a user wants reproducible builds but
> doesn't want the epoch then they won't be able to disable it (after all, an
> epoch of 0 is reproducible).

Ok, fair enough. Thanks for this patch!

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemd: ensure reproducible builds by clearly exposing the time epoch support

[Mikko.Rapeli]

On Fri, Sep 06, 2019 at 12:07:06AM +0100, Ross Burton wrote:
> systemd has the ability to check the time on boot and if it's earlier than an
> epoch determined at build time, set the time to that epoch.  This is useful 
> for
> systems where the system time is January 1st 1970 (because the unix timestamp
> was 0 at boot) as then at least the time is reset to something approximating 
> the
> right year at least.
> 
> By default systemd uses the mtime of the NEWS file, which is static for 
> tarballs
> and corresponds to the time the release was made, but for git checkouts this 
> is
> simply the time do_unpack() was executed.  Thus, rebuilding systemd will cause
> this embedded timestamp to change.
> 
> Remove the PACKAGECONFIG time-epoch which has the logic reversed: enabling
> time-epoch will set the epoch to the unix timestamp 0).  Replace with
> set-time-epoch with the following semantics:
> 
> - When disabled, the time epoch is set to 0 (1st January 1970), so there is no
>   time manipulation on boot.
> 
> - When enabled, if reproducible builds are configured by setting
>   SOURCE_DATE_EPOCH then that timestamp is used for the time epoch.  If
>   reproducible builds are not configured then the timestamp of NEWS (thus the
>   build time) is used.
> 
> The set-time-epoch flag is enabled by default.
> 
> [ YOCTO #13473 ]
> 
> Signed-off-by: Ross Burton 
> ---
>  meta/recipes-core/systemd/systemd_242.bb | 8 +++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-core/systemd/systemd_242.bb 
> b/meta/recipes-core/systemd/systemd_242.bb
> index 6bbe388b1f9..2c101cbbb4a 100644
> --- a/meta/recipes-core/systemd/systemd_242.bb
> +++ b/meta/recipes-core/systemd/systemd_242.bb
> @@ -83,6 +83,7 @@ PACKAGECONFIG ??= " \
>  quotacheck \
>  randomseed \
>  resolved \
> +set-time-epoch \

Could this be enabled automatically when local.conf has INHERIT += 
"reproducible_build" ?

-Mikko

>  smack \
>  sysusers \
>  timedated \
> @@ -166,7 +167,12 @@ PACKAGECONFIG[seccomp] = 
> "-Dseccomp=true,-Dseccomp=false,libseccomp"
>  PACKAGECONFIG[selinux] = 
> "-Dselinux=true,-Dselinux=false,libselinux,initscripts-sushell"
>  PACKAGECONFIG[smack] = "-Dsmack=true,-Dsmack=false"
>  PACKAGECONFIG[sysusers] = "-Dsysusers=true,-Dsysusers=false"
> -PACKAGECONFIG[time-epoch] = "-Dtime-epoch=0,,"
> +# When enabled use reproducble build timestamp if set as time epoch,
> +# or build time if not. When disabled, time epoch is unset.
> +def build_epoch(d):
> +epoch = d.getVar('SOURCE_DATE_EPOCH') or "-1"
> +return '-Dtime-epoch=%d' % int(epoch)
> +PACKAGECONFIG[set-time-epoch] = "${@build_epoch(d)},-Dtime-epoch=0"
>  PACKAGECONFIG[timedated] = "-Dtimedated=true,-Dtimedated=false"
>  PACKAGECONFIG[timesyncd] = "-Dtimesyncd=true,-Dtimesyncd=false"
>  PACKAGECONFIG[usrmerge] = "-Dsplit-usr=false,-Dsplit-usr=true"
> -- 
> 2.20.1
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] gcc-9.2: Security fix for CVE-2019-14250

[Mikko.Rapeli]

On Mon, Sep 02, 2019 at 02:33:02PM -0700, akuster808 wrote:
> 
> 
> On 9/2/19 5:40 AM, Adrian Bunk wrote:
> > On Sun, Sep 01, 2019 at 10:07:13AM -0700, akuster808 wrote:
> >>
> >> On 9/1/19 7:05 AM, Adrian Bunk wrote:
> >>> thud and zeus are providing 2 gcc versions each that need fixing.
> >> That is a true statement. What are you expecting?
> > The other versions also being fixed?
> >
> > gcc-8 being fixed in warrior before it gets fixed in master would be
> > the wrong order, and would introduce a security regression in master.
> sent a patch. hope it is what is meant by the above.
> 
> >
> > The code should be nearly identical in warrior and master, so fixing
> > this also in gcc-8 in master should be trivial.
> >
> > Fixing gcc-7 in thud would be a bonus.

FWIW, gcc-7-branch of https://github.com/gcc-mirror/gcc.git has this fix 
already.

-Mikko

commit 740d8b3baeea47cd5407be1752c5159223f77042
Author: rguenth 
AuthorDate: Thu Jul 25 10:50:47 2019 +
Commit: rguenth 
CommitDate: Thu Jul 25 10:50:47 2019 +

2019-07-25  Richard Biener  

PR lto/90924
Backport from mainline
2019-07-12  Ren Kimura  

* simple-object-elf.c (simple_object_elf_match): Check zero value
shstrndx.


git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/branches/gcc-7-branch@273795 
138bc75d-0d04-0410-961f-82ee72b054a4

diff --git a/libiberty/ChangeLog b/libiberty/ChangeLog
index b785e71..0ecdec0 100644
--- a/libiberty/ChangeLog
+++ b/libiberty/ChangeLog
@@ -1,3 +1,12 @@
+2019-07-25  Richard Biener  
+
+   PR lto/90924
+   Backport from mainline
+   2019-07-12  Ren Kimura  
+
+   * simple-object-elf.c (simple_object_elf_match): Check zero value
+   shstrndx.
+
 2018-12-06  Release Manager
 
* GCC 7.4.0 released.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] stress-ng: provide stress

[Mikko.Rapeli]

On Wed, Aug 14, 2019 at 03:38:56PM +0100, Richard Purdie wrote:
> On Wed, 2019-08-14 at 15:56 +0300, Mikko Rapeli wrote:
> > Since stress-ng replaces and is compatible with stress,
> > provide stress to be compatible with the old recipe
> > and binary packages.
> > 
> > Signed-off-by: Mikko Rapeli 
> > ---
> >  meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb | 4 
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
> > b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
> > index e800040..64e0928 100644
> > --- a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
> > +++ b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
> > @@ -14,6 +14,10 @@ SRC_URI[sha256sum] =
> > "d09dd2a1aea549e478995bf9be90b38906a4cdf33ea7b245ef9d46aa52
> >  
> >  DEPENDS = "coreutils-native"
> >  
> > +PROVIDES = "stress"
> > +RPROVIDES_${PN} = "stress"
> > +RPROVIDES_${PN}-dev = "stress-dev"
> > +
> 
> Isn't that going to need RCONFLICTS and RREPLACES too?

I did not need them. I'm not updating existing images or mixing
stress and stress-ng binary packages in a package repo.

But I will add them in a v2.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Long delays with latest bitbake (was: [PATCH 1/7] insane.bbclass: in file-rdeps do not look into RDEPENDS recursively)

[Mikko.Rapeli]

On Wed, Aug 14, 2019 at 02:08:01PM +0200, Alexander Kanavin wrote:
> On Wed, 14 Aug 2019 at 13:36,  wrote:
> 
> > On Wed, 2019-08-14 at 13:25 +0200, Alexander Kanavin wrote:
> > > On Tue, 13 Aug 2019 at 21:18, Richard Purdie <
> > > richard.pur...@linuxfoundation.org> wrote:
> > > > I had a glance at the profile output from master-next and the
> > > > problem
> > > > wasn't where I thought it would be, it was in the scheduler code.
> > > > That
> > > > is good as those classes are effectively independent of the other
> > > > changes and hence are a separate fix.
> > > >
> > > > I've put a patch in -next which takes the above test to 36s which
> > > > is
> > > > close to the older bitbake.
> > > >
> > > > Could be interesting to see how it looks for others and different
> > > > workloads.
> > >
> > > I just tried the same test I did yesterday with
> > > ab56d466452148e5fce330d279d13e2495eceb1f. Unfortunately it doesn't
> > > seem to improve things much: bitbake is stuck at "NOTE: Executing
> > > Tasks" for 15 minutes now.
> >
> > This might sound slightly crazy but can you try commenting out this
> > line in runqueue.py:
> >
> > logger.debug(2, "Holding off tasks %s" %
> > pprint.pformat(self.holdoff_tasks))
> >
> > ?
> >
> 
> Even crazier is the outcome: it helped! The whole thing completed after
> 15m49secons (with much of the time going to the empty task spin), that's
> some 3 minutes slower, but certainly it's usable again.
> 
> I have not enabled the hash server at any point.

Indeed, commenting that debug statement out removes few minutes of the
bitbake preparation times.

Here is a time stamped output from master branch:

2019-08-14_14:29:24  
2019-08-14_14:29:35  Initialising tasks...done.
2019-08-14_14:32:41  Checking sstate mirror object availability...done.
2019-08-14_14:32:41  Sstate summary: Wanted 436 Found 426 Missed 10 Current 
2407 (97% match, 99% complete)
2019-08-14_14:32:41  NOTE: Executing Tasks
2019-08-14_14:32:52  NOTE: Setscene tasks completed

And with the comment removed:

2019-08-14_14:35:10  
2019-08-14_14:35:21  Initialising tasks...done.
2019-08-14_14:35:30  Checking sstate mirror object availability...done.
2019-08-14_14:35:30  Sstate summary: Wanted 436 Found 426 Missed 10 Current 
2407 (97% match, 99% complete)
2019-08-14_14:35:30  NOTE: Executing Tasks

And just in case back to master branch state:

2019-08-14_14:41:32  
2019-08-14_14:41:43  Initialising tasks...done.
2019-08-14_14:45:02  Checking sstate mirror object availability...done.
2019-08-14_14:45:02  Sstate summary: Wanted 436 Found 426 Missed 10 Current 
2407 (97% match, 99% complete)
2019-08-14_14:45:02  NOTE: Executing Tasks

I did not flush caches in between and I stopped the builds once bitbake started 
doing stuff, so I think sstate cache was completely buffered in memory
from file system and IO delays did not affect the timings.

Change to poky was exactly:

--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -2216,7 +2216,7 @@ class RunQueueExecute:
 for dep in self.sqdata.sq_covered_tasks[tid]:
 if dep not in self.runq_complete:
 self.holdoff_tasks.add(dep)
-logger.debug(2, "Holding off tasks %s" % 
pprint.pformat(self.holdoff_tasks))
+# logger.debug(2, "Holding off tasks %s" % 
pprint.pformat(self.holdoff_tasks))
 
 def process_possible_migrations(self):
 changes = False

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Hash Equivalency - What this means for developer productivity

[Mikko.Rapeli]

Hi,

On Fri, Aug 02, 2019 at 05:17:21PM +0100, Richard Purdie wrote:
> On Fri, 2019-08-02 at 16:53 +0100, Richard Purdie wrote:
> > With the patches in master-next and this configuration in local.conf:
> > 
> > BB_HASHSERVE = "localhost:0"
> > BB_SIGNATURE_HANDLER = "OEEquivHash"
> > 
> > $ bitbake core-image-sato
> > $ bitbake m4-native -c install -f
> > $ bitbake core-image-sato
> > 
> > will result in do_populate_sysroot of m4-native running, it will see
> > the output matches the previous build and it will then skip to the
> > rootfs generation pulling all the other pieces from sstate.
> > 
> > Note that for this to work, m4-native has to have previously built
> > with the hashserv running, otherwise it has nothing to compare its
> > output to.
> > 
> > I think this should be a "big deal" for many developers, reducing
> > unneeded rebuilds and hence speeding up development.

Awesome, thanks for pushing this!

> I should have mentioned, this code relies on reproducibile builds as
> its comparing the binary output. The more reproducibile builds are, the
> more likely sstate reuse will happen.
> 
> This is one reason reproducibile builds are important!

What else do users need to enable to get more reproducible builds, or
are poky defaults enough?

Are there some tools available to debug build reproducibility issues e.g.
when task hashes suddenly changed?

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] kill-bb: Add it for killing abnormal bitbake processes

[Mikko.Rapeli]

On Fri, Aug 02, 2019 at 11:21:24AM +0100, Ross Burton wrote:
> On 02/08/2019 11:24, Robert Yang wrote:
> > There might be processes left after Ctr-C, e.g.:
> > $ rm -f tmp/cache/default-glibc/qemux86/x86_64/
> > $ bitbake -p
> > 
> > Press 'Ctrl-C' multiple times during parsing, then bitbake processes may not
> > exit, and the worse is that we can't start bitbake again, we can't always
> > reproduce this, but sometime. We can only use "ps ux" to find the processes 
> > and
> > kill them one by one. This tool can kill all of them easily.
> I've noticed this, and also noticed that it got a lot worse recently.
> 
> But let's fix bitbake instead of adding tools to work around it?

I run builds in lxc containers for this and host contamination reasons.

Several build tools can also escape the bitbake environment and keep
running in the build machine if bitbake itself ends or gets killed,
so the problems are not only with bitbake.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[Mikko.Rapeli]

Hi,

On Thu, Aug 01, 2019 at 04:51:38PM +, chris.laplante--- via 
Openembedded-core wrote:

> I'm interesting in adding SRC_URI support to buildhistory (or a similar 
> mechanism), and would like to get some input.

Yes to this.

Also would be nice if there was an easy way to add bitbake variables to
buildhistory.

I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are archived
in buildhistory. SRC_URI has many uses and changes and patches can
be easily identified. Same with LICENSE, any changes trigger a review.
CVE_PRODUCT is exported so that we can do QA check to make sure mapping
from CVE_PRODUCT for non CLOSED licenses exists to NVD database product
names (maintaining a white list of recipes which don't have any CVEs yet).

We've also changed the SDK name to be stable across builds and added
DISTRO to the path. In our case IMAGE_NAME and SDK_NAME will include
git tree tag and hash if tree is dirty, which changes buildhistory SDK paths
for every build with different input.

I could submit the patches if there is interest in them.

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] busybox: enable unicode support

[Mikko.Rapeli]

On Wed, Jul 17, 2019 at 12:44:54PM +0100, Richard Purdie wrote:
> On Wed, 2019-07-17 at 12:08 +0300, Mikko Rapeli wrote:
> > While creating and deleting files with unicode or other
> > encodings works, it's annoying when ls and other core utils
> > show questionmarks instead of the unicode characters.
> > In 2019, it's quite common that users of embedded devices
> > based on yocto need unicode support. Debugging a box with
> > unicode encoded file names is a bit annoying when core utils
> > from busybox don't support them.
> > 
> > The unicode config fragment has the same config as Debian in their
> > deb and udeb builds of version 1:1.30.1-4.
> > 
> > If developers do not want this or other default yocto features in
> > busybox,
> > or optimize the configuration for size, then they likely run a
> > completely
> > custom configuration. Thus I think it's safe to enable unicode
> > support
> > by default.
> > 
> > Signed-off-by: Mikko Rapeli 
> > ---
> >  meta/recipes-core/busybox/busybox/unicode.cfg | 10 ++
> >  meta/recipes-core/busybox/busybox_1.31.0.bb   |  1 +
> >  2 files changed, 11 insertions(+)
> >  create mode 100644 meta/recipes-core/busybox/busybox/unicode.cfg
> 
> Can you give an idea of the size implications of this change?

buildhistory isn't working for me, or maybe it is since the size difference
is zero bytes for busybox binaries and package. tmp has this patch,
tmp_master doesn't:

$ ls -l tmp*/deploy/images/qemux86/tmp/bin/busybox.*
-rwxr-xr-x 1 mcfrisk mcfrisk 612024 Jul 17 10:45 
tmp/deploy/images/qemux86/tmp/bin/busybox.nosuid
-rwxr-xr-x 1 mcfrisk mcfrisk  54656 Jul 17 10:45 
tmp/deploy/images/qemux86/tmp/bin/busybox.suid
-rwxr-xr-x 1 mcfrisk mcfrisk 612024 Jul 17 13:56 
tmp_master/deploy/images/qemux86/tmp/bin/busybox.nosuid
-rwxr-xr-x 1 mcfrisk mcfrisk  54656 Jul 17 13:56 
tmp_master/deploy/images/qemux86/tmp/bin/busybox.suid

Just to be sure, the files differ:

$ sha1sum tmp*/deploy/images/qemux86/tmp/bin/busybox.*
8866ced8ad8aa335bb5dfc8b229851b6e09397d7  
tmp/deploy/images/qemux86/tmp/bin/busybox.nosuid
01cb28f1254f4200b005122ce03114b034b3aaec  
tmp/deploy/images/qemux86/tmp/bin/busybox.suid
91ed5b26083c9b6f944344036b04ebf2aa5af6ff  
tmp_master/deploy/images/qemux86/tmp/bin/busybox.nosuid
d4a3ee35246c0f42ba4f0a8a34cd96269b89e906  
tmp_master/deploy/images/qemux86/tmp/bin/busybox.suid

buildhistory shows that busybox-dbg PKGSIZE increases from 4523596 to 4546396, 
busybox-ptest PKGSIZE decreases from 458949 to 458908 even when
/usr/lib/busybox/ptest/testsuite/tar.utf8.tar.bz2 is added, busybox-src
PKGSIZE increases from 4130952 to 4163274.

I found that hard to believe so I didn't trust buildhistory anymore.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] build failures due to pigz host tool

[Mikko.Rapeli]

On Wed, Jul 03, 2019 at 11:04:06AM -0400, Trevor Woerner wrote:
> Hi,
> 
> This came up as a topic in yesterday's Engineering Sync meeting. For roughly a
> year I've been seeing random build failures on my Jenkins setup due to pigz
> failing; apparently the project is now seeing them on their builds, so I'll
> share what I know of them.
> 
> At the time I started seeing these failures (Aug 2018) I had just upgraded my
> system to openSUSE 15.0. Since nobody else was seeing them, I assumed they
> were related to my setup. When I went out searching for an answer, I found
> there wasn't very much out there to help me. But I did notice that there were
> reports of other people seeing the issue who weren't using openSUSE and who
> weren't doing anything related to OE builds using Jenkins.
> 
> The build failure looks something like this:
> 
>   | DEBUG: Executing shell function sstate_create_package
>   | pigz: abort: internal threads error
>   | tar: 
> /z/jenkins-workspace/nightly/cubietruck/build/sstate-cache/8a/sstate:linux-mainline:cubietruck-oe-linux-gnueabi:4.19.46:r0:cubietruck:3:8a159ba1ffefb5fc2feeeff5b40abf8ad67658e5ff3ed3bf67d25d9c8f2805e0_package.tgz.9bA8tCje:
>  Wrote only 6144 of 10240 bytes
>   | tar: Child returned status 16
>   | tar: Error is not recoverable: exiting now
>   | WARNING: 
> /z/jenkins-workspace/nightly/cubietruck/build/tmp-glibc/work/cubietruck-oe-linux-gnueabi/linux-mainline/4.19.46-r0/temp/run.sstate_create_package.19996:1
>  exit 1 from 'exit 1'
>   | DEBUG: Python function sstate_task_postfunc finished
>   | ERROR: Function failed: sstate_create_package (log file is located at 
> /z/jenkins-workspace/nightly/cubietruck/build/tmp-glibc/work/cubietruck-oe-linux-gnueabi/linux-mainline/4.19.46-r0/temp/log.do_package.19996)
>   NOTE: recipe linux-mainline-4.19.46-r0: task do_package: Failed
>   ERROR: Task 
> (/opt/oe/configs/z/jenkins-workspace/nightly/cubietruck/layers/meta-sunxi/recipes-kernel/linux/linux-mainline_4.19.46.bb:do_package)
>  failed with exit code '1'
> 
> Here's another example:
> 
>   | DEBUG: Executing shell function sstate_create_package
>   | pigz: abort: internal threads error
>   | tar: 
> /z/jenkins-workspace/nightly/odroid-xu4/build/sstate-cache/d4/sstate:sqlite3:cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi:3.28.0:r0:cortexa15t2hf-neon-vfpv4:3:d4eb5692a1756a832d72fb2003a3d431108fbc736044747d33698ad7b6881dd9_package.tgz.herLUpYQ:
>  Wrote only 2048 of 10240 bytes
>   | tar: Child returned status 16
>   | tar: Error is not recoverable: exiting now
>   | WARNING: 
> /z/jenkins-workspace/nightly/odroid-xu4/build/tmp-glibc/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/sqlite3/3_3.28.0-r0/temp/run.sstate_create_package.24136:1
>  exit 1 from 'exit 1'
>   | DEBUG: Python function sstate_task_postfunc finished
>   | ERROR: Function failed: sstate_create_package (log file is located at 
> /z/jenkins-workspace/nightly/odroid-xu4/build/tmp-glibc/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/sqlite3/3_3.28.0-r0/temp/log.do_package.24136)
>   NOTE: recipe sqlite3-3_3.28.0-r0: task do_package: Failed
>   ERROR: Task 
> (/opt/oe/configs/z/jenkins-workspace/nightly/odroid-xu4/layers/openembedded-core/meta/recipes-support/sqlite/sqlite3_3.28.0.bb:do_package)
>  failed with exit code '1'
> 
> When I first started seeing this problem, I would see it quite frequently.
> Every morning, out of roughly 15 nightly builds, around 4-5 of them would have
> failed in this way. Back then I would also get a lot of errors that would
> report something along the lines of the following:
> 
>   fork: Resource temporarily unavailable
>   Cannot spawn thread (?)
> 
> I don't have an example of that error on hand, but I used to get a lot of
> those around the same time too.
> 
> My observations are:
> - I've never seen any of these errors with builds that I run by hand, oddly
>   enough, these errors only ever happen to builds that are run by Jenkins. I
>   have no idea if this is just a coincidence, or if there is something going
>   on related to kicking off a build from a large program (Jenkins)

This rings a bell. Jenkins spawns builds from java and the environment where
processes are spawned differs from normal console and login shells
considerably. Can't remember any details anymore, but environment variables
and various things like thread limits may be different in Java shells
compared to login ones. If pigz spawns too many threads and it happens on
too many bitbake tasks in parallel, at some point thread creation may fail
due to limits.

Hope this helps,

-Mikko

> - Back then these failures were quite frequent. Today, of the 20-ish or so
>   Jenkins builds that are kicked off every night, in a 2-week span I have only
>   2 such failures. So it seems that I've been able to reduce the occurrence
>   rate, but not eliminate it completely
> 
> - I haven't seen the "resource" failure in a while. I 

Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

On Fri, Jun 21, 2019 at 01:29:18PM +0100, Burton, Ross wrote:
> On Fri, 21 Jun 2019 at 12:11,  wrote:
> > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> > documentation could be updated too, e.g.
> > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages
> 
> Somehow I didn't notice it, thanks.
> 
> Pierre, can you rewrite this to use standard library instead of urllib3?

Also, the CVE db is updated using this custom task without link to
do_fetch, which means a fetchall task would not update the database for
off line NO_NETWORK builds.

Could the task be added as dependency to do_fetch() or are there some other
side effects?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

On Fri, Jun 21, 2019 at 02:03:36PM +0200, Alexander Kanavin wrote:
> On Fri, 21 Jun 2019 at 13:48,  wrote:
>
> >
> > Hmm, possibly? I cherry-picked the patches to sumo and saw this missing
> > dependency in my container.
> >
> > Did poky master switch from using host python to native after sumo?
> >
> 
> poky uses host python for some things and native python for other things.
> Generally where 'core python' is enough then we use the host python, but
> when additional libraries are needed, then native python plus those
> libraries is a better choice, as this doesn't require adding to the
> required host packages list, and allows better control over how they're
> built.

Currently do_populate_cve_db is executed using host python, and
the DEPENDS field is empty, and INHIBIT_DEFAULT_DEPS = "1". python-urllib3
is also not available in poky, but only in meta-openembedded.

I'm fine with additional python3-urllib3 dependency to bitbake build host
tools when running CVE check builds. Just wanted to document it.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

On Fri, Jun 21, 2019 at 01:42:11PM +0200, Alexander Kanavin wrote:
> On Fri, 21 Jun 2019 at 13:11,  wrote:
> 
> > This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
> > dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
> > documentation could be updated too, e.g.
> >
> > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages
> >
> > On my Debian 9 build container python3-urllib3 wasn't installed by default
> > or via other dependencies.
> >
> 
> Should this rather be provided via python3-native?

Hmm, possibly? I cherry-picked the patches to sumo and saw this missing
dependency in my container.

Did poky master switch from using host python to native after sumo?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/4] cve-update-db: New recipe to update CVE database

[Mikko.Rapeli]

Hi,

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment
dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe
documentation could be updated too, e.g.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

On my Debian 9 build container python3-urllib3 wasn't installed by default
or via other dependencies.

Thanks for these features!

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] elfutils: remove Elfutils-Exception and include GPLv2 for shared libraries

[Mikko.Rapeli]

Hi,

On Wed, May 15, 2019 at 12:53:44PM +0200, Martin Jansa wrote:
> Can we restore meta/files/common-licenses/Elfutils-Exception in oe-core to
> be used by meta-gplv2 version:
> 
> recipes-devtools/elfutils/elfutils_0.148.bb:LICENSE = "(GPL-2+ &
> Elfutils-Exception)"
> causing:
> do_rootfs: The license listed Elfutils-Exception was not in the licenses
> collected for recipe elfutils
> 
> or should I send patch for meta-gplv2 to restore it there together with
> another LICENSE_PATH += "${LAYERDIR}/licenses"?

I would do the latter and add the license to meta-gplv2.

-Mikko

> On Thu, Apr 11, 2019 at 6:01 PM Mikko Rapeli  wrote:
> 
> > Elfutils-Exception no longer exists after upstream release 0.154
> > and commit:
> >
> > commit de2ed97f33139af5c7a0811e4ec66fc896a13cf2
> > Author: Mark Wielaard 
> > Date:   Tue Jun 5 17:15:16 2012 +0200
> >
> > NEWS file in the sources says this about switch from GPLv2 to
> > GPLv3 license:
> >
> >
> > https://sourceware.org/git/?p=elfutils.git;a=blob;f=NEWS;h=5a06047f255e3c9a63828953759fd18a4ba9a3f3;hb=HEAD#l362
> >
> >  362 The license is now GPLv2/LGPLv3+ for the libraries and GPLv3+ for
> > stand-alone
> >  363 programs. There is now also a formal CONTRIBUTING document describing
> > how to
> >  364 submit patches.
> >
> > libasm, libdw and libelf are thus covered optionally by GPLv2 license.
> >
> > See also Debian copyright summary for elfutils:
> >
> > https://tracker.debian.org/media/packages/e/elfutils/copyright-0.176-1
> >
> > Signed-off-by: Mikko Rapeli 
> > ---
> >  meta/conf/licenses.conf  |  2 +-
> >  meta/files/common-licenses/Elfutils-Exception| 12 
> >  meta/recipes-devtools/elfutils/elfutils_0.176.bb | 14 +-
> >  3 files changed, 14 insertions(+), 14 deletions(-)
> >  delete mode 100644 meta/files/common-licenses/Elfutils-Exception
> >
> > v1:
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-April/281124.html
> >
> > v2:
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-April/281127.html
> >  * removed Elfutils-Exception
> >
> > v3:
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-April/281137.html
> >  * rebase to latest master branch with elfutils 0.176
> >
> > v4:
> >  * fixed typo in comment; GPLv3+ -> LGPLv3+
> >  * update commit message Debian copyright URL to same 0.176 version
> >
> > diff --git a/meta/conf/licenses.conf b/meta/conf/licenses.conf
> > index 1058084..7b01c57 100644
> > --- a/meta/conf/licenses.conf
> > +++ b/meta/conf/licenses.conf
> > @@ -16,7 +16,7 @@ SRC_DISTRIBUTE_LICENSES += "CC-BY-SA-1.0 CC-BY-SA-2.0
> > CC-BY-SA-2.5 CC-BY-SA-3.0
> >  SRC_DISTRIBUTE_LICENSES += "CDDL-1.0 CECILL-1.0 CECILL-2.0 CECILL-B
> > CECILL-C"
> >  SRC_DISTRIBUTE_LICENSES += "ClArtistic CPAL-1.0 CPL-1.0 CUA-OPL-1.0 DSSSL"
> >  SRC_DISTRIBUTE_LICENSES += "ECL-1.0 ECL-2.0 eCos-2.0 EDL-1.0 EFL-1.0
> > EFL-2.0"
> > -SRC_DISTRIBUTE_LICENSES += "Elfutils-Exception Entessa EPL-1.0 EPL-2.0
> > ErlPL-1.1"
> > +SRC_DISTRIBUTE_LICENSES += "Entessa EPL-1.0 EPL-2.0 ErlPL-1.1"
> >  SRC_DISTRIBUTE_LICENSES += "EUDatagrid EUPL-1.0 EUPL-1.1 Fair
> > Frameworx-1.0"
> >  SRC_DISTRIBUTE_LICENSES += "FreeType GFDL-1.1 GFDL-1.2 GFDL-1.3 GPL-1.0"
> >  SRC_DISTRIBUTE_LICENSES += "GPL-2.0 GPL-2.0-with-autoconf-exception"
> > diff --git a/meta/files/common-licenses/Elfutils-Exception
> > b/meta/files/common-licenses/Elfutils-Exception
> > deleted file mode 100644
> > index 627d769..000
> > --- a/meta/files/common-licenses/Elfutils-Exception
> > +++ /dev/null
> > @@ -1,12 +0,0 @@
> > -   This file describes the limits of the Exception under which you are
> > allowed
> > -   to distribute Non-GPL Code in linked combination with Red Hat elfutils.
> > -   For the full text of the license, please see one of the header files
> > -   included with the source distribution or the file COPYING found in the
> > -   top level directory of the source.
> > -
> > -   The Approved Interfaces are the functions declared in the files:
> > -
> > -   libelf.h
> > -   libdw.h
> > -   libdwfl.h
> > -
> > diff --git a/meta/recipes-devtools/elfutils/elfutils_0.176.bb
> > b/meta/recipes-devtools/elfutils/elfutils_0.176.bb
> > index fd901c9..4602544 100644
> > --- a/meta/recipes-devtools/elfutils/elfutils_0.176.bb
> > +++ b/meta/recipes-devtools/elfutils/elfutils_0.176.bb
> > @@ -1,7 +1,7 @@
> >  SUMMARY = "Utilities and libraries for handling compiled object files"
> >  HOMEPAGE = "https://sourceware.org/elfutils;
> >  SECTION = "base"
> > -LICENSE = "(GPLv3 & Elfutils-Exception)"
> > +LICENSE = "GPLv2 & LGPLv3+ & GPLv3+"
> >  LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
> >  DEPENDS = "libtool bzip2 zlib virtual/libintl"
> >  DEPENDS_append_libc-musl = " argp-standalone fts "
> > @@ -53,6 +53,18 @@ BBCLASSEXTEND = "native nativesdk"
> >
> >  # Package utilities separately
> >  PACKAGES =+ "${PN}-binutils libelf libasm libdw"
> > +
> > +# shared libraries are licensed 

Re: [OE-core] [PATCH 2/2] ptest: Add RDEPENDS frpm PN-ptest to PN package

[Mikko.Rapeli]

On Tue, May 14, 2019 at 10:56:04PM +0100, Richard Purdie wrote:
> Many different ptests are breaking as they assume that ${PN}-ptest
> depends on ${PN}. It doesn't currently but should. If we fix this, many
> different ptests start passing when they previously failed.
> 
> It does depend on fixing an issue in the dbus-test recipe which is done
> in the preceeding patch (mentioned in case this gets backported).

Awesome! Thanks for this. In our case we build many packages and their
ptests but they may not be installed to images by default. Thus this helps.

Same also applies to our SDK. We build more stuff for the SDK than what is
installed by default.

Reviewed-by: Mikko Rapeli 

> Signed-off-by: Richard Purdie 
> ---
>  meta/classes/ptest.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/ptest.bbclass b/meta/classes/ptest.bbclass
> index 936bf82736e..fa4c36ec769 100644
> --- a/meta/classes/ptest.bbclass
> +++ b/meta/classes/ptest.bbclass
> @@ -13,6 +13,7 @@ PTEST_ENABLED = "${@bb.utils.contains('DISTRO_FEATURES', 
> 'ptest', '1', '0', d)}"
>  PTEST_ENABLED_class-native = ""
>  PTEST_ENABLED_class-nativesdk = ""
>  PTEST_ENABLED_class-cross-canadian = ""
> +RDEPENDS_${PN}-ptest += "${PN}"
>  RDEPENDS_${PN}-ptest_class-native = ""
>  RDEPENDS_${PN}-ptest_class-nativesdk = ""
>  RRECOMMENDS_${PN}-ptest += "ptest-runner"
> -- 
> 2.20.1
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] openssh: usable sshd depends on rngd from rng-tools

[Mikko.Rapeli]

On Fri, May 10, 2019 at 01:18:21PM +0100, Burton, Ross wrote:
> I'm very dubious of the need to make this a dependency, as the
> hardware RNG should be used.  Note that there's been a slew of fixes
> to the kernel to enable this with modern stacks, for example:
> 
> https://github.com/torvalds/linux/commit/62f95ae805fa9e1e84d47d3219a97b2654b7
> 
> Maybe the IMX driver needs the same patch?

Possibly. Was running 4.14.89 kernel from NXP.

I would have benefitted from this patch, but it's ok if you reject it.

-Mikko

> Ross
> 
> On Thu, 9 May 2019 at 09:46, Rasmus Villemoes
>  wrote:
> >
> > On 08/05/2019 16.22, mikko.rap...@bmw.de wrote:
> > > On Wed, May 08, 2019 at 05:07:08PM +0300, Adrian Bunk wrote:
> > >> On Wed, May 08, 2019 at 04:26:09PM +0300, Mikko Rapeli wrote:
> > >>> Since openssl 1.1.1 and openssh which uses it, sshd
> > >>> startup is delayed. The delays range from few seconds
> > >>> to minutes and even to hours. The delays are visible
> > >>> in host keys generation and when sshd process is started
> > >>> in response to incoming TCP connection but is failing
> > >>> to provide SSH version string and clients or tests time out.
> > >>>
> > >>> In all cases traces show that sshd is waiting for getentropy()
> > >>> system call to return from Linux kernel, which returns only
> > >>> after kernel side random number pool is initialized. The pool
> > >>> is initialized via various entropy source which may be
> > >>> missing on embedded development boards or via rngd from
> > >>> rng-tools package from userspace. HW random number generation
> > >>> and kernel support help but rngd is till needed to feed that data
> > >>> back to the Linux kernel.
> > >>>
> > >>> Example from an NXP imx8 board shows that kernel random number pool
> > >>> initialization can take over 400 seconds without rngd,
> > >>> and with rngd it is initialized at around 4 seconds after boot.
> > >>> The completion of initialization is visible in kernel dmesg with line
> > >>> "random: crng init done".
> > >>> ...
> > >>> --- a/meta/recipes-connectivity/openssh/openssh_7.9p1.bb
> > >>> +++ b/meta/recipes-connectivity/openssh/openssh_7.9p1.bb
> > >>> @@ -148,6 +148,7 @@ FILES_${PN}-keygen = "${bindir}/ssh-keygen"
> > >>>
> > >>>  RDEPENDS_${PN} += "${PN}-scp ${PN}-ssh ${PN}-sshd ${PN}-keygen"
> > >>>  RDEPENDS_${PN}-sshd += "${PN}-keygen 
> > >>> ${@bb.utils.contains('DISTRO_FEATURES', 'pam', 'pam-plugin-keyinit 
> > >>> pam-plugin-loginuid', '', d)}"
> > >>> +RDEPENDS_${PN}-sshd += "rng-tools"
> > >>> ...
> > >>
> > >> This should only be an RRECOMMENDS so that people can opt out of it.
> > >>
> > >> E.g. CONFIG_RANDOM_TRUST_CPU in the kernel can solve the same
> > >> problem without using rng-tools on some platforms.
> > >
> > > I think this is a stronger dependency than just RRECOMMENDS. We build
> > > images and disable recommends but we care that sshd starts as fast as in
> > > sumo and earlier yocto releases for testing etc purposes.
> >
> > But why should boards without a hwrng be forced to spend disk space and
> > run-time resources on a daemon which they don't benefit from at all?
> >
> > I don't think RANDOM_TRUST_CPU works, though. That's just for stuff like
> > rdrand(), i.e. instructions built into the CPU - not for some other
> > on-chip hwrng. Whether those are used for seeding early on (i.e.,
> > without rngd doing its thing) depends on the ->quality parameter set by
> > the individual hwrng drivers. Very few set one, so they get assigned the
> > default_quality, which is a module parameter that defaults to 0.
> >
> > IOW, I think (but I haven't got around to testing this) one should set
> > rng_core.default_quality=512 (or something) on the kernel command line
> > to make the kernel start the hwrng_fill thread that will seed the
> > entropy pool from the hwrng. At least if I'm reading
> > drivers/char/hw_random/core.c correctly.
> >
> > Rasmus
> >
> >
> >
> >
> > --
> > ___
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] openssh: usable sshd depends on rngd from rng-tools

[Mikko.Rapeli]

On Wed, May 08, 2019 at 06:50:54PM +0300, Mark Hatle wrote:
> On 5/8/19 5:22 PM, mikko.rap...@bmw.de wrote:
> > On Wed, May 08, 2019 at 05:07:08PM +0300, Adrian Bunk wrote:
> >> On Wed, May 08, 2019 at 04:26:09PM +0300, Mikko Rapeli wrote:
> >>> Since openssl 1.1.1 and openssh which uses it, sshd
> >>> startup is delayed. The delays range from few seconds
> >>> to minutes and even to hours. The delays are visible
> >>> in host keys generation and when sshd process is started
> >>> in response to incoming TCP connection but is failing
> >>> to provide SSH version string and clients or tests time out.
> >>>
> >>> In all cases traces show that sshd is waiting for getentropy()
> >>> system call to return from Linux kernel, which returns only
> >>> after kernel side random number pool is initialized. The pool
> >>> is initialized via various entropy source which may be
> >>> missing on embedded development boards or via rngd from
> >>> rng-tools package from userspace. HW random number generation
> >>> and kernel support help but rngd is till needed to feed that data
> >>> back to the Linux kernel.
> >>>
> >>> Example from an NXP imx8 board shows that kernel random number pool
> >>> initialization can take over 400 seconds without rngd,
> >>> and with rngd it is initialized at around 4 seconds after boot.
> >>> The completion of initialization is visible in kernel dmesg with line
> >>> "random: crng init done".
> >>> ...
> >>> --- a/meta/recipes-connectivity/openssh/openssh_7.9p1.bb
> >>> +++ b/meta/recipes-connectivity/openssh/openssh_7.9p1.bb
> >>> @@ -148,6 +148,7 @@ FILES_${PN}-keygen = "${bindir}/ssh-keygen"
> >>>  
> >>>  RDEPENDS_${PN} += "${PN}-scp ${PN}-ssh ${PN}-sshd ${PN}-keygen"
> >>>  RDEPENDS_${PN}-sshd += "${PN}-keygen 
> >>> ${@bb.utils.contains('DISTRO_FEATURES', 'pam', 'pam-plugin-keyinit 
> >>> pam-plugin-loginuid', '', d)}"
> >>> +RDEPENDS_${PN}-sshd += "rng-tools"
> >>> ...
> >>
> >> This should only be an RRECOMMENDS so that people can opt out of it.
> >>
> >> E.g. CONFIG_RANDOM_TRUST_CPU in the kernel can solve the same 
> >> problem without using rng-tools on some platforms.
> > 
> > I think this is a stronger dependency than just RRECOMMENDS. We build
> > images and disable recommends but we care that sshd starts as fast as in
> > sumo and earlier yocto releases for testing etc purposes.
> 
> I agree with Adrian here.  It should be a recommend.  The system works without
> this, and there are valid use-cases without rngd existing on the system.  (In
> fact I have a couple of customers that would rather the system stall waiting 
> for
> 'real' entropy then use the values from rngd.)

Ok, at least I tried.

I can send a v2 with RRECOMMENDS instead though it is useless for me
since enabling recommends causes images to explode in size, complexity,
licensing with GPLv3 components etc.

> Note the warning on this page:  https://wiki.archlinux.org/index.php/Rng-tools
> 
> In a lot of cases, this dependency on urandom on an embedded target without 
> even
> a clock or entropy sources results in the system having effectively the same
> entropy each time it starts up -- even with rngd.  So you get a false sense of
> security.
> 
> Once you have a hardware (or other) rng source, the tool can be useful to
> increase the amount of entropy available however... but it all starts with
> having a reasonable starting source.
> 
> In your case, if using rngd has the entropy your device requires, based on 
> your
> system configuration (and you do not want recommends), then I think it's
> reasonable that you need to manually include it as an image dependency.

Yes, this lack of entropy on embedded devices is understood but in my case
rngd is the one reading /dev/hwrnd and pushing that back to kernel...

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] openssh: usable sshd depends on rngd from rng-tools

[Mikko.Rapeli]

On Wed, May 08, 2019 at 05:07:08PM +0300, Adrian Bunk wrote:
> On Wed, May 08, 2019 at 04:26:09PM +0300, Mikko Rapeli wrote:
> > Since openssl 1.1.1 and openssh which uses it, sshd
> > startup is delayed. The delays range from few seconds
> > to minutes and even to hours. The delays are visible
> > in host keys generation and when sshd process is started
> > in response to incoming TCP connection but is failing
> > to provide SSH version string and clients or tests time out.
> > 
> > In all cases traces show that sshd is waiting for getentropy()
> > system call to return from Linux kernel, which returns only
> > after kernel side random number pool is initialized. The pool
> > is initialized via various entropy source which may be
> > missing on embedded development boards or via rngd from
> > rng-tools package from userspace. HW random number generation
> > and kernel support help but rngd is till needed to feed that data
> > back to the Linux kernel.
> > 
> > Example from an NXP imx8 board shows that kernel random number pool
> > initialization can take over 400 seconds without rngd,
> > and with rngd it is initialized at around 4 seconds after boot.
> > The completion of initialization is visible in kernel dmesg with line
> > "random: crng init done".
> >...
> > --- a/meta/recipes-connectivity/openssh/openssh_7.9p1.bb
> > +++ b/meta/recipes-connectivity/openssh/openssh_7.9p1.bb
> > @@ -148,6 +148,7 @@ FILES_${PN}-keygen = "${bindir}/ssh-keygen"
> >  
> >  RDEPENDS_${PN} += "${PN}-scp ${PN}-ssh ${PN}-sshd ${PN}-keygen"
> >  RDEPENDS_${PN}-sshd += "${PN}-keygen 
> > ${@bb.utils.contains('DISTRO_FEATURES', 'pam', 'pam-plugin-keyinit 
> > pam-plugin-loginuid', '', d)}"
> > +RDEPENDS_${PN}-sshd += "rng-tools"
> >...
> 
> This should only be an RRECOMMENDS so that people can opt out of it.
> 
> E.g. CONFIG_RANDOM_TRUST_CPU in the kernel can solve the same 
> problem without using rng-tools on some platforms.

I think this is a stronger dependency than just RRECOMMENDS. We build
images and disable recommends but we care that sshd starts as fast as in
sumo and earlier yocto releases for testing etc purposes.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] eoqa: use bash to execute SDK test commands

[Mikko.Rapeli]

On Wed, May 08, 2019 at 08:41:21AM -0500, Joshua Watt wrote:
> On Wed, 2019-05-08 at 16:26 +0300, Mikko Rapeli wrote:
> > The commands only work with with bash. If /bin/sh is
> > dash like in Debian, the command execution fails with
> > errors like:
> 
> This might possibly be related to 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=11775 where it was
> discovered that in order to use dash, the CWD must be the same
> directory where the init script lives?

Yes, this is related.

In my case bitbake build is already called from bash and all scripts
seem to be calling bash correctly as /bin/bash, except for these
SDK tests.

Also on sumo, there is no meaningful debug output when these tests fail.

from oeqa.utils.subprocesstweak import errors_have_output
errors_have_output()

should maybe be used wherewhere with python subprocess...

-Mikko

> > 
> > Standard Output: /bin/sh: 5: export: --sysroot: bad variable name
> > 
> > and all SDK tests fail.
> > 
> > Signed-off-by: Mikko Rapeli 
> > ---
> >  meta/lib/oeqa/sdk/case.py  | 2 +-
> >  meta/lib/oeqa/sdk/utils/sdkbuildproject.py | 3 ++-
> >  2 files changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/lib/oeqa/sdk/case.py b/meta/lib/oeqa/sdk/case.py
> > index d8611c8..5334237 100644
> > --- a/meta/lib/oeqa/sdk/case.py
> > +++ b/meta/lib/oeqa/sdk/case.py
> > @@ -9,7 +9,7 @@ from oeqa.core.case import OETestCase
> >  class OESDKTestCase(OETestCase):
> >  def _run(self, cmd):
> >  return subprocess.check_output(". %s > /dev/null; %s;" % \
> > -(self.tc.sdk_env, cmd), shell=True,
> > +(self.tc.sdk_env, cmd), shell=True,
> > executable="/bin/bash",
> >  stderr=subprocess.STDOUT, universal_newlines=True)
> >  
> >  def fetch(self, workdir, dl_dir, url, archive=None):
> > diff --git a/meta/lib/oeqa/sdk/utils/sdkbuildproject.py
> > b/meta/lib/oeqa/sdk/utils/sdkbuildproject.py
> > index 6fed73e..eafbd7a 100644
> > --- a/meta/lib/oeqa/sdk/utils/sdkbuildproject.py
> > +++ b/meta/lib/oeqa/sdk/utils/sdkbuildproject.py
> > @@ -42,7 +42,8 @@ class SDKBuildProject(BuildProject):
> >  def _run(self, cmd):
> >  self.log("Running . %s; " % self.sdkenv + cmd)
> >  try:
> > -output = subprocess.check_output(". %s; " % self.sdkenv
> > + cmd, shell=True, stderr=subprocess.STDOUT)
> > +output = subprocess.check_output(". %s; " % self.sdkenv
> > + cmd, shell=True,
> > + executable='/bin/bash',
> > stderr=subprocess.STDOUT)
> >  except subprocess.CalledProcessError as exc:
> >  print(exc.output.decode('utf-8'))
> >  return exc.returncode
> > -- 
> > 1.9.1
> > 
> -- 
> Joshua Watt 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v3] elfutils: remove Elfutils-Exception and include GPLv2 for shared libraries

[Mikko.Rapeli]

On Thu, Apr 11, 2019 at 10:45:21AM -0500, Mark Hatle wrote:
> On 4/11/19 8:41 AM, Mikko Rapeli wrote:
> > Elfutils-Exception no longer exists after upstream release 0.154
> > and commit:
> > 
> > commit de2ed97f33139af5c7a0811e4ec66fc896a13cf2
> > Author: Mark Wielaard 
> > Date:   Tue Jun 5 17:15:16 2012 +0200
> > 
> > NEWS file in the sources says this about switch from GPLv2 to
> > GPLv3 license:
> > 
> > https://sourceware.org/git/?p=elfutils.git;a=blob;f=NEWS;h=5a06047f255e3c9a63828953759fd18a4ba9a3f3;hb=HEAD#l362
> > 
> >  362 The license is now GPLv2/LGPLv3+ for the libraries and GPLv3+ for 
> > stand-alone
> >  363 programs. There is now also a formal CONTRIBUTING document describing 
> > how to
> >  364 submit patches.
> > 
> > libasm, libdw and libelf are thus covered optionally by GPLv2 license.
> > 
> > See also Debian copyright summary for elfutils:
> > 
> > https://tracker.debian.org/media/packages/e/elfutils/copyright-0.175-1
> > 
> > Signed-off-by: Mikko Rapeli 
> > ---
> >  meta/conf/licenses.conf  |  2 +-
> >  meta/files/common-licenses/Elfutils-Exception| 12 
> >  meta/recipes-devtools/elfutils/elfutils_0.176.bb | 14 +-
> >  3 files changed, 14 insertions(+), 14 deletions(-)
> >  delete mode 100644 meta/files/common-licenses/Elfutils-Exception
> > 
> > v1: 
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-April/281124.html
> > 
> > v2: 
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-April/281127.html
> >  * removed Elfutils-Exception
> > 
> > v3:
> >  * rebase to latest master branch with elfutils 0.176
> > 
> > diff --git a/meta/conf/licenses.conf b/meta/conf/licenses.conf
> > index 1058084..7b01c57 100644
> > --- a/meta/conf/licenses.conf
> > +++ b/meta/conf/licenses.conf
> > @@ -16,7 +16,7 @@ SRC_DISTRIBUTE_LICENSES += "CC-BY-SA-1.0 CC-BY-SA-2.0 
> > CC-BY-SA-2.5 CC-BY-SA-3.0
> >  SRC_DISTRIBUTE_LICENSES += "CDDL-1.0 CECILL-1.0 CECILL-2.0 CECILL-B 
> > CECILL-C"
> >  SRC_DISTRIBUTE_LICENSES += "ClArtistic CPAL-1.0 CPL-1.0 CUA-OPL-1.0 DSSSL"
> >  SRC_DISTRIBUTE_LICENSES += "ECL-1.0 ECL-2.0 eCos-2.0 EDL-1.0 EFL-1.0 
> > EFL-2.0"
> > -SRC_DISTRIBUTE_LICENSES += "Elfutils-Exception Entessa EPL-1.0 EPL-2.0 
> > ErlPL-1.1"
> > +SRC_DISTRIBUTE_LICENSES += "Entessa EPL-1.0 EPL-2.0 ErlPL-1.1"
> >  SRC_DISTRIBUTE_LICENSES += "EUDatagrid EUPL-1.0 EUPL-1.1 Fair 
> > Frameworx-1.0"
> >  SRC_DISTRIBUTE_LICENSES += "FreeType GFDL-1.1 GFDL-1.2 GFDL-1.3 GPL-1.0"
> >  SRC_DISTRIBUTE_LICENSES += "GPL-2.0 GPL-2.0-with-autoconf-exception"
> > diff --git a/meta/files/common-licenses/Elfutils-Exception 
> > b/meta/files/common-licenses/Elfutils-Exception
> > deleted file mode 100644
> > index 627d769..000
> > --- a/meta/files/common-licenses/Elfutils-Exception
> > +++ /dev/null
> > @@ -1,12 +0,0 @@
> > -   This file describes the limits of the Exception under which you are 
> > allowed
> > -   to distribute Non-GPL Code in linked combination with Red Hat elfutils.
> > -   For the full text of the license, please see one of the header files
> > -   included with the source distribution or the file COPYING found in the
> > -   top level directory of the source.
> > -
> > -   The Approved Interfaces are the functions declared in the files:
> > -
> > -   libelf.h
> > -   libdw.h
> > -   libdwfl.h
> > -
> > diff --git a/meta/recipes-devtools/elfutils/elfutils_0.176.bb 
> > b/meta/recipes-devtools/elfutils/elfutils_0.176.bb
> > index fd901c9..cd824e2 100644
> > --- a/meta/recipes-devtools/elfutils/elfutils_0.176.bb
> > +++ b/meta/recipes-devtools/elfutils/elfutils_0.176.bb
> > @@ -1,7 +1,7 @@
> >  SUMMARY = "Utilities and libraries for handling compiled object files"
> >  HOMEPAGE = "https://sourceware.org/elfutils;
> >  SECTION = "base"
> > -LICENSE = "(GPLv3 & Elfutils-Exception)"
> > +LICENSE = "GPLv2 & LGPLv3+ & GPLv3+"'
> 
> I realize we don't have a choice but to conform to the licensing when we
> rebase... but has anyone looked at what links to elfutils to see if this could
> end up introducing any licensing issues?
> 
> The previous exception specifically allowed Non-GPL code to use interfaces, 
> I'm
> not sure if there is actually any non-GPL code using these interfaces without
> OE, but it may be worth a quick look.

At least poky and meta-openembedded users have GPLv2 licenses:

$ grep LICENSE $( egrep -l "DEPENDS.*elfutils" poky/meta* meta-openembedded -rn 
)
poky/meta/recipes-devtools/dwarfsrcfiles/dwarfsrcfiles.bb:LICENSE = "GPLv2+"
poky/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb:LICENSE = "GPL-2.0"
poky/meta/recipes-devtools/prelink/prelink_git.bb:LICENSE = "GPLv2"
poky/meta/recipes-connectivity/iproute2/iproute2.inc:LICENSE = "GPLv2+"
poky/meta/recipes-kernel/perf/perf.bb:LICENSE = "GPLv2"
poky/meta/recipes-kernel/linux/kernel-devsrc.bb:LICENSE = "GPLv2"
meta-openembedded/meta-oe/recipes-devtools/ltrace/ltrace_git.bb:LICENSE = 
"GPLv2"

Re: [OE-core] [PATCH v2] elfutils: remove Elfutils-Exception and include GPLv2 for shared libraries

[Mikko.Rapeli]

Sorry, this wasn't rebased to latest master with elfutils 0.176. I'll rebase
and resend.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] elfutils: update LICENSE to include GPLv2 for shared libraries

[Mikko.Rapeli]

On Thu, Apr 11, 2019 at 10:40:43AM +, 
openembedded-core-boun...@lists.openembedded.org wrote:
> On Thu, Apr 11, 2019 at 12:48:40PM +0300, Mikko Rapeli wrote:
> > NEWS file in the sources says this about switch from GPLv2 to
> > GPLv3 license:
> > 
> > https://sourceware.org/git/?p=elfutils.git;a=blob;f=NEWS;h=5a06047f255e3c9a63828953759fd18a4ba9a3f3;hb=HEAD#l362
> > 
> >  362 The license is now GPLv2/LGPLv3+ for the libraries and GPLv3+ for 
> > stand-alone
> >  363 programs. There is now also a formal CONTRIBUTING document describing 
> > how to
> >  364 submit patches.
> > 
> > libasm, libdw and libelf are thus covered optionally by GPLv2 license.
> > 
> > Signed-off-by: Mikko Rapeli 
> > ---
> >  meta/recipes-devtools/elfutils/elfutils_0.175.bb | 14 +-
> >  1 file changed, 13 insertions(+), 1 deletion(-)
> > 
> > diff --git a/meta/recipes-devtools/elfutils/elfutils_0.175.bb 
> > b/meta/recipes-devtools/elfutils/elfutils_0.175.bb
> > index b0b9ddc..6d1279a 100644
> > --- a/meta/recipes-devtools/elfutils/elfutils_0.175.bb
> > +++ b/meta/recipes-devtools/elfutils/elfutils_0.175.bb
> > @@ -1,7 +1,7 @@
> >  SUMMARY = "Utilities and libraries for handling compiled object files"
> >  HOMEPAGE = "https://sourceware.org/elfutils;
> >  SECTION = "base"
> > -LICENSE = "(GPLv3 & Elfutils-Exception)"
> > +LICENSE = "(GPLv2 & LGPLv3+ & GPLv3+ & Elfutils-Exception)"
> 
> Can anyone find the Elfutils-Exception in current 0.175 or sumo 0.170 sources?
> 
> Debian is also listing libraries with GPLv2+ | LGPLv3+ and executables
> with GPLv3+ in
> 
> https://tracker.debian.org/media/packages/e/elfutils/copyright-0.176-1
> 
> Should I also drop the Elfutils-Exception?

I will send a new version which drops Elfutils-Exception since it no longer
exists after tag elfutils-0.154 and commit:

commit de2ed97f33139af5c7a0811e4ec66fc896a13cf2
Author: Mark Wielaard 
Date:   Tue Jun 5 17:15:16 2012 +0200

Update name, license and contributor policy.

* Change name from "Red Hat elfutils" to "elfutils".
* Update license of standalone tools and test from GPLv2 to GPLv3+.
* Change license of libraries from GPLv2+exception to GPLv2/LGPLv3+.
* Add Developer Certificate of Origin based contributor policy.

top-level:

- COPYING: Upgraded from GPLv2 to GPLv3.
- CONTRIBUTING, COPYING-GPLv2, COPYING-LGPLv3: New files.
- NEWS: Added note about new contribution and license policy.
- Makefile.am: Updated to GPLv3, added new files to EXTRA_DIST.
- configure.ac: Update to GPLv3, changed AC_INIT name to 'elfutils'.

backends, lib, libasm, libcpu, libdw, libdwfl, libebl, libelf:

- All files updated to GPLv2/LGPLv3+. Except some very small files
  (<5 lines) which didn't have any headers at all before, the linker
  .maps files and the libcpu/defs files which only contain data and
  libelf/elf.h which comes from glibc and is under LGPLv2+.

config:

- elfutils.spec.in: Add new License: headers and new %doc files.
- Update all license headers to GPLv2/LGPLv3+ for files used by libs.

src, tests:

- All files updated to GPLv3+. Except for the test bz2 data files, the
  linker maps and script files and some very small files (<5 lines)
  that don't have any headers.

Signed-off-by: Richard Fontana 
Signed-off-by: Mark Wielaard 

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] elfutils: update LICENSE to include GPLv2 for shared libraries

[Mikko.Rapeli]

On Thu, Apr 11, 2019 at 12:48:40PM +0300, Mikko Rapeli wrote:
> NEWS file in the sources says this about switch from GPLv2 to
> GPLv3 license:
> 
> https://sourceware.org/git/?p=elfutils.git;a=blob;f=NEWS;h=5a06047f255e3c9a63828953759fd18a4ba9a3f3;hb=HEAD#l362
> 
>  362 The license is now GPLv2/LGPLv3+ for the libraries and GPLv3+ for 
> stand-alone
>  363 programs. There is now also a formal CONTRIBUTING document describing 
> how to
>  364 submit patches.
> 
> libasm, libdw and libelf are thus covered optionally by GPLv2 license.
> 
> Signed-off-by: Mikko Rapeli 
> ---
>  meta/recipes-devtools/elfutils/elfutils_0.175.bb | 14 +-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-devtools/elfutils/elfutils_0.175.bb 
> b/meta/recipes-devtools/elfutils/elfutils_0.175.bb
> index b0b9ddc..6d1279a 100644
> --- a/meta/recipes-devtools/elfutils/elfutils_0.175.bb
> +++ b/meta/recipes-devtools/elfutils/elfutils_0.175.bb
> @@ -1,7 +1,7 @@
>  SUMMARY = "Utilities and libraries for handling compiled object files"
>  HOMEPAGE = "https://sourceware.org/elfutils;
>  SECTION = "base"
> -LICENSE = "(GPLv3 & Elfutils-Exception)"
> +LICENSE = "(GPLv2 & LGPLv3+ & GPLv3+ & Elfutils-Exception)"

Can anyone find the Elfutils-Exception in current 0.175 or sumo 0.170 sources?

Debian is also listing libraries with GPLv2+ | LGPLv3+ and executables
with GPLv3+ in

https://tracker.debian.org/media/packages/e/elfutils/copyright-0.176-1

Should I also drop the Elfutils-Exception?

-Mikko

>  LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
>  DEPENDS = "libtool bzip2 zlib virtual/libintl"
>  DEPENDS_append_libc-musl = " argp-standalone fts "
> @@ -53,6 +53,18 @@ BBCLASSEXTEND = "native nativesdk"
>  
>  # Package utilities separately
>  PACKAGES =+ "${PN}-binutils libelf libasm libdw"
> +
> +# shared libraries are licensed GPLv2 or GPLv3+, binaries GPLv3+
> +# according to NEWS file:
> +# "The license is now GPLv2/LGPLv3+ for the libraries and GPLv3+ for 
> stand-alone
> +# programs. There is now also a formal CONTRIBUTING document describing how 
> to
> +# submit patches."
> +LICENSE_${PN}-binutils = "GPLv3+"
> +LICENSE_${PN} = "GPLv3+"
> +LICENSE_libelf = "GPLv2 | LGPLv3+"
> +LICENSE_libasm = "GPLv2 | LGPLv3+"
> +LICENSE_libdw = "GPLv2 | LGPLv3+"
> +
>  FILES_${PN}-binutils = "\
>  ${bindir}/eu-addr2line \
>  ${bindir}/eu-ld \
> -- 
> 1.9.1
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [thud][PATCH] Revert "boost: update to 1.69.0"

[Mikko.Rapeli]

On Mon, Mar 18, 2019 at 06:03:05PM +0100, Andreas Müller wrote:
> On Mon, Mar 18, 2019 at 5:45 PM Armin Kuster  wrote:
> >
> > This reverts commit a384248938ea9db096866bf4ec8678d35ca62a12.
> >
> > This package update slipped in doing the maint process. Removing it.

> Just my opinion - don't consider this as NAK.
> 
> * I already fixed the recipes that failed for me. For at least one the
> change is no more compatible to 1.68.0.
> * This makes PV going backwards
> 
> Thanks for addressing - what do others think?

I'm not using thud yet, but updating boost in stable branch would break
too many things and I would have to revert that change in our trees. Some boost
updates are in the end quite trivial and just require recompiling
everything but still, I would prefer that boost is not updated in stable
branches unless there is a huge security/stability issue with the old version.

Hope this helps,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] systemd: RDEPENDS on util-linux-umount

[Mikko.Rapeli]

On Mon, Feb 11, 2019 at 12:08:46PM +, André Draszik wrote:
> Please ignore this patch. Looks like a red-herring. Sorry for the noise.

FWIW, I would like to see this patch merged. Had some issues in the past
with busybox umount and added same change as a bbappend.

-Mikko

> On Mon, 2019-02-11 at 12:04 +, André Draszik wrote:
> > From: André Draszik 
> > 
> > It looks like there is an implicit dependency on util-linux'
> > umount - as otherwise when using busybox' umount we see a
> > long delay on shutdown / reboot.
> > 
> > [YOCTO #13058]
> > 
> > Signed-off-by: André Draszik 
> > 
> > ---
> > this should only be merged with or after the util-linux
> > packaging rework
> > ---
> >  meta/recipes-core/systemd/systemd_239.bb | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/recipes-core/systemd/systemd_239.bb b/meta/recipes-
> > core/systemd/systemd_239.bb
> > index f843f588bd..e2dfe639b3 100644
> > --- a/meta/recipes-core/systemd/systemd_239.bb
> > +++ b/meta/recipes-core/systemd/systemd_239.bb
> > @@ -556,7 +556,7 @@ FILES_${PN} = " ${base_bindir}/* \
> >  
> >  FILES_${PN}-dev += "${base_libdir}/security/*.la ${datadir}/dbus-
> > 1/interfaces/ ${sysconfdir}/rpm/macros.systemd"
> >  
> > -RDEPENDS_${PN} += "kmod dbus util-linux-mount udev (= ${EXTENDPKGV})
> > util-linux-agetty util-linux-fsck"
> > +RDEPENDS_${PN} += "kmod dbus util-linux-mount util-linux-umount udev (=
> > ${EXTENDPKGV}) util-linux-agetty util-linux-fsck"
> >  RDEPENDS_${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'serial-getty-
> > generator', '', 'systemd-serialgetty', d)}"
> >  RDEPENDS_${PN} += "volatile-binds update-rc.d systemd-conf"
> >  
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [RFC][PATCH 2/2] buildhistory: support generating md5sum of files

[Mikko.Rapeli]

On Tue, Jan 08, 2019 at 11:36:24AM +0100, Jacob Kroon wrote:
> On Tue, Jan 8, 2019 at 11:32 AM  wrote:
> >
> > On Mon, Jan 07, 2019 at 03:17:00PM +0100, Jacob Kroon wrote:
> > > On Sun, Jan 6, 2019 at 7:14 PM Jacob Kroon  wrote:
> > > >
> > > > Introduce 'md5' in BUILDHISTORY_FEATURES and enable it by default
> > > > when doing reproducible builds.
> > > >
> > > > When enabled this will additionally create:
> > > >
> > > >   files-in-package-md5.txt
> > > >   files-in-image-md5.txt
> > > >   files-in-sdk-md5.txt
> > > >
> > > > containing the md5 checksums of regular files.
> > > >
> > > > Signed-off-by: Jacob Kroon 
> > > > ---
> > > >  meta/classes/buildhistory.bbclass | 10 --
> > > >  1 file changed, 8 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/meta/classes/buildhistory.bbclass 
> > > > b/meta/classes/buildhistory.bbclass
> > > > index 33eb1b00f6..00f0701dec 100644
> > > > --- a/meta/classes/buildhistory.bbclass
> > > > +++ b/meta/classes/buildhistory.bbclass
> > > > @@ -7,7 +7,8 @@
> > > >  # Copyright (C) 2007-2011 Koen Kooi 
> > > >  #
> > > >
> > > > -BUILDHISTORY_FEATURES ?= "image package sdk"
> > > > +BUILDHISTORY_FEATURES ?= "image package sdk \
> > > > +  ${@ "md5" if 
> > > > bb.utils.to_boolean(d.getVar('BUILD_REPRODUCIBLE_BINARIES')) else ""}"
> > > >  BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory"
> > > >  BUILDHISTORY_DIR_IMAGE = 
> > > > "${BUILDHISTORY_DIR}/images/${MACHINE_ARCH}/${TCLIBC}/${IMAGE_BASENAME}"
> > > >  BUILDHISTORY_DIR_PACKAGE = 
> > > > "${BUILDHISTORY_DIR}/packages/${MULTIMACH_TARGET_SYS}/${PN}"
> > > > @@ -526,7 +527,12 @@ buildhistory_list_files() {
> > > > eval ${FAKEROOTENV} ${FAKEROOTCMD} $find_cmd
> > > > else
> > > > eval $find_cmd
> > > > -   fi | sort -k5 | sed 's/ * -> $//' > $2 )
> > > > +   fi | sort -k5 | sed 's/ * -> $//' > $2
> > > > +   if [ "${@bb.utils.contains('BUILDHISTORY_FEATURES', 'md5', '1', 
> > > > '0', d)}" = "1" ] ; then
> > > > +   md5filename=$(echo $2 | sed 's/\.txt$/-md5.txt/')
> > > > +   find -type f | xargs -I{} -n1 md5sum {} | sort -k2 > 
> > > > $md5filename
> > > > +   [ -s $md5filename ] || rm $md5filename # remove result 
> > > > if empty
> > >
> > > I added this remove because I thought it didn't make sense to keep
> > > empty files around, but I now realize that the "files-in-package.txt"
> > > file is kept around, even if empty. Is there a preference on what to
> > > do here ?
> >
> > FWIW, I'm wiping the all buildhistory data with external scripts before 
> > doing a
> > clean build. Basically a "git rm -rf *" in buildhistory directory. Otherwise
> > stale data about images, packages etc which are no longer built remain in
> > buildhistory.
> 
> I think one can also do:
> 
> BB_ENV_EXTRAWHITE=BUILDHISTORY_RESET BUILDHISTORY_RESET=1 bitbake 
> 
> (see buildhistory.bbclass)

Thanks for the hint!

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [RFC][PATCH 2/2] buildhistory: support generating md5sum of files

[Mikko.Rapeli]

On Mon, Jan 07, 2019 at 03:17:00PM +0100, Jacob Kroon wrote:
> On Sun, Jan 6, 2019 at 7:14 PM Jacob Kroon  wrote:
> >
> > Introduce 'md5' in BUILDHISTORY_FEATURES and enable it by default
> > when doing reproducible builds.
> >
> > When enabled this will additionally create:
> >
> >   files-in-package-md5.txt
> >   files-in-image-md5.txt
> >   files-in-sdk-md5.txt
> >
> > containing the md5 checksums of regular files.
> >
> > Signed-off-by: Jacob Kroon 
> > ---
> >  meta/classes/buildhistory.bbclass | 10 --
> >  1 file changed, 8 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/buildhistory.bbclass 
> > b/meta/classes/buildhistory.bbclass
> > index 33eb1b00f6..00f0701dec 100644
> > --- a/meta/classes/buildhistory.bbclass
> > +++ b/meta/classes/buildhistory.bbclass
> > @@ -7,7 +7,8 @@
> >  # Copyright (C) 2007-2011 Koen Kooi 
> >  #
> >
> > -BUILDHISTORY_FEATURES ?= "image package sdk"
> > +BUILDHISTORY_FEATURES ?= "image package sdk \
> > +  ${@ "md5" if 
> > bb.utils.to_boolean(d.getVar('BUILD_REPRODUCIBLE_BINARIES')) else ""}"
> >  BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory"
> >  BUILDHISTORY_DIR_IMAGE = 
> > "${BUILDHISTORY_DIR}/images/${MACHINE_ARCH}/${TCLIBC}/${IMAGE_BASENAME}"
> >  BUILDHISTORY_DIR_PACKAGE = 
> > "${BUILDHISTORY_DIR}/packages/${MULTIMACH_TARGET_SYS}/${PN}"
> > @@ -526,7 +527,12 @@ buildhistory_list_files() {
> > eval ${FAKEROOTENV} ${FAKEROOTCMD} $find_cmd
> > else
> > eval $find_cmd
> > -   fi | sort -k5 | sed 's/ * -> $//' > $2 )
> > +   fi | sort -k5 | sed 's/ * -> $//' > $2
> > +   if [ "${@bb.utils.contains('BUILDHISTORY_FEATURES', 'md5', '1', 
> > '0', d)}" = "1" ] ; then
> > +   md5filename=$(echo $2 | sed 's/\.txt$/-md5.txt/')
> > +   find -type f | xargs -I{} -n1 md5sum {} | sort -k2 > 
> > $md5filename
> > +   [ -s $md5filename ] || rm $md5filename # remove result if 
> > empty
> 
> I added this remove because I thought it didn't make sense to keep
> empty files around, but I now realize that the "files-in-package.txt"
> file is kept around, even if empty. Is there a preference on what to
> do here ?

FWIW, I'm wiping the all buildhistory data with external scripts before doing a
clean build. Basically a "git rm -rf *" in buildhistory directory. Otherwise
stale data about images, packages etc which are no longer built remain in
buildhistory.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/5] RFC image_types.bbclass: add image size limit for tar image type

[Mikko.Rapeli]

On Thu, Nov 29, 2018 at 02:04:14PM +, Richard Purdie wrote:
> On Thu, 2018-11-29 at 14:21 +0200, Mikko Rapeli wrote:
> > If IMAGE_ROOTFS_SIZE_LIMIT is defined in image configuration, then
> > build will fail if for tar image type the uncompressed tar ball size
> > exceeds the value, as reported by 'du -ks'.
> > 
> > This check could be extended to other image types as well.
> > 
> > There already exists a check with IMAGE_ROOTFS_SIZE variable
> > but I could not figure out how to actually use it. It does
> > some quite complex overhead calculations which did not seem
> > to work for me on sumo.
> > 
> > When the image size is exceeded, build fails like this:
> > 
> > ERROR: image-1.0-r0 do_image_tar: Image size 170712 of
> > /home/builder/src/base/build/tmp/work/linux/image/1.0-r0/deploy-
> > image-complete/image.rootfs.tar reported by 'du -ks' is larger than
> > limit IMAGE_ROOTFS_SIZE_LIMIT 17
> 
> What about IMAGE_ROOTFS_MAXSIZE?
>
> I think IMAGE_ROOTFS_SIZE is about adding extra space to the image, not
> limiting its size...

Yea, forgot to mention that I tried that too but got inconsisten results.
I know it's bad but it was easier for me to add this new test than to figure
out what's wrong with current yocto image size checks. Hence RFC.

-Mikko

> Cheers,
> 
> Richard
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH RFC] insane.bbclass: add buildpaths_cmake and buildpaths_pkgconfig checks

[Mikko.Rapeli]

On Tue, Sep 25, 2018 at 07:29:22PM +0100, richard.pur...@linuxfoundation.org 
wrote:
> On Tue, 2018-09-25 at 17:11 +, mikko.rap...@bmw.de wrote:
> > On Tue, Sep 25, 2018 at 04:19:39PM +0100, richard.purdie@linuxfoundat
> > ion.org wrote:
> > > On Tue, 2018-09-25 at 11:21 +, mikko.rap...@bmw.de wrote:
> > > > On Tue, Sep 25, 2018 at 12:08:24PM +0100, Richard Purdie wrote:
> > > > > I suspect we need to talk to cmake upstream about fixing this
> > > > > properly
> > > > > but adding something in the class may be an option until a
> > > > > better
> > > > > upstream solution can be found.
> > > > > 
> > > > > I am puzzled by the need to add a .pc file path check since I
> > > > > thought
> > > > > there was already a test for that in insane.bbclass?
> > > > > 
> > > > > package_qa_check_staged maybe? "Check staged la and pc files
> > > > > for
> > > > > common
> > > > > problems like references to the work directory."
> > > > 
> > > > That check is not enabled by default. At least bash is producing
> > > > a
> > > > broken
> > > > bash.pc (and several other files like Makefile.in and bashbug) in
> > > > sumo
> > > > with embedded absolute paths to build sysroot.
> > > 
> > > I still felt I was missing something:
> > > 
> > > ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig
> > > la 
> > > 
> > > which sets "pkgconfig" and "la". package_qa_check_staged calls
> > > package_qa_handle_error("la",...) and
> > > package_qa_handle_error("pkgconfig"...).
> > > 
> > > do_qa_staging calls package_qa_check_staged() and is triggered by:
> > > 
> > > do_populate_sysroot[postfuncs] += "do_qa_staging "
> > > 
> > > which is set for everything and should be running on master?
> > > 
> > > I had a look at bash.pc in a random local build and there is no
> > > hardcoded path in it for master. I then found a sumo build and
> > > looked
> > > at bash.pc there and also no hardcoded paths.
> > > 
> > > The issues would have appeared to have been fixed by:
> > > 
> > > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=dabbfe23de0615
> > > a958fac31b83684ade024cf17d 
> > > which should be in sumo.
> > 
> > Cool, but this ignores nativesdk, which is where I saw the problems
> > with this test. Somehow missed that nativesdk part in my reply,
> > possibly because of plain recipe name in the error message.
> 
> With the addition of nativesdk, this starts to make more sense...
> 
> > > What may be the real issue is that sanity checker is quite specific
> > > about what its looking for and does do:
> > > 
> > > file_content = file_content.replace(recipesysroot, "")
> > > 
> > > which may make sense for .la files but perhaps not .pc files, I'd
> > > guess
> > > its to stop compiler flags triggering errors. 
> > > 
> > > So the real fix here may be to remove that line from the .pc check,
> > > at
> > > least in the target case (native case it would make sense)?
> > 
> > And .cmake files?
> 
> We should add something to cover those. I was just worried about why
> tests I thought were there weren't working. I do think we should
> fix/improve the existing pkgconfig test rather than add another similar
> one.

Ok, I'll see what I can do about the tests...

> > I've fixed issues found by this test in my tree by following the
> > pattern from these "improve reproducibility" fixes. Some recipes were
> > installing generated .cmake files from build tree (bad, bad) and
> > several recipes were somehow generating .cmake files with absolute
> > paths for libraries. It's completely unclear to me when and why CMake
> > decides to fill in absolute paths, like with libical.
> 
> I'm afraid you probably know more about cmake than I do at this
> point...

I wonder what would be exposed if recipe builds would really be confined
in their sysroots with seccomp or filesystem namespaces..

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH RFC] insane.bbclass: add buildpaths_cmake and buildpaths_pkgconfig checks

[Mikko.Rapeli]

On Tue, Sep 25, 2018 at 04:19:39PM +0100, richard.pur...@linuxfoundation.org 
wrote:
> On Tue, 2018-09-25 at 11:21 +, mikko.rap...@bmw.de wrote:
> > On Tue, Sep 25, 2018 at 12:08:24PM +0100, Richard Purdie wrote:
> > > I suspect we need to talk to cmake upstream about fixing this
> > > properly
> > > but adding something in the class may be an option until a better
> > > upstream solution can be found.
> > > 
> > > I am puzzled by the need to add a .pc file path check since I
> > > thought
> > > there was already a test for that in insane.bbclass?
> > > 
> > > package_qa_check_staged maybe? "Check staged la and pc files for
> > > common
> > > problems like references to the work directory."
> > 
> > That check is not enabled by default. At least bash is producing a
> > broken
> > bash.pc (and several other files like Makefile.in and bashbug) in
> > sumo
> > with embedded absolute paths to build sysroot.
> 
> I still felt I was missing something:
> 
> ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la 
> 
> which sets "pkgconfig" and "la". package_qa_check_staged calls
> package_qa_handle_error("la",...) and
> package_qa_handle_error("pkgconfig"...).
> 
> do_qa_staging calls package_qa_check_staged() and is triggered by:
> 
> do_populate_sysroot[postfuncs] += "do_qa_staging "
> 
> which is set for everything and should be running on master?
> 
> I had a look at bash.pc in a random local build and there is no
> hardcoded path in it for master. I then found a sumo build and looked
> at bash.pc there and also no hardcoded paths.
> 
> The issues would have appeared to have been fixed by:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=dabbfe23de0615a958fac31b83684ade024cf17d
>  
> which should be in sumo.

Cool, but this ignores nativesdk, which is where I saw the problems with
this test. Somehow missed that nativesdk part in my reply, possibly because
of plain recipe name in the error message.

> What may be the real issue is that sanity checker is quite specific
> about what its looking for and does do:
> 
> file_content = file_content.replace(recipesysroot, "")
> 
> which may make sense for .la files but perhaps not .pc files, I'd guess
> its to stop compiler flags triggering errors. 
> 
> So the real fix here may be to remove that line from the .pc check, at
> least in the target case (native case it would make sense)?

And .cmake files?

I've fixed issues found by this test in my tree by following the pattern from
these "improve reproducibility" fixes. Some recipes were installing generated
.cmake files from build tree (bad, bad) and several recipes were somehow
generating .cmake files with absolute paths for libraries. It's completely
unclear to me when and why CMake decides to fill in absolute paths, like with
libical.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH RFC] insane.bbclass: add buildpaths_cmake and buildpaths_pkgconfig checks

[Mikko.Rapeli]

On Tue, Sep 25, 2018 at 12:08:24PM +0100, Richard Purdie wrote:
> On Tue, 2018-09-25 at 10:44 +, mikko.rap...@bmw.de wrote:
> > On Tue, Sep 25, 2018 at 01:28:13PM +0300, Mikko Rapeli wrote:
> > > And enable them by default as ERROR_QA. Reason is that
> > > absolute build directory paths in CMake .cmake modules
> > > and in pkg-config .pc files cause recipe builds to escape
> > > their recipe specific sysroots and triggers hard to debug
> > > and timing sensitive build failures. It's better to fail
> > > early.
> > > 
> > > A failure from sumo version of libical looks like:
> > > 
> > > ERROR: libical-2.0.0-r0 do_package_qa: QA Issue: CMake module
> > > /work/i586-poky-linux/libical/2.0.0-r0/packages-split/libical-
> > > dev/usr/lib/cmake/LibIcal/LibIcalTargets-noconfig.cmake contains
> > > reference to tmpdir which causes build raceconditions between
> > > recipes [buildpaths_cmake]
> > > ERROR: libical-2.0.0-r0 do_package_qa: QA run found fatal errors.
> > > Please consider fixing them.
> > > ERROR: libical-2.0.0-r0 do_package_qa: Function failed:
> > > do_package_qa
> > > ERROR: Logfile of failure stored in:
> > > /home/builder/src/yocto/poky/build/tmp/work/i586-poky-
> > > linux/libical/2.0.0-r0/temp/log.do_package_qa.4934
> > > NOTE: recipe libical-2.0.0-r0: task do_package_qa: Failed
> > > ERROR: Task (/home/builder/src/yocto/poky/meta/recipes-
> > > support/libical/libical_2.0.0.bb:do_package_qa) failed with exit
> > > code '1'
> > > 
> > > For some reason libical from poky master branch is not affected.
> > 
> > The reason why master branch is not affected is:
> > 
> > commit 26cccb93059b8963651b7d17cea2ee95f52633b7
> > Author: Juro Bystricky 
> > Date:   Tue Mar 20 15:36:36 2018 -0700
> > 
> > libical-dev_2.0: improve reproducibility
> > 
> > Remove build host references from distributed files.
> > 
> > (From OE-Core rev: 20f2670e755bcbf90b2b6c08192c022fe7e7eaad)
> > 
> > Signed-off-by: Juro Bystricky 
> > Signed-off-by: Ross Burton 
> > Signed-off-by: Richard Purdie  > >
> > 
> > diff --git a/meta/recipes-support/libical/libical_2.0.0.bb
> > b/meta/recipes-support/libical/libical_2.0.0.
> > index dcc21cc..daa47ab 100644
> > --- a/meta/recipes-support/libical/libical_2.0.0.bb
> > +++ b/meta/recipes-support/libical/libical_2.0.0.bb
> > @@ -17,3 +17,10 @@ SRC_URI[sha256sum] =
> > "654c11f759c19237be39f6ad401d917e5a05f36f1736385ed958e60cf2
> >  UPSTREAM_CHECK_URI = "https://github.com/libical/libical/releases;
> >  
> >  inherit cmake pkgconfig
> > +
> > +do_install_append_class-target () {
> > +# Remove build host references
> > +sed -i \
> > +   -e 's,${STAGING_LIBDIR},${libdir},g' \
> > +   ${D}${libdir}/cmake/LibIcal/LibIcalTargets-noconfig.cmake
> > +}
> > 
> > Now I'm struggling horribly with the same problem in various custom
> > CMake modules, so how about doing that same STAGING_LIBDIR to libdir
> > switch for all CMake modules automatically in cmake.bbclass?
> > 
> > I tried to correctly fix the libical CMake files too with help from
> > #cmake, but eventually had to give up. libical has an embedded
> > FindICU.cmake module but it seems like the upstream FindICU.cmake
> > ends
> > up producing the same absolute paths while it does fix some other
> > bugs, and of course introduces new ones like no longer providing
> > ICU_I18N_FOUND variable when i18n ICU module is found.
> 
> I suspect we need to talk to cmake upstream about fixing this properly
> but adding something in the class may be an option until a better
> upstream solution can be found.
> 
> I am puzzled by the need to add a .pc file path check since I thought
> there was already a test for that in insane.bbclass?
> 
> package_qa_check_staged maybe? "Check staged la and pc files for common
> problems like references to the work directory."

That check is not enabled by default. At least bash is producing a broken
bash.pc (and several other files like Makefile.in and bashbug) in sumo
with embedded absolute paths to build sysroot.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH RFC] insane.bbclass: add buildpaths_cmake and buildpaths_pkgconfig checks

[Mikko.Rapeli]

On Tue, Sep 25, 2018 at 01:28:13PM +0300, Mikko Rapeli wrote:
> And enable them by default as ERROR_QA. Reason is that
> absolute build directory paths in CMake .cmake modules
> and in pkg-config .pc files cause recipe builds to escape
> their recipe specific sysroots and triggers hard to debug
> and timing sensitive build failures. It's better to fail
> early.
> 
> A failure from sumo version of libical looks like:
> 
> ERROR: libical-2.0.0-r0 do_package_qa: QA Issue: CMake module 
> /work/i586-poky-linux/libical/2.0.0-r0/packages-split/libical-dev/usr/lib/cmake/LibIcal/LibIcalTargets-noconfig.cmake
>  contains reference to tmpdir which causes build raceconditions between 
> recipes [buildpaths_cmake]
> ERROR: libical-2.0.0-r0 do_package_qa: QA run found fatal errors. Please 
> consider fixing them.
> ERROR: libical-2.0.0-r0 do_package_qa: Function failed: do_package_qa
> ERROR: Logfile of failure stored in: 
> /home/builder/src/yocto/poky/build/tmp/work/i586-poky-linux/libical/2.0.0-r0/temp/log.do_package_qa.4934
> NOTE: recipe libical-2.0.0-r0: task do_package_qa: Failed
> ERROR: Task 
> (/home/builder/src/yocto/poky/meta/recipes-support/libical/libical_2.0.0.bb:do_package_qa)
>  failed with exit code '1'
> 
> For some reason libical from poky master branch is not affected.

The reason why master branch is not affected is:

commit 26cccb93059b8963651b7d17cea2ee95f52633b7
Author: Juro Bystricky 
Date:   Tue Mar 20 15:36:36 2018 -0700

libical-dev_2.0: improve reproducibility

Remove build host references from distributed files.

(From OE-Core rev: 20f2670e755bcbf90b2b6c08192c022fe7e7eaad)

Signed-off-by: Juro Bystricky 
Signed-off-by: Ross Burton 
Signed-off-by: Richard Purdie 

diff --git a/meta/recipes-support/libical/libical_2.0.0.bb 
b/meta/recipes-support/libical/libical_2.0.0.
index dcc21cc..daa47ab 100644
--- a/meta/recipes-support/libical/libical_2.0.0.bb
+++ b/meta/recipes-support/libical/libical_2.0.0.bb
@@ -17,3 +17,10 @@ SRC_URI[sha256sum] = 
"654c11f759c19237be39f6ad401d917e5a05f36f1736385ed958e60cf2
 UPSTREAM_CHECK_URI = "https://github.com/libical/libical/releases;
 
 inherit cmake pkgconfig
+
+do_install_append_class-target () {
+# Remove build host references
+sed -i \
+   -e 's,${STAGING_LIBDIR},${libdir},g' \
+   ${D}${libdir}/cmake/LibIcal/LibIcalTargets-noconfig.cmake
+}

Now I'm struggling horribly with the same problem in various custom
CMake modules, so how about doing that same STAGING_LIBDIR to libdir
switch for all CMake modules automatically in cmake.bbclass?

I tried to correctly fix the libical CMake files too with help from
#cmake, but eventually had to give up. libical has an embedded
FindICU.cmake module but it seems like the upstream FindICU.cmake ends
up producing the same absolute paths while it does fix some other
bugs, and of course introduces new ones like no longer providing
ICU_I18N_FOUND variable when i18n ICU module is found.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] kernel.bbclass: allow exporting files from kernel recipes to sysroot

[Mikko.Rapeli]

On Mon, Sep 24, 2018 at 02:56:49PM +0100, richard.pur...@linuxfoundation.org 
wrote:
> On Mon, 2018-09-24 at 09:43 -0400, Bruce Ashfield wrote:
> > > On Mon, 2018-09-24 at 13:20 +, mikko.rap...@bmw.de wrote:
> > > > On Mon, Sep 24, 2018 at 02:11:13PM +0100, Richard Purdie wrote:
> > > > > On Mon, 2018-09-24 at 12:19 +, mikko.rap...@bmw.de wrote:
> > > > > > > That was one old way, but not the only. And not for
> > > > > > > exposing
> > > > > > > non
> > > > > > > uapi
> > > > > > > headers.
> > > > > > 
> > > > > > What other ways exist?
> > > > > > 
> > > > > > I don't care how, but I must export custom kernel specific
> > > > > > headers
> > > > > > and
> > > > > > other files to other recipes in a build in ways which are
> > > > > > compatible
> > > > > > with
> > > > > > yocto upstream.
> > > > > > 
> > > > > > I have not seen any documented ways for this.
> > > > > 
> > > > > It may not be documented, perhaps because its actually very
> > > > > simple.
> > > > > 
> > > > > Any recipe can expose headers into the recipe sysroot, they
> > > > > simply
> > > > > install them where needed in do_install as normal.
> > > > > 
> > > > > So all you need is a recipe which installs the right headers
> > > > > and
> > > > > then
> > > > > you DEPEND on that recipe. Where that recipe gets the headers
> > > > > isn't
> > > > > relevant.
> > > > 
> > > > No, this does not work on sumo. My patch is needed for this to
> > > > work.
> > > > 
> > > > Without my patch, users of kernel.bbclass have zero files in
> > > > tmp/sysroot-components even if they install extra files and extra
> > > > header only binary packages.
> > > > 
> > > > A generated image or SDK will have the files if the binary
> > > > package is
> > > > installed but sysroot not.
> > > 
> > > I was replying from the perspective of how this should work in
> > > general.
> > > I agree that for this to work with a kernel recipe we do need the
> > > change that started this thread and that is probably a reasonable
> > > thing
> > > to do.
> > 
> > We have a Yocto bugzilla that can be closed if you are ok with the
> > approach.
> > 
> > To answer the other question about what I've done (and proposed
> > before) was
> > to maintain the kernel.bbclass protections, all call the staging
> > routines myself.
> > 
> > i.e.
> > 
> > do_install_append() {
> > make headers_install
> > INSTALL_HDR_PATH=${D}${KERNEL_ALT_HEADER_DIR}
> > 
> > # remove export artifacts
> > find ${D}${KERNEL_ALT_HEADER_DIR} -name .install -exec rm {}
> > \;
> > find ${D}${KERNEL_ALT_HEADER_DIR} -name ..install.cmd -exec
> > rm {} \;
> > }
> > 
> > sysroot_stage_all_append() {
> > sysroot_stage_dir ${D}${KERNEL_ALT_HEADER_DIR}
> > ${SYSROOT_DESTDIR}${KERNEL_ALT_HEADER_DIR}
> > }
> > 
> > And that works to get things exported.
> 
> I'm fine with this approach, I'm kind of surprised anyone thinks
> otherwise as I've always assumed this was what people were doing!
> 
> I'd propose that:
> 
> a) We document the above approach. I prefer it to Mikko's patch since
> it doesn't mess with the blacklist and installs exactly what the recipe
> wants to install which is how we normally write recipes.
> b) To document it, I suggest a comment/reference in kernel.bbclass and
> we add something somewhere in the manual. Adding Scott Rifenbark to cc
> so he can help us sort this out.
> c) Close out the bug Bruce mentions with this documentation as a
> reference.
> 
> I think this means we probably don't need Mikko's patch and it is
> mainly a documentation issue?

My only complaint is that it's not obvious in a kernel recipe that
more than do_install() is needed to export files to sysroot.
It's easy to miss the sysroot_stage_all_append() step.

Overwriting files from kernel recipe fails when they are used to prepare
sysroots for user recipes, but not when the kernel recipe is build:

ERROR: linux-image-1.0.0-r21 do_prepare_recipe_sysroot: The file 
/usr/include/scsi/scsi_bsg_fc.h is installed by both linux-libc-headers and 
linux, aborting
ERROR: linux-image-1.0.0-r21 do_prepare_recipe_sysroot: Function failed: 
extend_recipe_sysroot

In this case linux recipe added kernel specific headers to /usr/include
which conflict with linux-libc-headers and this was only cought when building
the kernel image.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] kernel.bbclass: allow exporting files from kernel recipes to sysroot

[Mikko.Rapeli]

On Mon, Sep 24, 2018 at 09:48:26AM -0400, Bruce Ashfield wrote:
> On Mon, Sep 24, 2018 at 9:46 AM, Bruce Ashfield
>  wrote:
> > On Mon, Sep 24, 2018 at 9:44 AM,   
> > wrote:
> >> On Mon, 2018-09-24 at 13:42 +, mikko.rap...@bmw.de wrote:
> >>> On Mon, Sep 24, 2018 at 02:38:36PM +0100, richard.purdie@linuxfoundat
> >>> ion.org wrote:
> >>> > I was replying from the perspective of how this should work in
> >>> > general.
> >>> > I agree that for this to work with a kernel recipe we do need the
> >>> > change that started this thread and that is probably a reasonable
> >>> > thing
> >>> > to do.
> >>>
> >>> Good, then my patch is not going in the wrong direction.
> >>>
> >>> Remaining problem is if file overwrites will be detected or not so
> >>> that
> >>> accidental overrides of files in linux-libc-headers from kernel
> >>> recipes
> >>> should not be possible...
> >>
> >> That is a general problem and the core sstate code shouldn't let that
> >> happen these days so I think that is already covered?
> >
> > Yep, it should be covered. I was just looking for confirmation.
> >
> > Can everyone have a look at:
> >
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=5305
> >
> > And confirm that it can be closed by this .. ? I suggested my approach
> > in that bug, but it
> > never went anywhere at the time.
> 
> Sorry for all the email, I hit send a bit too soon.
> 
> The only issue that would remain for that bug is that there's no
> standard/default location for those exported extra headers, and it
> is up to the kernel recipe packager to make sure that their recipes
> know where to look for them.
> 
> I'm not convinced there has to be a standard location for them, so
> I'll all for closing that old bug.

It would be nice for kernel.bbclass to help creating the custom kernel specific
-dev package with headers, and it would be nice to have a kernel-headers.bbclass
to help users to find them by adding the path to default include paths for
the recipe. And while writing wishlists, would be nice to support multiple
kernel recipes per machine. We forced a second recipe by cloning the
kernel and other bbclasses and added a different prefix...

But I'm ok with my patch and doing other things per recipe for now.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] kernel.bbclass: allow exporting files from kernel recipes to sysroot

[Mikko.Rapeli]

On Mon, Sep 24, 2018 at 02:38:36PM +0100, richard.pur...@linuxfoundation.org 
wrote:
> On Mon, 2018-09-24 at 13:20 +, mikko.rap...@bmw.de wrote:
> > On Mon, Sep 24, 2018 at 02:11:13PM +0100, Richard Purdie wrote:
> > > On Mon, 2018-09-24 at 12:19 +, mikko.rap...@bmw.de wrote:
> > > > > That was one old way, but not the only. And not for exposing
> > > > > non
> > > > > uapi
> > > > > headers.
> > > > 
> > > > What other ways exist?
> > > > 
> > > > I don't care how, but I must export custom kernel specific
> > > > headers
> > > > and
> > > > other files to other recipes in a build in ways which are
> > > > compatible
> > > > with
> > > > yocto upstream.
> > > > 
> > > > I have not seen any documented ways for this.
> > > 
> > > It may not be documented, perhaps because its actually very simple.
> > > 
> > > Any recipe can expose headers into the recipe sysroot, they simply
> > > install them where needed in do_install as normal.
> > > 
> > > So all you need is a recipe which installs the right headers and
> > > then
> > > you DEPEND on that recipe. Where that recipe gets the headers isn't
> > > relevant.
> > 
> > No, this does not work on sumo. My patch is needed for this to work.
> > 
> > Without my patch, users of kernel.bbclass have zero files in
> > tmp/sysroot-components even if they install extra files and extra
> > header only binary packages.
> > 
> > A generated image or SDK will have the files if the binary package is
> > installed but sysroot not.
> 
> I was replying from the perspective of how this should work in general.
> I agree that for this to work with a kernel recipe we do need the
> change that started this thread and that is probably a reasonable thing
> to do.

Good, then my patch is not going in the wrong direction.

Remaining problem is if file overwrites will be detected or not so that
accidental overrides of files in linux-libc-headers from kernel recipes
should not be possible...

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] kernel.bbclass: allow exporting files from kernel recipes to sysroot

[Mikko.Rapeli]

On Mon, Sep 24, 2018 at 02:11:13PM +0100, Richard Purdie wrote:
> On Mon, 2018-09-24 at 12:19 +, mikko.rap...@bmw.de wrote:
> > > That was one old way, but not the only. And not for exposing non
> > > uapi
> > > headers.
> > 
> > What other ways exist?
> > 
> > I don't care how, but I must export custom kernel specific headers
> > and
> > other files to other recipes in a build in ways which are compatible
> > with
> > yocto upstream.
> > 
> > I have not seen any documented ways for this.
> 
> It may not be documented, perhaps because its actually very simple.
> 
> Any recipe can expose headers into the recipe sysroot, they simply
> install them where needed in do_install as normal.
> 
> So all you need is a recipe which installs the right headers and then
> you DEPEND on that recipe. Where that recipe gets the headers isn't
> relevant.

No, this does not work on sumo. My patch is needed for this to work.

Without my patch, users of kernel.bbclass have zero files in
tmp/sysroot-components even if they install extra files and extra header
only binary packages.

A generated image or SDK will have the files if the binary package is installed
but sysroot not.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] kernel.bbclass: allow exporting files from kernel recipes to sysroot

[Mikko.Rapeli]

On Mon, Sep 24, 2018 at 08:12:53AM -0400, Bruce Ashfield wrote:
> On Mon, Sep 24, 2018 at 3:25 AM,   wrote:
> >> On Fri, Sep 21, 2018 at 7:49 AM, Mikko Rapeli  
> >> wrote:
> >> > This change enables kernel recipes to share files with other
> >> > recipes. Firmware, modules and kernel-depmod are still not shared
> >> > since according to git history they cause problems with multiarch,
> >> > but all others are allowed. Examples of shared files are
> >> > kernel version and recipe specific headers and scripts which
> >> > are not needed by common linux-libc-headers to bootstrap glibc.
> >> >
> >> > For example, if a kernel recipe wants to share headers, it can do:
> >> >
> >> > PACKAGES =+ "${PN}-headers"
> >> >
> >> > do_install_append() {
> >> > install -d "${D}${includedir}/${PN}"
> >> > oe_runmake INSTALL_HDR_PATH="${D}${includedir}/${PN}" headers_install
> >> }
> >>
> >> This is what I've always done in the past (and in fact, there's an
> >> open Yocto bug
> >> to track this), but I haven't actually needed to do what you are
> >> modifying in the
> >> bbclass itself.
> >
> > Without this patch, users of kernel.bbclass can provide the headers etc 
> > packages,
> > but other recipes can never see the files in their sysroot even if they 
> > depend
> > on the kernel recipe because kernel.bbclass has disabled sysroot_stage_all()
> > completely.
> 
> It doesn't need to be in a class, any kernel recipe can create a task to
> do this. The class implementation isn't special in this manner.
> 
> >
> > The recipes could define their own sysroot_stage_all() but I don't see
> > why kernel would be that special and why all of their installed files should
> > be excluded from build sysroots by default.
> 
> Exactly. It is because of the ability to clobber the libc-headers that is is
> special.
> 
> >
> > The old way to do this was to fork linux-libc-headers for the custom kernel
> > but now it has a big fat warning to not do this, but exposing header etc
> > files to other recipes to build against was still not resolved.
> 
> That was one old way, but not the only. And not for exposing non uapi
> headers.

What other ways exist?

I don't care how, but I must export custom kernel specific headers and
other files to other recipes in a build in ways which are compatible with
yocto upstream.

I have not seen any documented ways for this.

> >
> > With this patch the kernel recipe can just install the files needed and
> > users can see them unless they are filtered.
> 
> And as the warning points out, risk the libc-interface, which has caused
> many problems in the past .. and problems that are hard to detect and
> dig out at runtime.
> 
> >
> >> If you call the sysroot stage routines directly in that
> >> install_append, are you really
> >> not seeing the files appear in the recipe's sysroot ?
> >
> > Of course this can be done but why on earth is kernel so special that
> > it's installed files are not visible in sysroots by default?
> 
> because they clobber the uapi libc-headers :D
> 
> >
> > If certain files are problematic, they can be filtered out from the
> > sysroot like my patch tries to do.
> 
> Or we could catch that the clobbering is happening, and not let it happen.
> 
> >
> >> Have you confirmed that we get a warning/error from bitbake about
> >> conflicting files
> >> from multiple recipes if someone doesn't know to use a custom path for 
> >> their
> >> headers ? That has always been the main concern about allowing something 
> >> like
> >> this.
> >
> > No, but I do rely on this with other recipes as well. At least I know that
> > conflicting files warning will fail image and SDK generation.
> 
> For something that is being proposed as a general purpose addition to
> the core, it would probably be a good idea to test these extra cases.
> 
> >
> > I would expect that recipe specific sysroots don't allow conflicting files.
> 
> Agreed, but definitely worth testing and logging. That way the change
> won't be blamed for causing issues later.

If files from a recipe can replace files from another recipe when preparing
recipe specific sysroot we have bigger problems than just the kernel.

I can try test this out with the kernel recipe though.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] kernel.bbclass: allow exporting files from kernel recipes to sysroot

[Mikko.Rapeli]

> On Fri, Sep 21, 2018 at 7:49 AM, Mikko Rapeli  wrote:
> > This change enables kernel recipes to share files with other
> > recipes. Firmware, modules and kernel-depmod are still not shared
> > since according to git history they cause problems with multiarch,
> > but all others are allowed. Examples of shared files are
> > kernel version and recipe specific headers and scripts which
> > are not needed by common linux-libc-headers to bootstrap glibc.
> >
> > For example, if a kernel recipe wants to share headers, it can do:
> >
> > PACKAGES =+ "${PN}-headers"
> >
> > do_install_append() {
> > install -d "${D}${includedir}/${PN}"
> > oe_runmake INSTALL_HDR_PATH="${D}${includedir}/${PN}" headers_install
> }
> 
> This is what I've always done in the past (and in fact, there's an
> open Yocto bug
> to track this), but I haven't actually needed to do what you are
> modifying in the
> bbclass itself.

Without this patch, users of kernel.bbclass can provide the headers etc 
packages,
but other recipes can never see the files in their sysroot even if they depend
on the kernel recipe because kernel.bbclass has disabled sysroot_stage_all()
completely.

The recipes could define their own sysroot_stage_all() but I don't see
why kernel would be that special and why all of their installed files should
be excluded from build sysroots by default.

The old way to do this was to fork linux-libc-headers for the custom kernel
but now it has a big fat warning to not do this, but exposing header etc
files to other recipes to build against was still not resolved.

With this patch the kernel recipe can just install the files needed and
users can see them unless they are filtered.

> If you call the sysroot stage routines directly in that
> install_append, are you really
> not seeing the files appear in the recipe's sysroot ?

Of course this can be done but why on earth is kernel so special that
it's installed files are not visible in sysroots by default?

If certain files are problematic, they can be filtered out from the
sysroot like my patch tries to do.

> Have you confirmed that we get a warning/error from bitbake about
> conflicting files
> from multiple recipes if someone doesn't know to use a custom path for their
> headers ? That has always been the main concern about allowing something like
> this.

No, but I do rely on this with other recipes as well. At least I know that
conflicting files warning will fail image and SDK generation.

I would expect that recipe specific sysroots don't allow conflicting files.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] openssl10: remove extra slash from libdir path

[Mikko.Rapeli]

On Fri, Sep 21, 2018 at 04:41:50PM +, Peter Kjellerstedt wrote:
> > diff --git a/meta/recipes-connectivity/openssl/openssl10_1.0.2p.bb
> > b/meta/recipes-connectivity/openssl/openssl10_1.0.2p.bb
> > index b7297fc..f09782c 100644
> > --- a/meta/recipes-connectivity/openssl/openssl10_1.0.2p.bb
> > +++ b/meta/recipes-connectivity/openssl/openssl10_1.0.2p.bb
> > @@ -191,7 +191,7 @@ do_configure () {
> > if [ "x$useprefix" = "x" ]; then
> > useprefix=/
> > fi
> > -   libdirleaf="$(echo ${libdir} | sed s:$useprefix::)"
> > +   libdirleaf="$( basename "${libdir}" )"
> 
> You are making assumptions about the value of ${libdir} here.
> May I suggest the following instead (just in case someone has 
> defined libdir as ${prefix}/foo/lib):
> 
>   libdirleaf="$(echo ${libdir} | sed s:^$useprefix/*::)"

Ok, sending v2 with this.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] cmake: add CMAKE_SYSROOT to generated toolchain file

[Mikko.Rapeli]

On Fri, Aug 24, 2018 at 09:23:45AM -0700, Khem Raj wrote:
> On Fri, Aug 24, 2018 at 9:17 AM  wrote:
> >
> > On Fri, Aug 24, 2018 at 08:51:42AM -0700, Khem Raj wrote:
> > > On Fri, Aug 24, 2018 at 8:48 AM  wrote:
> > > > So to me it looks like using SYSTEM with target_include_directories()
> > > > is no longer possible with CMake and gcc > 6:
> > > >
> > > >
> > > > https://cmake.org/cmake/help/v3.10/command/target_include_directories.html?highlight=target_include_directories
> > > >
> > >
> > > That’s right there is a patch I did  in WebKitgtk recipe in Oe Core you
> > > might want to include similar fixes for this package the patch actually it
> > > a workaround but then I also think using isystem in apps is a bad idea
> >
> > Yes, I tend to agree, but developers have used this to silence compiler
> > warnings from system header files.
> >
> > In a few cases I saw that there were no longer any warnings coming from the
> > headers but in some others there were.
> >
> 
> Unlike olden days, cross gcc compiler now a days knows about its
> system headers and adds them implicitly with right options
> so doing this include dance manually is not useful. Let compiler
> decide to do the needed
> for system headers, one must just specify --sysroot and let it handle
> everything from sysroot.
> 
> > Any other ideas how to suppress compiler wanings from system headers?
> >
> 
> what kind of warnings do you see ?

After a ton of components have exported their headers to system paths,
I see all kinds of crap. Some developers have cleaned up their own tree
and added strict compiler flags, which I now have to relax since
header file warning suppressions don't work, and new compiler produces
new bunch of warnings. Here are few sample warnings from an ongoing build
with counts redacted:

$ egrep -ho "warning: .*" tmp-glibc/work/*/*/*/temp/log.do_compile| sort | uniq 
-c|sort -rn | head -8

warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
warning: 'template class std::auto_ptr' is deprecated 
[-Wdeprecated-declarations]
warning: conversion to 'int' from 'long unsigned int' may alter its value 
[-Wconversion]
warning: missed loop optimization, the loop counter may overflow 
[-Wunsafe-loop-optimizations]
warning: command line option '-std=c++11' is valid for C++/ObjC++ but not for C
warning: this statement may fall through [-Wimplicit-fallthrough=]
warning: ISO C++ forbids zero-size array 'f_handle' [-Wpedantic]

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] cmake: add CMAKE_SYSROOT to generated toolchain file

[Mikko.Rapeli]

On Fri, Aug 24, 2018 at 08:51:42AM -0700, Khem Raj wrote:
> On Fri, Aug 24, 2018 at 8:48 AM  wrote:
> > So to me it looks like using SYSTEM with target_include_directories()
> > is no longer possible with CMake and gcc > 6:
> >
> >
> > https://cmake.org/cmake/help/v3.10/command/target_include_directories.html?highlight=target_include_directories
> >
> 
> That’s right there is a patch I did  in WebKitgtk recipe in Oe Core you
> might want to include similar fixes for this package the patch actually it
> a workaround but then I also think using isystem in apps is a bad idea

Yes, I tend to agree, but developers have used this to silence compiler
warnings from system header files.

In a few cases I saw that there were no longer any warnings coming from the
headers but in some others there were.

Any other ideas how to suppress compiler wanings from system headers?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] cmake: add CMAKE_SYSROOT to generated toolchain file

[Mikko.Rapeli]

On Fri, Aug 24, 2018 at 03:10:28PM +, Bach, Pascal wrote:
>  
> > > Fixes problems which we have also seen on sumo branch. Thanks for this!
> > 
> > Sorry, spoke too soon. This fixes compilation on a few recipes in my tree 
> > but
> > there are still lots failures with the same error message. We had a hacky
> > patch to cmake as a workaround but that too causes some problems:
> 
> I have not yet figured out what CMake based projects are affected and why.
> The error seems to happen only for certain projects.

In our case it looks like a library is adding target_include_directories()
with SYSTEM keyword in its cmake module. Everyone who includes the cmake module
fails to build. This repeats with a handful of cmake modules and causes all
of their dependencies to fail build with errors like:

.../recipe-sysroot/usr/include/c++/7.3.0/cstdlib:75:15: fatal error: stdlib.h: 
No such file or directory
|  #include_next 
|^~
| compilation terminated.

or

| cc1plus: warning: include location "/usr/include" is unsafe for 
cross-compilation [-Wpoison-system-directories]
...
.../recipe-sysroot/usr/include/stdlib.h:133:35: error: missing binary operator 
before token 
"("
|  #if __HAVE_FLOAT16 && __GLIBC_USE (IEC_60559_TYPES_EXT)
|^

So to me it looks like using SYSTEM with target_include_directories()
is no longer possible with CMake and gcc > 6:

https://cmake.org/cmake/help/v3.10/command/target_include_directories.html?highlight=target_include_directories

> > --- a/Modules/Compiler/GNU.cmake
> > +++ b/Modules/Compiler/GNU.cmake
> > @@ -45,7 +45,7 @@
> >set(CMAKE_${lang}_CREATE_PREPROCESSED_SOURCE
> > "-E
> >  > ")
> >set(CMAKE_${lang}_CREATE_ASSEMBLY_SOURCE
> > "-S
> >  -o ")
> >if(NOT APPLE OR NOT CMAKE_${lang}_COMPILER_VERSION VERSION_LESS
> > 4) # work around #4462
> > -set(CMAKE_INCLUDE_SYSTEM_FLAG_${lang} "-isystem ")
> > +set(CMAKE_INCLUDE_SYSTEM_FLAG_${lang} "-I ")
> >endif()
> > 
> >set(_CMAKE_${lang}_IPO_SUPPORTED_BY_CMAKE YES)
> > 
> > I wonder what the real fix could be...
> > 
> 
> This looks similar to another workaround I saw.
> Setting CMAKE_NO_SYSTEM_FROM_IMPORTED to True. I'm also not sure what the 
> real fix is.
> The issue is also discussed here: 
> https://gitlab.kitware.com/cmake/cmake/issues/16291

Thanks for the ticket link.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] cmake: add CMAKE_SYSROOT to generated toolchain file

[Mikko.Rapeli]

> Fixes problems which we have also seen on sumo branch. Thanks for this!

Sorry, spoke too soon. This fixes compilation on a few recipes in my tree
but there are still lots failures with the same error message. We had a hacky
patch to cmake as a workaround but that too causes some problems:

--- a/Modules/Compiler/GNU.cmake
+++ b/Modules/Compiler/GNU.cmake
@@ -45,7 +45,7 @@
   set(CMAKE_${lang}_CREATE_PREPROCESSED_SOURCE " 
   -E  > ")
   set(CMAKE_${lang}_CREATE_ASSEMBLY_SOURCE "  
  -S  -o ")
   if(NOT APPLE OR NOT CMAKE_${lang}_COMPILER_VERSION VERSION_LESS 4) # work 
around #4462
-set(CMAKE_INCLUDE_SYSTEM_FLAG_${lang} "-isystem ")
+set(CMAKE_INCLUDE_SYSTEM_FLAG_${lang} "-I ")
   endif()
 
   set(_CMAKE_${lang}_IPO_SUPPORTED_BY_CMAKE YES)

I wonder what the real fix could be...

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] cmake: add CMAKE_SYSROOT to generated toolchain file

[Mikko.Rapeli]

On Fri, Aug 24, 2018 at 02:33:46PM +0200, Pascal Bach wrote:
> This already got fixed in the toolchain file that is used during development
> in 
> https://github.com/openembedded/openembedded-core/commit/cb42802f2fe1760f894a435b07286bca3a220364
> 
> The toolchain file generated by the cmake.bbclass however does not set
> CMAKE_SYSROOT. Under certain circumstances this also leads to the error:
> `"stdlib.h: No such file or directory #include_next "`
> during the build of a recipe.
> 
> An example where this accured was during the upgrade of the Apache Thrift
> recipe in meta-openembedded to 0.11.0. With this change the build works out of
> the box.
> 
> CMAKE_SYSROOT must only be set when crosscompiling, otherwise it will 
> interfere
> with the native compiler headers.
> 
> Signed-off-by: Pascal Bach 
> ---
>  meta/classes/cmake.bbclass | 5 +
>  1 file changed, 5 insertions(+)
> 
> diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
> index fd40a9863e..251ddd9afe 100644
> --- a/meta/classes/cmake.bbclass
> +++ b/meta/classes/cmake.bbclass
> @@ -64,9 +64,12 @@ def map_target_arch_to_uname_arch(target_arch):
>  return "ppc64"
>  return target_arch
>  
> +
>  cmake_do_generate_toolchain_file() {
>   if [ "${BUILD_SYS}" = "${HOST_SYS}" ]; then
>   cmake_crosscompiling="set( CMAKE_CROSSCOMPILING FALSE )"
> + else
> + cmake_sysroot="set( CMAKE_SYSROOT \"${RECIPE_SYSROOT}\" )"
>   fi
>   cat > ${WORKDIR}/toolchain.cmake <  # CMake system name must be something like "Linux".
> @@ -95,6 +98,8 @@ set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM 
> ${OECMAKE_FIND_ROOT_PATH_MODE_PROGRAM} )
>  set( CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY )
>  set( CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY )
>  
> +$cmake_sysroot
> +
>  # Use qt.conf settings
>  set( ENV{QT_CONF_PATH} ${WORKDIR}/qt.conf )

Fixes problems which we have also seen on sumo branch. Thanks for this!

Tested-by: Mikko Rapeli 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] perf: fail if src path does not exist

[Mikko.Rapeli]

Martin Jansa wrote:
> I think this will break support for older kernels added in:
> http://git.openembedded.org/openembedded-core/commit/?id=19fb2d11a8bb3c6dfdd5edc1b9155d642dc0f5e0
> 
> kernel-source/tools/arch and kernel-source/tools/build are missing in 3.16
> kernel even when it's not completely broken.
> 
> I'm not completely against this change, maybe just add to commit message
> that for older kernels you need to add .bbappend which removes the
> directories which really doesn't exist in your kernel version from PERF_SRC.

Exactly. For older kernels like 4.1 the recipe did not work at all and the
failure was a total one at compile time. Once error was clear, solution
was to create a perf.bbappend for the kernel which overwrites
the PERF_SRC variable with needed directories from kernel source tree.

I can document this in the commit message and send a v2.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] cve-check.bbclass: do not download the CVE DB in package-specific tasks

[Mikko.Rapeli]

On Mon, Aug 13, 2018 at 10:23:28AM +0300, Konstantin Shemyak wrote:
> Disable downloading of the vulnerability DB in do_check_cves() task.
> 
> When invoked in this task, cve-check-tool attempts re-download of the CVE DB
> if the latter is older than certain threshold. While reasonable for a
> stand-alone CVE checker, this behavior can cause errors in parallel builds
> if the build time is longer than this threshold: 
> * Other tasks might be using the DB.
> * Several packages can start the download of the same file at the same time.
> 
> This check is not really needed, as the DB has been downloaded by
> cve_check_tool:do_populate_cve_db() which is a prerequisite of any do_build().
> The DB will be at most (threshold + build_time) old.
> 
> Signed-off-by: Konstantin Shemyak 
> ---
>  meta/classes/cve-check.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
> index 4d99838..12ad3e5 100644
> --- a/meta/classes/cve-check.bbclass
> +++ b/meta/classes/cve-check.bbclass
> @@ -179,7 +179,7 @@ def check_cves(d, patched_cves):
>  cve_db_dir = d.getVar("CVE_CHECK_DB_DIR")
>  cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST"))
>  cve_cmd = "cve-check-tool"
> -cmd = [cve_cmd, "--no-html", "--csv", "--not-affected", "-t", "faux", 
> "-d", cve_db_dir]
> +cmd = [cve_cmd, "--no-html", "--skip-update", "--csv", "--not-affected", 
> "-t", "faux", "-d", cve_db_dir]
>  
>  # If the recipe has been whitlisted we return empty lists
>  if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split():
> -- 

ACK.

Reviewed-by: Mikko Rapeli 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/2] cve-report: add scripts to generate CVE reports

[Mikko.Rapeli]

On Fri, Aug 03, 2018 at 10:37:05PM +, Grygorii Tertychnyi (gtertych) via 
Openembedded-core wrote:
> cvert-kernel - generate CVE report for the Linux kernel.
>   NVD entries for the Linux kernel is almost always outdated.
>   For example, https://nvd.nist.gov/vuln/detail/CVE-2018-1065
>   is shown as matched for "versions up to (including) 4.15.7",
>   however the patch 57ebd808a97d has been back ported for 4.14.
>   cvert-kernel script checks NVD Resource entries for the patch URLs
>   and looking for the commits in the local git tree.

This is an interesting approach.

For the kernel I've been using information not from NVD but from
https://github.com/nluedtke/linux_kernel_cves/

As an example, all CVE fixed in 4.14 kernel series point releases AND all
non-fixed CVE are listed in:

https://github.com/nluedtke/linux_kernel_cves/blob/master/4.14/4.14_security.txt

I have not tried to automate this, but I do find the information there
much better than NVD.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [oe] Use of github /archive/ tarballs in SRC_URI

[Mikko.Rapeli]

Yes, we have been affected by this several times too.

In addition to fixing recipes, a sanity test to warn about the situation
would be nice to have so that the issue could be detected in other
meta layers too.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] ✗ patchtest: failure for "gitsm.py: use download cache f..." and 2 more

[Mikko.Rapeli]

On Wed, Sep 06, 2017 at 12:14:48PM +, mikko.rap...@bmw.de wrote:
> On Wed, Sep 06, 2017 at 11:34:54AM +, Patchwork wrote:
> > == Series Details ==
> > 
> > Series: "gitsm.py: use download cache f..." and 2 more
> > Revision: 1
> > URL   : https://patchwork.openembedded.org/series/8734/
> > State : failure
> > 
> > == Summary ==
> > 
> > 
> > Thank you for submitting this patch series to OpenEmbedded Core. This is
> > an automated response. Several tests have been executed on the proposed
> > series by patchtest resulting in the following failures:
> > 
> > 
> > 
> > * Issue Series does not apply on top of target branch 
> > [test_series_merge_on_head] 
> >   Suggested fixRebase your series on top of targeted branch
> >   Targeted branch  master (currently at cc319b6dcc)
> 
> Uh, the patch is based on poky master at:
> 
> commit 8b4f16a9cbbaf521461f699b7264fac2ac872581 
> (refs/remotes/upstream/master-ne
> Author: Jussi Kukkonen 
> Date:   Mon Sep 4 11:39:24 2017 +0300
> 
> mesa-gl: Fix build after recent mesa PACKAGECONFIG changes
> 
> 48d39cf43b added "opengl" PACKAGECONFIG option to mesa: before that
> the configuration was always enabled. "opengl" should have been added
> to mesa-gl default PACKAGECONFIG but wasn't: do it now.
> 
> (From OE-Core rev: cc319b6dcc5b4a5019fb91c9771b12ce17f3c953)
> 
> Signed-off-by: Jussi Kukkonen 
> Signed-off-by: Richard Purdie 
> 
> and there are no newer changes available. Should I port the patch to oe-core
> directly and submit again?
> 
> Maybe this worked in the past because someone else did the applying and
> merging manually?

Ah, sorry. The patch is for bitbake, not oe-core. Will send to correct list.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] ✗ patchtest: failure for "gitsm.py: use download cache f..." and 2 more

[Mikko.Rapeli]

On Wed, Sep 06, 2017 at 11:34:54AM +, Patchwork wrote:
> == Series Details ==
> 
> Series: "gitsm.py: use download cache f..." and 2 more
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/8734/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Issue Series does not apply on top of target branch 
> [test_series_merge_on_head] 
>   Suggested fixRebase your series on top of targeted branch
>   Targeted branch  master (currently at cc319b6dcc)

Uh, the patch is based on poky master at:

commit 8b4f16a9cbbaf521461f699b7264fac2ac872581 (refs/remotes/upstream/master-ne
Author: Jussi Kukkonen 
Date:   Mon Sep 4 11:39:24 2017 +0300

mesa-gl: Fix build after recent mesa PACKAGECONFIG changes

48d39cf43b added "opengl" PACKAGECONFIG option to mesa: before that
the configuration was always enabled. "opengl" should have been added
to mesa-gl default PACKAGECONFIG but wasn't: do it now.

(From OE-Core rev: cc319b6dcc5b4a5019fb91c9771b12ce17f3c953)

Signed-off-by: Jussi Kukkonen 
Signed-off-by: Richard Purdie 

and there are no newer changes available. Should I port the patch to oe-core
directly and submit again?

Maybe this worked in the past because someone else did the applying and
merging manually?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 01/23] buildhistory.bbclass: add LICENSE and CVE_PRODUCT to recipe and package data

[Mikko.Rapeli]

On Thu, Jul 20, 2017 at 04:22:49PM +0300, Mikko Rapeli wrote:
> LICENSE can be used in various checks after builds. Reading license data
> from buildhistory is better than trying to parse recipes in a source tree.
> 
> CVE_PRODUCT can be used by scripts to e.g. check if it matches to the
> CVE product names in CVE/NVD database.
> 
> It the two are combined, a CVE product name check can for example ignore
> recipes with CLOSED license.
> 
> Note about sstate caching: recipe and package buildhistory data is
> regenarated only when the recipe is rebuild from sources. New fields
> like LICENSE and CVE_PRODUCT in buildhistory will be deployed only after
> the recipes are recompiled.
> 
> Example:
> 
> $ bitbake -c cleanall busybox && bitbake busybox
> $ egrep "LICENSE|CVE_PRODUCT" 
> buildhistory/packages/i586-poky-linux/busybox/busybox/latest
> LICENSE = GPLv2 & bzip2
> CVE_PRODUCT = busybox

Are there some problems with this approach? Is CVE_PRODUCT too much?

I find this much better than trying to parse recipes for the data, and I like
that history data of these variables across builds/releases is available.

-Mikko

> Signed-off-by: Mikko Rapeli 
> ---
>  meta/classes/buildhistory.bbclass | 21 +
>  1 file changed, 21 insertions(+)
> 
> diff --git a/meta/classes/buildhistory.bbclass 
> b/meta/classes/buildhistory.bbclass
> index 81784ee..cc3b144 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -92,6 +92,8 @@ python buildhistory_emit_pkghistory() {
>  self.packages = ""
>  self.srcrev = ""
>  self.layer = ""
> +self.license = ""
> +self.cve_product = ""
>  
>  
>  class PackageInfo:
> @@ -105,6 +107,8 @@ python buildhistory_emit_pkghistory() {
>  self.pkge = ""
>  self.pkgv = ""
>  self.pkgr = ""
> +self.license = ""
> +self.cve_product = ""
>  self.size = 0
>  self.depends = ""
>  self.rprovides = ""
> @@ -141,6 +145,10 @@ python buildhistory_emit_pkghistory() {
>  pkginfo.pkgv = value
>  elif name == "PKGR":
>  pkginfo.pkgr = value
> +elif name == "LICENSE":
> +pkginfo.license = value
> +elif name == "CVE_PRODUCT":
> +pkginfo.cve_product = value
>  elif name == "RPROVIDES":
>  pkginfo.rprovides = value
>  elif name == "RDEPENDS":
> @@ -193,6 +201,9 @@ python buildhistory_emit_pkghistory() {
>  pv = d.getVar('PV')
>  pr = d.getVar('PR')
>  layer = bb.utils.get_file_layer(d.getVar('FILE', True), d)
> +license = d.getVar('LICENSE') or ''
> +# If recipe does not define CVE_PRODUCT, the default is pn
> +cve_product = d.getVar('CVE_PRODUCT') or pn
>  
>  pkgdata_dir = d.getVar('PKGDATA_DIR')
>  packages = ""
> @@ -233,6 +244,8 @@ python buildhistory_emit_pkghistory() {
>  rcpinfo.depends = sortlist(oe.utils.squashspaces(d.getVar('DEPENDS') or 
> ""))
>  rcpinfo.packages = packages
>  rcpinfo.layer = layer
> +rcpinfo.license = license
> +rcpinfo.cve_product = cve_product
>  write_recipehistory(rcpinfo, d)
>  
>  pkgdest = d.getVar('PKGDEST')
> @@ -249,6 +262,8 @@ python buildhistory_emit_pkghistory() {
>  pkge = pkgdata.get('PKGE', '0')
>  pkgv = pkgdata['PKGV']
>  pkgr = pkgdata['PKGR']
> +pkg_license = d.getVar('LICENSE_%s' % (pkg,), True) or license
> +pkg_cve_product = d.getVar('CVE_PRODUCT_%s' % (pkg,), True) or 
> cve_product
>  #
>  # Find out what the last version was
>  # Make sure the version did not decrease
> @@ -272,6 +287,8 @@ python buildhistory_emit_pkghistory() {
>  pkginfo.pkge = pkge
>  pkginfo.pkgv = pkgv
>  pkginfo.pkgr = pkgr
> +pkginfo.license = pkg_license
> +pkginfo.cve_product = pkg_cve_product
>  pkginfo.rprovides = 
> sortpkglist(oe.utils.squashspaces(pkgdata.get('RPROVIDES', "")))
>  pkginfo.rdepends = 
> sortpkglist(oe.utils.squashspaces(pkgdata.get('RDEPENDS', "")))
>  pkginfo.rrecommends = 
> sortpkglist(oe.utils.squashspaces(pkgdata.get('RRECOMMENDS', "")))
> @@ -347,6 +364,8 @@ def write_recipehistory(rcpinfo, d):
>  f.write(u"DEPENDS = %s\n" %  rcpinfo.depends)
>  f.write(u"PACKAGES = %s\n" %  rcpinfo.packages)
>  f.write(u"LAYER = %s\n" %  rcpinfo.layer)
> +f.write(u"LICENSE = %s\n" % rcpinfo.license)
> +f.write(u"CVE_PRODUCT = %s\n" % rcpinfo.cve_product)
>  
>  write_latest_srcrev(d, pkghistdir)
>  
> @@ -374,6 +393,8 @@ def write_pkghistory(pkginfo, d):
>  f.write(u"PKGV = %s\n" % pkginfo.pkgv)
>  if pkginfo.pkgr != pkginfo.pr:
>  f.write(u"PKGR = %s\n" % pkginfo.pkgr)
> +

Re: [OE-core] [PATCH 02/23] buildhistory.bbclass: add BUILDHISTORY_FORCE_UPDATE option

[Mikko.Rapeli]

On Thu, Jul 20, 2017 at 04:22:50PM +0300, Mikko Rapeli wrote:
> Setting BUILDHISTORY_FORCE_UPDATE = "1" in local.conf forces buildhistory
> updates and recipe rebuilds also when only buildhistory.bbclass changed.
> 
> This is handy when using sstate cache and modifying buildhistory to include
> new data like recipe and package LICENSE fields.
> 
> By default new buildhistory data is updated when recipes are rebuild and
> new fields like LICENSE are not deployed immediately.

Are there some problems with this approach?

I did not find other ways to force updates of buildhistory data without
writing custom functions to update LICENSE and CVE_PRODUCT (which I care about
now) for every build. If there was a way to invalidate the full sstate
cache then that would work too, but I doubt a simple PE update somewhere would
do it.

-Mikko

> Signed-off-by: Mikko Rapeli 
> ---
>  meta/classes/buildhistory.bbclass | 7 ---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/buildhistory.bbclass 
> b/meta/classes/buildhistory.bbclass
> index cc3b144..bc3145e 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -42,10 +42,11 @@ BUILDHISTORY_COMMIT_AUTHOR ?= "buildhistory 
> "
>  BUILDHISTORY_PUSH_REPO ?= ""
>  
>  SSTATEPOSTINSTFUNCS_append = " buildhistory_emit_pkghistory"
> -# We want to avoid influencing the signatures of sstate tasks - first the 
> function itself:
> -sstate_install[vardepsexclude] += "buildhistory_emit_pkghistory"
> +# If BUILDHISTORY_FORCE_UPDATE is not set, we want to avoid influencing the
> +# signatures of sstate tasks - first the function itself:
> +sstate_install[vardepsexclude] += "${@'' if 
> d.getVar('BUILDHISTORY_FORCE_UPDATE', True) == '1' else 
> 'buildhistory_emit_pkghistory'}"
>  # then the value added to SSTATEPOSTINSTFUNCS:
> -SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "| buildhistory_emit_pkghistory"
> +SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "${@'' if 
> d.getVar('BUILDHISTORY_FORCE_UPDATE', True) == '1' else '| 
> buildhistory_emit_pkghistory'"
>  
>  # Similarly for our function that gets the output signatures
>  SSTATEPOSTUNPACKFUNCS_append = " buildhistory_emit_outputsigs"
> -- 
> 1.9.1
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] linux-libc-headers: fix duplicate IFF_LOWER_UP DORMANT ECHO on musl

[Mikko.Rapeli]

On Fri, Jun 23, 2017 at 04:20:41PM -0700, Khem Raj wrote:
> On Fri, Jun 23, 2017 at 7:17 AM,   wrote:
> > Hi,
> >
> > I'm chipping in since I've been messing with these things a bit in upstream
> > Linux kernel.
> >
> > On Fri, Jun 23, 2017 at 06:37:52AM -0700, Khem Raj wrote:
> >> On Fri, Jun 23, 2017 at 3:46 AM, André Draszik  wrote:
> >> > connman is not doing anything wrong here.
> >> >
> >>
> >> yes I am aware of this
> >>
> >> > The kernel is redefining IFF_LOWER_UP, because it thinks the libc doesn't
> >> > define it yet (and glibc doesn't).
> >> >
> >> > libc-compat.h is the way to solve these kind of issues. There also is 
> >> > https:
> >> > //lkml.org/lkml/2017/3/12/238 which is very similar. I'll pick that 
> >> > instead.
> >> >
> >> see the comment https://lkml.org/lkml/2017/3/16/121
> >> that worries me for this patch
> >
> > I'm aware of those review comments but I have not seen any patches posted 
> > which
> > fix the problem in some other way. Thus I would propose to apply these 
> > patches
> > as a workaround until upstream fixes the issues.
> >
> > These header files do not change that often either.
> 
> problem is you become incompatible ABI forever that worries me.
> However if bruce is fine to carry this patch as part of linux-yocto
> I might relent. It still will be hassle where folks will have to apply
> this patch to there kernels if they are building musl based systems.

API or ABI? But agreed this is a problem.

> >
> >> I am not questioning the correctness of patch too. But
> >> it would be better to get this patch accepted into kernel
> >> before applying to OE since these are kind of patches which
> >> you can get stuck with for life if upstream is not accepting it.
> >
> > Upstream-Status: Denied
> >
> > would be a correct marker for now I guess.
> 
> I would rather see some progress made to get it resolved :)
> we need to actually remove glibc'ness completely from kernel.
> and this will fix itself.

Yes. With the glibc'ness in the kernel headers I was just following existing
examples and adding some more. Some of those changes got accepted and only
later came the resistance to remove glibc'ness completely. No-one else
has proposed patches. Maybe I should invent some hobby project time again
for this...

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] linux-libc-headers: fix duplicate IFF_LOWER_UP DORMANT ECHO on musl

[Mikko.Rapeli]

Hi,

I'm chipping in since I've been messing with these things a bit in upstream
Linux kernel.

On Fri, Jun 23, 2017 at 06:37:52AM -0700, Khem Raj wrote:
> On Fri, Jun 23, 2017 at 3:46 AM, André Draszik  wrote:
> > connman is not doing anything wrong here.
> >
> 
> yes I am aware of this
> 
> > The kernel is redefining IFF_LOWER_UP, because it thinks the libc doesn't
> > define it yet (and glibc doesn't).
> >
> > libc-compat.h is the way to solve these kind of issues. There also is https:
> > //lkml.org/lkml/2017/3/12/238 which is very similar. I'll pick that instead.
> >
> see the comment https://lkml.org/lkml/2017/3/16/121
> that worries me for this patch

I'm aware of those review comments but I have not seen any patches posted which
fix the problem in some other way. Thus I would propose to apply these patches
as a workaround until upstream fixes the issues.

These header files do not change that often either.

> I am not questioning the correctness of patch too. But
> it would be better to get this patch accepted into kernel
> before applying to OE since these are kind of patches which
> you can get stuck with for life if upstream is not accepting it.

Upstream-Status: Denied

would be a correct marker for now I guess.

Hope this helps,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [poky] [PATCH 3/3 v2] meta: Fix return value checks from subprocess.call()'s

[Mikko.Rapeli]

On Thu, Jun 22, 2017 at 01:38:15PM +0100, Burton, Ross wrote:
> On 20 June 2017 at 13:48,  wrote:
> 
> > The bitbake and scripts parts from this patch series were merged already,
> > but
> > this not.
> >
> > Are there problems with this patch?
> >
> 
> Sorry, didn't get around to replying.  It doesn't apply to master, can you
> rebase please?

Sure, no problem. I'll rebase and send again.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/3 v2] meta: Fix return value checks from subprocess.call()'s

[Mikko.Rapeli]

Hi,

On Thu, Jun 01, 2017 at 06:53:14PM +0300, Mikko Rapeli wrote:
> Python function subprocess.call() returns the return value of the
> executed process. If return values are not checked, errors may
> go unnoticed and bad things can happen.
> 
> Change all callers of subprocess.call() which do not check for
> the return value to use subprocess.check_call() which raises
> CalledProcessError if the subprocess returns with non-zero value.
> 
> https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module
> 
> All users of the function were found with:
> 
> $ git grep "subprocess\.call" | \
>   egrep -v 'if.*subprocess\.call|=\ 
> +subprocess\.call|return.*subprocess\.call'
> 
> Tested similar patch on top of yocto jethro. Only compile tested
> core-image-minimal on poky master branch.

The bitbake and scripts parts from this patch series were merged already, but
this not.

Are there problems with this patch?

-Mikko

> Signed-off-by: Mikko Rapeli 
> ---
>  meta/classes/archiver.bbclass| 2 +-
>  meta/classes/cml1.bbclass| 2 +-
>  meta/classes/kernel-module-split.bbclass | 2 +-
>  meta/classes/sstate.bbclass  | 4 ++--
>  meta/lib/oeqa/utils/buildproject.py  | 2 +-
>  meta/lib/oeqa/utils/targetbuild.py   | 4 ++--
>  meta/recipes-extended/cups/cups.inc  | 2 +-
>  7 files changed, 9 insertions(+), 9 deletions(-)
> 
> v2:
> split patches to bitbake, meta and scripts
> 
> v1:
> http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136901.html
> 
> diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
> index 2c04557..703eacb 100644
> --- a/meta/classes/archiver.bbclass
> +++ b/meta/classes/archiver.bbclass
> @@ -288,7 +288,7 @@ def create_diff_gz(d, src_orig, src, ar_outdir):
>  os.chdir(dirname)
>  out_file = os.path.join(ar_outdir, '%s-diff.gz' % d.getVar('PF'))
>  diff_cmd = 'diff -Naur %s.orig %s.patched | gzip -c > %s' % (basename, 
> basename, out_file)
> -subprocess.call(diff_cmd, shell=True)
> +subprocess.check_call(diff_cmd, shell=True)
>  bb.utils.remove(src_patched, recurse=True)
>  
>  # Run do_unpack and do_patch
> diff --git a/meta/classes/cml1.bbclass b/meta/classes/cml1.bbclass
> index 38e6613..eb8e790 100644
> --- a/meta/classes/cml1.bbclass
> +++ b/meta/classes/cml1.bbclass
> @@ -63,7 +63,7 @@ python do_diffconfig() {
>  
>  if isdiff:
>  statement = 'diff --unchanged-line-format= --old-line-format= 
> --new-line-format="%L" ' + configorig + ' ' + config + '>' + fragment
> -subprocess.call(statement, shell=True)
> +subprocess.check_call(statement, shell=True)
>  
>  shutil.copy(configorig, config)
>  
> diff --git a/meta/classes/kernel-module-split.bbclass 
> b/meta/classes/kernel-module-split.bbclass
> index 5e10dcf..1035525 100644
> --- a/meta/classes/kernel-module-split.bbclass
> +++ b/meta/classes/kernel-module-split.bbclass
> @@ -47,7 +47,7 @@ python split_kernel_module_packages () {
>  tf = tempfile.mkstemp()
>  tmpfile = tf[1]
>  cmd = "%sobjcopy -j .modinfo -O binary %s %s" % 
> (d.getVar("HOST_PREFIX") or "", file, tmpfile)
> -subprocess.call(cmd, shell=True)
> +subprocess.check_call(cmd, shell=True)
>  f = open(tmpfile)
>  l = f.read().split("\000")
>  f.close()
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 0a12935..f446c3d 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -404,7 +404,7 @@ python sstate_hardcode_path_unpack () {
>  return
>  
>  bb.note("Replacing fixme paths in sstate package: %s" % 
> (sstate_hardcode_cmd))
> -subprocess.call(sstate_hardcode_cmd, shell=True)
> +subprocess.check_call(sstate_hardcode_cmd, shell=True)
>  
>  # Need to remove this or we'd copy it into the target directory and 
> may 
>  # conflict with another writer
> @@ -453,7 +453,7 @@ def sstate_clean_manifest(manifest, d, prefix=None):
>  if os.path.exists(manifest + ".postrm"):
>  import subprocess
>  os.chmod(postrm, 0o755)
> -subprocess.call(postrm, shell=True)
> +subprocess.check_call(postrm, shell=True)
>  oe.path.remove(postrm)
>  
>  oe.path.remove(manifest)
> diff --git a/meta/lib/oeqa/utils/buildproject.py 
> b/meta/lib/oeqa/utils/buildproject.py
> index 487f08b..721f35d 100644
> --- a/meta/lib/oeqa/utils/buildproject.py
> +++ b/meta/lib/oeqa/utils/buildproject.py
> @@ -52,4 +52,4 @@ class BuildProject(metaclass=ABCMeta):
>  
>  def clean(self):
>  self._run('rm -rf %s' % self.targetdir)
> -subprocess.call('rm -f %s' % self.localarchive, shell=True)
> +subprocess.check_call('rm -f %s' % self.localarchive, shell=True)
> diff --git a/meta/lib/oeqa/utils/targetbuild.py 
> b/meta/lib/oeqa/utils/targetbuild.py
> index 9249fa2..1202d57 100644
> --- 

Re: [OE-core] [PATCH 1/3 v2] bitbake: Fix return value checks from subprocess.call()'s

[Mikko.Rapeli]

Hmm, do I need to rebase the patches from yocto to upstream
bitbake and meta-poky git trees?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] Fix return value checks from subprocess.call()'s

[Mikko.Rapeli]

On Thu, Jun 01, 2017 at 01:40:10PM +0100, Burton, Ross wrote:
> On 19 May 2017 at 08:17, Mikko Rapeli  wrote:
> 
> >  bitbake/lib/bb/ui/ncurses.py | 2 +-
> >  bitbake/lib/bb/utils.py  | 2 +-
> >  meta/classes/archiver.bbclass| 2 +-
> >  meta/classes/cml1.bbclass| 2 +-
> >  meta/classes/kernel-module-split.bbclass | 2 +-
> >  meta/classes/sstate.bbclass  | 4 ++--
> >  meta/lib/oeqa/utils/buildproject.py  | 2 +-
> >  meta/lib/oeqa/utils/targetbuild.py   | 4 ++--
> >  meta/recipes-extended/cups/cups.inc  | 2 +-
> >  scripts/runqemu  | 8 
> >
> 
> Poky is a generated repository and you've patches that touch two upstream
> repositories, can you split this into bitbake/ (for the bitbake repo) and
> meta/ scripts/ (for the openembedded-core repository).

Yes, I can do that.

> I presume the transformation wasn't automated and you checked that throwing
> exceptions was the right thing to do?

I did the changes manually and to me throwing an exception in those places
is the right thing to do instead of ignoring all errors.

I will send a v2.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] Fix return value checks from subprocess.call()'s

[Mikko.Rapeli]

Hi,

On Fri, May 19, 2017 at 10:17:17AM +0300, Mikko Rapeli wrote:
> Python function subprocess.call() returns the return value of the
> executed process. If return values are not checked, errors may
> go unnoticed and bad things can happen.
> 
> Change all callers of subprocess.call() which do not check for
> the return value to use subprocess.check_call() which raises
> CalledProcessError if the subprocess returns with non-zero value.
> 
> https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module
> 
> All users of the function were found with:
> 
> $ git grep "subprocess\.call" | \
>   egrep -v 'if.*subprocess\.call|=\ 
> +subprocess\.call|return.*subprocess\.call'
> 
> Tested similar patch on top of jethro. Only compile tested core-image-minimal
> on poky master branch.

Any comments to this patch?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Tue, Mar 01, 2016 at 09:15:37AM -0600, Mariano Lopez wrote:
> 
> 
> On 02/29/2016 08:19 AM, mikko.rap...@bmw.de wrote:
> >On Mon, Feb 29, 2016 at 02:17:26PM +, Burton, Ross wrote:
> >>On 26 February 2016 at 08:14,  wrote:
> >>
> >>>17:45:37  *** 0013:with open(patch_file, "r") as f:
> >>>17:45:37  0014:patch_text = f.read()
> >>>17:45:37  0015:
> >>>17:45:37  0016:# Search for the "CVE: " line
> >>>17:45:37  0017:match = cve_match.search(patch_text)
> >>>17:45:37 Exception: IOError: [Errno 2] No such file or directory:
> >>>'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> >>>17:45:37
> >>>17:45:37 ERROR: Function failed: do_cve_check
> >>>
> >>>So could this be caused by cve-check changes or is this just a side effect
> >>>of some other recipe problems?
> >>>
> >>Do you have rm_work enabled?
> >Yes.
> >
> >-Mikko
> 
> I think I have found the problem, when you do devshell it will execute
> do_unpack and the cve_check task must run after that for some recipes. Try
> this:
> 
> addtask cve_check after do_unpack before do_build

Thanks, with this change the scan builds pass on dizzy.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Mon, Feb 29, 2016 at 02:17:26PM +, Burton, Ross wrote:
> On 26 February 2016 at 08:14,  wrote:
> 
> > 17:45:37  *** 0013:with open(patch_file, "r") as f:
> > 17:45:37  0014:patch_text = f.read()
> > 17:45:37  0015:
> > 17:45:37  0016:# Search for the "CVE: " line
> > 17:45:37  0017:match = cve_match.search(patch_text)
> > 17:45:37 Exception: IOError: [Errno 2] No such file or directory:
> > '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> > 17:45:37
> > 17:45:37 ERROR: Function failed: do_cve_check
> >
> > So could this be caused by cve-check changes or is this just a side effect
> > of some other recipe problems?
> >
> 
> Do you have rm_work enabled?

Yes.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Fri, Feb 26, 2016 at 03:56:24PM +0100, Mikko Rapeli wrote:
> On Fri, Feb 26, 2016 at 08:48:47AM -0600, Mariano Lopez wrote:
> > On 02/26/2016 02:14 AM, mikko.rap...@bmw.de wrote:
> > >Hi,
> > >
> > >On my developer machine the cve-check ran ok for dizzy but on build server
> > >with sstate-cache and rmwork enabled it failed with what looks like a race
> > >condition when scanning the patch files:
> > >
> > >17:45:36 ERROR: Error executing a python function in 
> > >/home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
> > >17:45:36
> > >17:45:36 The stack trace of python calls that resulted in this 
> > >exception/failure was:
> > >17:45:36 File: 'do_cve_check', lineno: 17, function: 
> > >17:45:36  0013:else:
> > >17:45:36  0014:bb.note("Failed to update CVE database, 
> > >skipping CVE check")
> > >17:45:36  0015:
> > >17:45:36  0016:
> > >17:45:36  *** 0017:do_cve_check(d)
> > >17:45:36  0018:
> > >17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
> > >17:45:37  0004:Check recipe for patched and unpatched CVEs
> > >17:45:37  0005:"""
> > >17:45:37  0006:
> > >17:45:37  0007:if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", 
> > >True)):
> > >17:45:37  *** 0008:patched_cves = get_patches_cves(d)
> > >17:45:37  0009:patched, unpatched = check_cves(d, patched_cves)
> > >17:45:37  0010:if patched or unpatched:
> > >17:45:37  0011:cve_data = get_cve_info(d, patched + 
> > >unpatched)
> > >17:45:37  0012:cve_write_data(d, patched, unpatched, 
> > >cve_data)
> > >17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
> > >17:45:37  0009:cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
> > >17:45:37  0010:patched_cves = set()
> > >17:45:37  0011:for url in src_patches(d):
> > >17:45:37  0012:patch_file = bb.fetch.decodeurl(url)[2]
> > >17:45:37  *** 0013:with open(patch_file, "r") as f:
> > >17:45:37  0014:patch_text = f.read()
> > >17:45:37  0015:
> > >17:45:37  0016:# Search for the "CVE: " line
> > >17:45:37  0017:match = cve_match.search(patch_text)
> > >17:45:37 Exception: IOError: [Errno 2] No such file or directory: 
> > >'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> > >17:45:37
> > >17:45:37 ERROR: Function failed: do_cve_check
> > >
> > >So could this be caused by cve-check changes or is this just a side effect
> > >of some other recipe problems?
> > >
> > >I could not see that kind of fixes in master.
> > >
> > >-Mikko
> > 
> > The changes in patch series were minimal and actually this part of the code
> > wasn't touched at all. That part of the code will look for all the files in
> > the SRC_URI variable and will look for the "CVE:" tag in order to find
> > patches that solve CVEs.
> 
> Yep, the code seems straight forward.
> 
> > It seems the problem is with the bitbake fetcher, or the recipe;
> > unfortunately the fetcher is one of the components that most change between
> > releases. Another thing to check is that if actually there is a
> > heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You
> > can check this in the cve_check or patch log in the work directory of the
> > recipe.
> 
> Unfortunately the file is there if I check with devshell but I have now
> four different CI runs with this failure. Only difference to my developer
> machine is sstate cache. Build machines maintain their own sstate cache.

Last two runs were with v2 patches.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Fri, Feb 26, 2016 at 08:48:47AM -0600, Mariano Lopez wrote:
> On 02/26/2016 02:14 AM, mikko.rap...@bmw.de wrote:
> >Hi,
> >
> >On my developer machine the cve-check ran ok for dizzy but on build server
> >with sstate-cache and rmwork enabled it failed with what looks like a race
> >condition when scanning the patch files:
> >
> >17:45:36 ERROR: Error executing a python function in 
> >/home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
> >17:45:36
> >17:45:36 The stack trace of python calls that resulted in this 
> >exception/failure was:
> >17:45:36 File: 'do_cve_check', lineno: 17, function: 
> >17:45:36  0013:else:
> >17:45:36  0014:bb.note("Failed to update CVE database, skipping 
> >CVE check")
> >17:45:36  0015:
> >17:45:36  0016:
> >17:45:36  *** 0017:do_cve_check(d)
> >17:45:36  0018:
> >17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
> >17:45:37  0004:Check recipe for patched and unpatched CVEs
> >17:45:37  0005:"""
> >17:45:37  0006:
> >17:45:37  0007:if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", 
> >True)):
> >17:45:37  *** 0008:patched_cves = get_patches_cves(d)
> >17:45:37  0009:patched, unpatched = check_cves(d, patched_cves)
> >17:45:37  0010:if patched or unpatched:
> >17:45:37  0011:cve_data = get_cve_info(d, patched + 
> >unpatched)
> >17:45:37  0012:cve_write_data(d, patched, unpatched, 
> >cve_data)
> >17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
> >17:45:37  0009:cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
> >17:45:37  0010:patched_cves = set()
> >17:45:37  0011:for url in src_patches(d):
> >17:45:37  0012:patch_file = bb.fetch.decodeurl(url)[2]
> >17:45:37  *** 0013:with open(patch_file, "r") as f:
> >17:45:37  0014:patch_text = f.read()
> >17:45:37  0015:
> >17:45:37  0016:# Search for the "CVE: " line
> >17:45:37  0017:match = cve_match.search(patch_text)
> >17:45:37 Exception: IOError: [Errno 2] No such file or directory: 
> >'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
> >17:45:37
> >17:45:37 ERROR: Function failed: do_cve_check
> >
> >So could this be caused by cve-check changes or is this just a side effect
> >of some other recipe problems?
> >
> >I could not see that kind of fixes in master.
> >
> >-Mikko
> 
> The changes in patch series were minimal and actually this part of the code
> wasn't touched at all. That part of the code will look for all the files in
> the SRC_URI variable and will look for the "CVE:" tag in order to find
> patches that solve CVEs.

Yep, the code seems straight forward.

> It seems the problem is with the bitbake fetcher, or the recipe;
> unfortunately the fetcher is one of the components that most change between
> releases. Another thing to check is that if actually there is a
> heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You
> can check this in the cve_check or patch log in the work directory of the
> recipe.

Unfortunately the file is there if I check with devshell but I have now
four different CI runs with this failure. Only difference to my developer
machine is sstate cache. Build machines maintain their own sstate cache.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

Hi,

On my developer machine the cve-check ran ok for dizzy but on build server
with sstate-cache and rmwork enabled it failed with what looks like a race
condition when scanning the patch files:

17:45:36 ERROR: Error executing a python function in 
/home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
17:45:36 
17:45:36 The stack trace of python calls that resulted in this 
exception/failure was:
17:45:36 File: 'do_cve_check', lineno: 17, function: 
17:45:36  0013:else:
17:45:36  0014:bb.note("Failed to update CVE database, skipping CVE 
check")
17:45:36  0015:
17:45:36  0016:
17:45:36  *** 0017:do_cve_check(d)
17:45:36  0018:
17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
17:45:37  0004:Check recipe for patched and unpatched CVEs
17:45:37  0005:"""
17:45:37  0006:
17:45:37  0007:if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
17:45:37  *** 0008:patched_cves = get_patches_cves(d)
17:45:37  0009:patched, unpatched = check_cves(d, patched_cves)
17:45:37  0010:if patched or unpatched:
17:45:37  0011:cve_data = get_cve_info(d, patched + unpatched)
17:45:37  0012:cve_write_data(d, patched, unpatched, cve_data)
17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
17:45:37  0009:cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
17:45:37  0010:patched_cves = set()
17:45:37  0011:for url in src_patches(d):
17:45:37  0012:patch_file = bb.fetch.decodeurl(url)[2]
17:45:37  *** 0013:with open(patch_file, "r") as f:
17:45:37  0014:patch_text = f.read()
17:45:37  0015:
17:45:37  0016:# Search for the "CVE: " line
17:45:37  0017:match = cve_match.search(patch_text)
17:45:37 Exception: IOError: [Errno 2] No such file or directory: 
'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff'
17:45:37 
17:45:37 ERROR: Function failed: do_cve_check

So could this be caused by cve-check changes or is this just a side effect
of some other recipe problems?

I could not see that kind of fixes in master.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

For openssh there must be some bugs or tunings needed to match the version
numbers used in CVE to ones in yocto. openssh-6.6p1 has zero matches
with the check but I think there are several:

downloads/CVE_CHECK$ grep openssh *xml| grep 6\.6\:p1
nvdcve-2.0-2016.xml:
nvdcve-2.0-2016.xml:  
cpe:/a:openbsd:openssh:6.6:p1
nvdcve-2.0-2016.xml:
nvdcve-2.0-2016.xml:  
cpe:/a:openbsd:openssh:6.6:p1

How should these tunings be made?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Thu, Feb 25, 2016 at 01:29:13PM +0100, Mikko Rapeli wrote:
> On Thu, Feb 25, 2016 at 01:14:21PM +0100, Mikko Rapeli wrote:
> > On Wed, Feb 24, 2016 at 03:27:05PM +, mariano.lo...@linux.intel.com 
> > wrote:
> > > From: Mariano Lopez 
> > > 
> > > This series add the cve-check-tool recipe, a tool used to identify
> > > potentially vulnerable software through version matching. It will
> > > check if a vulnerability has been addressed by a patch.
> > > 
> > > Also add the new cve-check class that will add a task for all recipes
> > > to check for CVEs using cve-check-tool. This tool can be used by recipe,
> > > image (will generate an image report in deploy dir), and with "world"
> > > and "universe"
> > > 
> > > To run it just inherit the class and enter:
> > > 
> > > bitbake -c cve_check 
> > 
> > I tried these on yocto/dizzy but:

Full changes needed in dizzy are:

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 670e592..f24a584 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -893,3 +893,21 @@ def multiprocessingpool(*args, **kwargs):
 
 return multiprocessing.Pool(*args, **kwargs)
 
+# export common proxies variables from datastore to environment
+def export_proxies(d):
+import os
+
+variables = ['http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY',
+'ftp_proxy', 'FTP_PROXY', 'no_proxy', 'NO_PROXY']
+exported = False
+
+for v in variables:
+if v in os.environ.keys():
+exported = True
+else:
+v_proxy = d.getVar(v, True)
+if v_proxy is not None:
+os.environ[v] = v_proxy
+exported = True
+
+return exported
diff --git a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb 
b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
index 9df81cb..b98d991 100644
--- a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
+++ b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
@@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons"
 do_install_append () {
install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor
 }
+
+BBCLASSEXTEND = "native"
diff --git a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb 
b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
index ce00709..26f8f7f 100644
--- a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
+++ b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
@@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = 
"dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097
 inherit gnome gettext lib_package
 
 EXTRA_OECONF = "--disable-introspection"
+
+BBCLASSEXTEND = "native"

And with this I get nice reports with "bitbake -c cve_check openssl" to
tmp/deploy/cve/openssl.

I'll try with a full image build next, but I really, really like this stuff.

Thanks!

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Thu, Feb 25, 2016 at 01:14:21PM +0100, Mikko Rapeli wrote:
> On Wed, Feb 24, 2016 at 03:27:05PM +, mariano.lo...@linux.intel.com wrote:
> > From: Mariano Lopez 
> > 
> > This series add the cve-check-tool recipe, a tool used to identify
> > potentially vulnerable software through version matching. It will
> > check if a vulnerability has been addressed by a patch.
> > 
> > Also add the new cve-check class that will add a task for all recipes
> > to check for CVEs using cve-check-tool. This tool can be used by recipe,
> > image (will generate an image report in deploy dir), and with "world"
> > and "universe"
> > 
> > To run it just inherit the class and enter:
> > 
> > bitbake -c cve_check 
> 
> I tried these on yocto/dizzy but:
> 
> ERROR: Task do_cve_check in 
> /home/builder/src/base/poky/meta/recipes-core/busybox/busybox_1.22.1.bb 
> depends upon non-existent task do_populate_cve_db in 
> virtual:native:/home/builder/src/base/poky/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
> 
> Is there some simple way to make this work there too?
> 
> For testing purposes I tried this only with busybox:
> 
> $ cat busybox_%.bbappend 
> inherit cve-check
> 
> The cve-check-tool itself needed a few native backports/fixes:
> 
> diff --git a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb 
> b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
> index 9df81cb..b98d991 100644
> --- a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
> +++ b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
> @@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons"
>  do_install_append () {
>   install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor
>  }
> +
> +BBCLASSEXTEND = "native"
> diff --git a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb 
> b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
> index ce00709..26f8f7f 100644
> --- a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
> +++ b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
> @@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = 
> "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097
>  inherit gnome gettext lib_package
>  
>  EXTRA_OECONF = "--disable-introspection"
> +
> +BBCLASSEXTEND = "native"

Sorry, I guess this is needed to enable the class properly:

$ grep cve-check conf/local.conf
INHERIT += "cve-check"

but there are some other backports needed in python modules...

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/3] Add initial capability to check CVEs for recipes

[Mikko.Rapeli]

On Wed, Feb 24, 2016 at 03:27:05PM +, mariano.lo...@linux.intel.com wrote:
> From: Mariano Lopez 
> 
> This series add the cve-check-tool recipe, a tool used to identify
> potentially vulnerable software through version matching. It will
> check if a vulnerability has been addressed by a patch.
> 
> Also add the new cve-check class that will add a task for all recipes
> to check for CVEs using cve-check-tool. This tool can be used by recipe,
> image (will generate an image report in deploy dir), and with "world"
> and "universe"
> 
> To run it just inherit the class and enter:
> 
> bitbake -c cve_check 

I tried these on yocto/dizzy but:

ERROR: Task do_cve_check in 
/home/builder/src/base/poky/meta/recipes-core/busybox/busybox_1.22.1.bb depends 
upon non-existent task do_populate_cve_db in 
virtual:native:/home/builder/src/base/poky/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb

Is there some simple way to make this work there too?

For testing purposes I tried this only with busybox:

$ cat busybox_%.bbappend 
inherit cve-check

The cve-check-tool itself needed a few native backports/fixes:

diff --git a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb 
b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
index 9df81cb..b98d991 100644
--- a/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
+++ b/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb
@@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons"
 do_install_append () {
install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor
 }
+
+BBCLASSEXTEND = "native"
diff --git a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb 
b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
index ce00709..26f8f7f 100644
--- a/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
+++ b/meta/recipes-gnome/json-glib/json-glib_1.0.0.bb
@@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = 
"dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097
 inherit gnome gettext lib_package
 
 EXTRA_OECONF = "--disable-introspection"
+
+BBCLASSEXTEND = "native"

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 02/12] hostap-utils: Use C99 stddefs in defining local typedefs

[Mikko.Rapeli]

On Tue, Sep 15, 2015 at 08:13:26AM -0700, Khem Raj wrote:
> On Mon, Sep 14, 2015 at 11:33 PM,   wrote:
> > On Mon, Sep 14, 2015 at 04:31:17PM +, Khem Raj wrote:
> >> The code is creating more abstract types which is nice however it should
> >> be using standard defines from stdint.h and not random defines to base
> >> its own type system
> >
> > These types are not random. They are standard Linux kernel types used by 
> > headers
> > exported to userspace and their definitions come from .
> > These headers should not depend on libc headers like stdint.h.
> 
> Right they are not random in general but they are randomly being
> redefined by the application,
> if it should be using linux/types.h those are different types than
> what is being defined here. I have just
> made the semantics of existing logic to be more c99 compliant.
> 
> >
> > Also, this file is actually a convenience copy of  which 
> > should
> > be used directly instead.
> 
> There must be a reason to make own copy. May be hostap-utils want to
> be portable to more than linux

A private copy of  is only usefull with Linux kernel. Since
mid 2000's Linux kernel has a way to properly export these headers to
userspace. Thus the private copies should not be needed anymore.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 02/12] hostap-utils: Use C99 stddefs in defining local typedefs

[Mikko.Rapeli]

On Mon, Sep 14, 2015 at 04:31:17PM +, Khem Raj wrote:
> The code is creating more abstract types which is nice however it should
> be using standard defines from stdint.h and not random defines to base
> its own type system

These types are not random. They are standard Linux kernel types used by headers
exported to userspace and their definitions come from .
These headers should not depend on libc headers like stdint.h.

Also, this file is actually a convenience copy of  which 
should
be used directly instead.

-Mikko

> Signed-off-by: Khem Raj 
> ---
>  ...-__s32-__u16-__s16-__u8-in-terms-of-c99-t.patch | 36 
> ++
>  meta/recipes-bsp/hostap/hostap-utils.inc   |  4 ++-
>  2 files changed, 39 insertions(+), 1 deletion(-)
>  create mode 100644 
> meta/recipes-bsp/hostap/hostap-utils-0.4.7/0001-Define-_u32-__s32-__u16-__s16-__u8-in-terms-of-c99-t.patch
> 
> diff --git 
> a/meta/recipes-bsp/hostap/hostap-utils-0.4.7/0001-Define-_u32-__s32-__u16-__s16-__u8-in-terms-of-c99-t.patch
>  
> b/meta/recipes-bsp/hostap/hostap-utils-0.4.7/0001-Define-_u32-__s32-__u16-__s16-__u8-in-terms-of-c99-t.patch
> new file mode 100644
> index 000..b44dca3
> --- /dev/null
> +++ 
> b/meta/recipes-bsp/hostap/hostap-utils-0.4.7/0001-Define-_u32-__s32-__u16-__s16-__u8-in-terms-of-c99-t.patch
> @@ -0,0 +1,36 @@
> +From 742fb110d9841a04b3ced256b0bf80ff304dcaff Mon Sep 17 00:00:00 2001
> +From: Khem Raj 
> +Date: Mon, 31 Aug 2015 05:45:08 +
> +Subject: [PATCH] Define _u32/__s32/__u16/__s16/__u8 in terms of c99 types
> +
> +Signed-off-by: Khem Raj 
> +---
> +Upstream-Status: Pending
> +
> + wireless_copy.h | 10 +-
> + 1 file changed, 5 insertions(+), 5 deletions(-)
> +
> +diff --git a/wireless_copy.h b/wireless_copy.h
> +index 8208258..1171a35 100644
> +--- a/wireless_copy.h
>  b/wireless_copy.h
> +@@ -86,11 +86,11 @@
> + #else
> + #include 
> + #include 
> +-typedef __uint32_t __u32;
> +-typedef __int32_t __s32;
> +-typedef __uint16_t __u16;
> +-typedef __int16_t __s16;
> +-typedef __uint8_t __u8;
> ++typedef u_int32_t __u32;
> ++typedef int32_t __s32;
> ++typedef u_int16_t __u16;
> ++typedef int16_t __s16;
> ++typedef u_int8_t __u8;
> + #ifndef __user
> + #define __user
> + #endif /* __user */
> +-- 
> +2.5.1
> +
> diff --git a/meta/recipes-bsp/hostap/hostap-utils.inc 
> b/meta/recipes-bsp/hostap/hostap-utils.inc
> index 89d977a..140321d 100644
> --- a/meta/recipes-bsp/hostap/hostap-utils.inc
> +++ b/meta/recipes-bsp/hostap/hostap-utils.inc
> @@ -10,7 +10,9 @@ SECTION = "kernel/userland"
>  PR = "r4"
>  
>  SRC_URI = "http://hostap.epitest.fi/releases/hostap-utils-${PV}.tar.gz \
> -file://hostap-fw-load.patch"
> +   file://hostap-fw-load.patch \
> +   
> file://0001-Define-_u32-__s32-__u16-__s16-__u8-in-terms-of-c99-t.patch \
> +"
>  S = "${WORKDIR}/hostap-utils-${PV}"
>  
>  BINARIES = "hostap_crypt_conf hostap_diag hostap_fw_load hostap_io_debug \
> -- 
> 2.5.2
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Getting rid of "The license listed internal was not in the licenses collected" warnings?

[Mikko.Rapeli]

On Tue, Sep 08, 2015 at 12:19:51PM +0200, Mike Looijmans wrote:
> Ah, wait, on closer inspection it actually exists, but it's a symlink to
> some none-existing location. Apparently, to save some diskspace and reduce
> maintenance on text files I linked a few files together. Somehow, the link
> got copied instead of the file contents.
>
> I un-symlinked the "internal" file, and (together with some forced rebuilds)
> that finally made the warning go away.

Since I had similar stupid-user moments too, there is a generic function in
sanity tests to check for broken symlinks. Would be great if you could add
license files to this check.

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] shell script guidelines in oe-core? (was Re: [PATCH v4] create-pull-request: cleanup bashisms)

[Mikko.Rapeli]

On Sat, Aug 15, 2015 at 06:38:18PM -0700, Christopher Larson wrote:
 Update: checkbashisms and shellcheck both check for this now.

This is great! How about formulating that oe-core shell scripts should
pass shellcheck and checkbashisms checks without warnings?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] shell script guidelines in oe-core? (was Re: [PATCH v4] create-pull-request: cleanup bashisms)

[Mikko.Rapeli]

On Wed, Aug 12, 2015 at 10:51:26AM -0700, Christopher Larson wrote:
 That reminds me, there's a shell portability issue / standards-complaince
 issue that's not identified by shellcheck. Typically, to negate a  bracket
 expression in a regular expression, one uses ^, e.g. [^a-z] is everything
 that isn't in the range a to z, but that's not the case in shell, e.g. at a
 prompt or in a ${foo#pattern}:
 
 [...]a bracket expression as in XBD *RE Bracket Expression* , except
 that the exclamation-mark character ( '!' ) shall replace the
 circumflex character ( '^' ) in its role in a non-matching list in the
 regular expression notation
 
 So in shell, you'd want [!a-z] rather than [^a-z]. Nearly all shells handle
 both, but the behavior of the latter is actually unspecified according to
 the standard:
 
 A bracket expression starting with an unquoted circumflex character
 produces unspecified results.
 
 I recently got bitten by this with one of my shell scripts on a system
 running dash.

Does checkbashisms warn about this?

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core