Re: [OE-core] Yocto : Need help in building latest openembedded branch code

2021-11-16 Thread Mark Hatle

The variable syntax has changed between older and newer versions of bitbake.

See: 
https://docs.yoctoproject.org/migration-guides/migration-3.4.html#override-syntax-changes


On 11/15/21 11:59 PM, mohammed aqdam wrote:

Hi There,

I am working on a yocto image for the imx8 board, for this I have created a 
project with zeus layers for all layers(including meta-imx) using the below 
manifest file.


|repo init -u https:||//source||.codeaurora.org
||/external/imx/imx-manifest||-b imx-linux-zeus -m
imx-5.4.47-2.2.0.xml|


I am trying to fetch master/honister code of openEmbedded on my 
existing project(Earlier i was using Zeus branch) to use latest modules of 
perl(5.34.0), chrony(4.1) & gpsd(gpsd 3.23.1 
), by modifying 
.repo/manifests/imx-5.4.47-2.2.0.xml with master/honister code for 
openEmbedded(revision="refs/heads/honister"), later i ran into below error,


$ bitbake -k imx-image-full
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session
(/data/home/maqdam/imx_yocto_master/build/bitbake-cookerdaemon.log):
--- Starting bitbake server pid 6315 at 2021-11-16 10:36:44.095759 ---
ERROR: ParseError at

/data/home/maqdam/imx_yocto_master/sources/meta-openembedded/meta-oe/conf/layer.conf:106:
unparsed line: 'DEFAULT_TEST_SUITES:pn-meta-oe-ptest-image = "
${PTESTTESTSUITE}"'


Tried updating the bitbake layer(poky) to master/honister in similar fashion, 
Later seeing the below error in meta-imx/meta-bsp/conf/layer.conf file.


$ bitbake -k imx-image-full
ERROR: Variable FSL_EULA_FILE_MD5SUMS_append contains an operation using the
old override syntax. Please convert this layer/metadata before attempting to
use with a newer bitbake.


I could not find how to use FSL_EULA_FILE_MD5SUMS_append with the latest 
bitbake.
Request you to suggest how can i use this master/honister code of openEmbedded & 
with which version of bitbake.


Thanks,
Aqdam





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#158345): 
https://lists.openembedded.org/g/openembedded-core/message/158345
Mute This Topic: https://lists.openembedded.org/mt/87089467/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH] openssh: openssh-dev shouldn't depend on openssh

2021-09-29 Thread Mark Hatle


On 9/29/21 3:36 AM, Matt Johnston wrote:
> On Wed, 2021-09-29 at 09:24 +0100, Richard Purdie wrote:
>>> +RDEPENDS:${PN}-dev = ""
>>
>> At that point what is the point of the -dev package? I think you could make
>> this argument about a lot of the -dev packages and I'm not sure I'd want to
>> see every recipe doing this.
>>
>> What are you using the -dev package for and do you need it at all?
> 
> The -dev package isn't needed for anything, but it gets pulled in when
> "populate_sdk" is built. That does a glob over all -dev packages for
> any built package.

The dev package's SHOULD depend on the regular package by default.  There are
VERY VERY few instances where the dev package is "useful" without the
corresponding run-time package in a 'normal' use-case.  (There are always
exceptions and corner cases, but they're just that exceptions and corner cases
in my experience and lead to massive confusion by developers if exploited.)

populate_sdk, the entire purpose of this is to install the -dev package (and
it's dependencies) for anything in your root filesystem.  This says that you
root filesystem contains "openssh", thus openssh-dev will be installed.

If your root filesystem does NOT have openssh in it, then we need to answer the
question why was the -dev version added?

You mention above it's "any built package", but that should not be the case.  It
should be for any package installed into the corresponding root filesystem.

i.e. bitbake core-image-minimal -c do_populate_sdk

This should NOT install -dev packages for things not located in
core-image-minimal.  If it does, then there is another problem, that was not the
intended design.  The intention when the do_populate_sdk was written was
construct the rootfs, parse what was install and add -dev packages + their
dependencies.

--Mark

> Cheers,
> Matt
> 
>>
>> [As an aside, I do think the -dev package dependencies need rethinking but
>> that is a lot more work which nobody seems very interested in]
>>
>> Cheers,
>>
>> Richard
>>
> 
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156456): 
https://lists.openembedded.org/g/openembedded-core/message/156456
Mute This Topic: https://lists.openembedded.org/mt/85941131/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] user/group XXX does not exist, using root with RPM/DNF packaging in Hardknott and Honister

2021-09-24 Thread Mark Hatle


On 9/24/21 9:02 AM, Zoltan Boszormenyi via lists.openembedded.org wrote:
> Hi,
> 
> I have a special package that creates users and groups
> via inherit useradd. This package doesn't depend on any
> others but it is depended on, both via DEPENDS and RDEPENDS
> by packages using those users/groups in their do_install
> scripts.
> 
> This works for packaging becase these ownerships
> are encoded in the packages, confirmed by rpm -qp --dump ...

Does it show the useradd in the _PREINSTALL_ (you can use --scriptlets in the
rpm -qp)?

> However, during do_rootfs, a couple of
> "user/group XXX does not exist, using root"
> messages appear for the packages depending on others
> creating these users/groups.

Do the using packages contain RDEPENDS on the package that adds the user/group
to the system?

> log.do_rootfs shows that the package installation ordering
> does not follow RDEPENDS. Instead, it's practically an
> alphabetical order when running dnf.
> 
> This doesn't just involve my custom packages, but also clamav
> plus another one in which I ship a small limited set of
> virus signatures, also chown'd to clamav and with RDEPENDS
> on clamav.
> 
> What is the correct solution to this?

Typically the combination of the pre-install scriptlet, along with RDEPENDS will
ensure that the user has been added before the install completes.

--Mark

> Thanks in advance,
> Zoltán Böszörményi
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156337): 
https://lists.openembedded.org/g/openembedded-core/message/156337
Mute This Topic: https://lists.openembedded.org/mt/85839631/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] [PATCH] tcf-agent: Move to the latest master version

2021-09-16 Thread Mark Hatle
There has not been a release since 2018, the 1.7.0 release.  A number of
recent improvements around thumb and clang debugging prompted this move
to a newer version.

The patch is no longer necessary as it was a backport patch.

Signed-off-by: Mark Hatle 
---
 .../0001-Fixed-copyright-messages.patch   | 56 ---
 .../tcf-agent/tcf-agent_git.bb|  3 +-
 2 files changed, 1 insertion(+), 58 deletions(-)
 delete mode 100644 
meta/recipes-devtools/tcf-agent/tcf-agent/0001-Fixed-copyright-messages.patch

diff --git 
a/meta/recipes-devtools/tcf-agent/tcf-agent/0001-Fixed-copyright-messages.patch 
b/meta/recipes-devtools/tcf-agent/tcf-agent/0001-Fixed-copyright-messages.patch
deleted file mode 100644
index ed73808e990..000
--- 
a/meta/recipes-devtools/tcf-agent/tcf-agent/0001-Fixed-copyright-messages.patch
+++ /dev/null
@@ -1,56 +0,0 @@
-From 20ac1f939a8a97b03abec55d865050fdaa0f343a Mon Sep 17 00:00:00 2001
-From: Eugene Tarassov 
-Date: Tue, 8 Jan 2019 21:56:16 -0800
-Subject: [oe-core][PATCH 1/1] Fixed copyright messages
-
-Upstream-Status: Backport 
[https://git.eclipse.org/c/tcf/org.eclipse.tcf.agent.git/commit/?id=20ac1f939a8a97b03abec55d865050fdaa0f343a]
-
-Signed-off-by: Joe Slater 
-Signed-off-by: Mingli Yu 

- agent/tcf/framework/channel_lws.c  |  2 +-
- agent/tcf/framework/cpudefs-mdep-mux.h | 19 +--
- 2 files changed, 14 insertions(+), 7 deletions(-)
-
-diff --git a/agent/tcf/framework/channel_lws.c 
b/agent/tcf/framework/channel_lws.c
-index 0cb9585..d9352f3 100644
 a/tcf/framework/channel_lws.c
-+++ b/tcf/framework/channel_lws.c
-@@ -1,5 +1,5 @@
- 
/***
-- * Copyright (c) 2016-2017 Wind River Systems, Inc.
-+ * Copyright (c) 2016-2018 Wind River Systems, Inc. and others.
-  * All rights reserved. This program and the accompanying materials
-  * are made available under the terms of the Eclipse Public License v1.0
-  * and Eclipse Distribution License v1.0 which accompany this distribution.
-diff --git a/agent/tcf/framework/cpudefs-mdep-mux.h 
b/agent/tcf/framework/cpudefs-mdep-mux.h
-index c9e0db7..0397a6a 100644
 a/tcf/framework/cpudefs-mdep-mux.h
-+++ b/tcf/framework/cpudefs-mdep-mux.h
-@@ -1,10 +1,17 @@
--/*
-- * Copyright (c) 2013 Wind River Systems, Inc.
-+/***
-+ * Copyright (c) 2013 Wind River Systems, Inc. and others.
-+ * All rights reserved. This program and the accompanying materials
-+ * are made available under the terms of the Eclipse Public License v1.0
-+ * and Eclipse Distribution License v1.0 which accompany this distribution.
-+ * The Eclipse Public License is available at
-+ * http://www.eclipse.org/legal/epl-v10.html
-+ * and the Eclipse Distribution License is available at
-+ * http://www.eclipse.org/org/documents/edl-v10.php.
-+ * You may elect to redistribute this code under either of these licenses.
-  *
-- * The right to copy, distribute, modify or otherwise make use
-- * of this software may be licensed only pursuant to the terms
-- * of an applicable Wind River license agreement.
-- */
-+ * Contributors:
-+ * Wind River Systems - initial API and implementation
-+ 
***/
- 
- #include 
- 
--- 
-2.7.4
-
diff --git a/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb 
b/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb
index 4ff309dd80d..e67eccc75ce 100644
--- a/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb
+++ b/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb
@@ -6,7 +6,7 @@ BUGTRACKER = "https://bugs.eclipse.org/bugs/;
 LICENSE = "EPL-1.0 | EDL-1.0"
 LIC_FILES_CHKSUM = "file://edl-v10.html;md5=522a390a83dc186513f0500543ad3679"
 
-SRCREV = "a022ef2f1acfd9209a1bf792dda14ae4b0d1b60f"
+SRCREV = "2735e3d6b7eccb05ab232825c618c837d27a5010"
 PV = "1.7.0+git${SRCPV}"
 
 UPSTREAM_CHECK_GITTAGREGEX = "(?P(\d+(\.\d+)+))"
@@ -15,7 +15,6 @@ SRC_URI = 
"git://git.eclipse.org/r/tcf/org.eclipse.tcf.agent.git;protocol=https
file://ldflags.patch \
file://tcf-agent.init \
file://tcf-agent.service \
-   file://0001-Fixed-copyright-messages.patch \
   "
 
 DEPENDS = "util-linux openssl"
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156119): 
https://lists.openembedded.org/g/openembedded-core/message/156119
Mute This Topic: https://lists.openembedded.org/mt/85657893/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core][PATCH v3 2/2] externalsrc: Work with reproducible_build

2021-09-09 Thread Mark Hatle
From: Mark Hatle 

Externalsrc removes do_fetch, do_unpack, and do_patch.  The system normally
discovers the correct reproducible date as a postfuncs of do_unpack, so this
date is never found, so it falls back to the default epoch.

Instead we can move the discovery function to a prefuncs on the epoch
deploy task.  This task will run before do_configure, and since the source
is already available can run safely at anytime.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/externalsrc.bbclass | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 54b08adf62..7f1a760eec 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -110,6 +110,16 @@ python () {
 continue
 bb.build.deltask(task, d)
 
+if bb.data.inherits_class('reproducible_build', d) and 'do_unpack' in 
d.getVar("SRCTREECOVEREDTASKS").split():
+# The reproducible_build's create_source_date_epoch_stamp function 
must
+# be run after the source is available and before the
+# do_deploy_source_date_epoch task.  In the normal case, it's 
attached
+# to do_unpack as a postfuncs, but since we removed do_unpack 
(above)
+# we need to move the function elsewhere.  The easiest thing to do 
is
+# move it into the prefuncs of the do_deploy_source_date_epoch 
task.
+# This is safe, as externalsrc runs with the source already 
unpacked.
+d.prependVarFlag('do_deploy_source_date_epoch', 'prefuncs', 
'create_source_date_epoch_stamp ')
+
 d.prependVarFlag('do_compile', 'prefuncs', 
"externalsrc_compile_prefunc ")
 d.prependVarFlag('do_configure', 'prefuncs', 
"externalsrc_configure_prefunc ")
 
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155881): 
https://lists.openembedded.org/g/openembedded-core/message/155881
Mute This Topic: https://lists.openembedded.org/mt/85500953/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core][PATCH v3 0/2] Fixes for reproducible build and externalsrc

2021-09-09 Thread Mark Hatle
v3:
SAME AS V2, except I corrected a minor typographical error in a comment in
2/2.

v2:
I've reworked the patches completely based on the feedback and a discusion
with RP on IRC.

The first patch both simplifies and expands the description of what the
class is doing.  (To fully disable the behavior for a single recipe that
recipe would need to delVar SOURCE_DATE_EPOCH.  With this set, all sorts
of programs pay attention to the value, if it's blank it may error,
otherwise it falls back to either 0, or another date.)

The documentation is expanded to make it clear how to override the behavior
but it was not desired to explain how to 'disable' the behavior for a
single recipe.

For the externalsrc, the code was moved from the reproducible_build into
the externalsrc and made contingent on the do_unpack being removed.

Mark Hatle (2):
  reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking
  externalsrc: Work with reproducible_build

 meta/classes/externalsrc.bbclass| 10 +
 meta/classes/reproducible_build.bbclass | 53 -
 2 files changed, 44 insertions(+), 19 deletions(-)

-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155880): 
https://lists.openembedded.org/g/openembedded-core/message/155880
Mute This Topic: https://lists.openembedded.org/mt/85500952/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core][PATCH v3 1/2] reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking

2021-09-09 Thread Mark Hatle
From: Mark Hatle 

Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would
fall back and select the default epoch (April 2011), but still perform
the reproducible build actions.  This resulted in binaries that had an
unusually old date.

Simplify the functions and remove the anonymous python as no longer
necessary.

Also improve the documentation to better explain what the class is doing
and how a recipe can override the behavior if necessary.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/reproducible_build.bbclass | 53 -
 1 file changed, 34 insertions(+), 19 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass 
b/meta/classes/reproducible_build.bbclass
index 378121903d..96bb012243 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -1,17 +1,38 @@
 # reproducible_build.bbclass
 #
-# Sets SOURCE_DATE_EPOCH in each component's build environment.
+# Sets the default SOURCE_DATE_EPOCH in each component's build environment.
+# The format is number of seconds since the system epoch.
+#
 # Upstream components (generally) respect this environment variable,
 # using it in place of the "current" date and time.
 # See https://reproducible-builds.org/specs/source-date-epoch/
 #
-# After sources are unpacked but before they are patched, we set a 
reproducible value for SOURCE_DATE_EPOCH.
-# This value should be reproducible for anyone who builds the same revision 
from the same sources.
+# The default value of SOURCE_DATE_EPOCH comes from the function
+# get_source_date_epoch_value which reads from the SDE_FILE, or if the file
+# is not available (or set to 0) will use the fallback of
+# SOURCE_DATE_EPOCH_FALLBACK.
+#
+# The SDE_FILE is normally constructed from the function
+# create_source_date_epoch_stamp which is typically added as a postfuncs to
+# the do_unpack task.  If a recipe does NOT have do_unpack, it should be added
+# to a task that runs after the source is available and before the
+# do_deploy_source_date_epoch task is executed.
+#
+# If a recipe wishes to override the default behavior it should set it's own
+# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task
+# with recipe-specific functionality to write the appropriate
+# SOURCE_DATE_EPOCH into the SDE_FILE.
+#
+# SOURCE_DATE_EPOCH is intended to be a reproducible value.  This value should
+# be reproducible for anyone who builds the same revision from the same
+# sources.
 #
-# There are 4 ways we determine SOURCE_DATE_EPOCH:
+# There are 4 ways the create_source_date_epoch_stamp function determines what
+# becomes SOURCE_DATE_EPOCH:
 #
 # 1. Use the value from __source_date_epoch.txt file if this file exists.
-#This file was most likely created in the previous build by one of the 
following methods 2,3,4.
+#This file was most likely created in the previous build by one of the
+#following methods 2,3,4.
 #Alternatively, it can be provided by a recipe via SRC_URI.
 #
 # If the file does not exist:
@@ -22,20 +43,16 @@
 # 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
 #This works for well-kept repositories distributed via tarball.
 #
-# 4. Use the modification time of the youngest file in the source tree, if 
there is one.
+# 4. Use the modification time of the youngest file in the source tree, if
+#there is one.
 #This will be the newest file from the distribution tarball, if any.
 #
-# 5. Fall back to a fixed timestamp.
+# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK).
 #
-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the 
recipe's SDE_FILE.
-# If none of these mechanisms are suitable, replace the 
do_deploy_source_date_epoch task
-# with recipe-specific functionality to write the appropriate 
SOURCE_DATE_EPOCH into the SDE_FILE.
-#
-# If this file is found by other tasks, the value is exported in the 
SOURCE_DATE_EPOCH variable.
-# SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, 
do_compile, do_package, ...)
+# Once the value is determined, it is stored in the recipe's SDE_FILE.
 
 BUILD_REPRODUCIBLE_BINARIES ??= '1'
-inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 
'reproducible_build_simple', '')}
+inherit reproducible_build_simple
 
 SDE_DIR = "${WORKDIR}/source-date-epoch"
 SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
@@ -92,6 +109,9 @@ python create_source_date_epoch_stamp() {
 os.rename(tmp_file, epochfile)
 }
 
+# Generate the stamp after do_unpack runs
+do_unpack[postfuncs] += "create_source_date_epoch_stamp"
+
 def get_source_date_epoch_value(d):
 cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
 if cached:
@@ -120,8 +140,3 @@ def get_source_date_epoch_value(d):
 
 export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
 BB_HASHBASE_WHITELIST += "SOURCE_DATE_EP

[OE-core][PATCH v2 0/2] Fixes for reproducible build and externalsrc

2021-09-09 Thread Mark Hatle
I've reworked the patches completely based on the feedback and a discusion
with RP on IRC.

The first patch both simplifies and expands the description of what the
class is doing.  (To fully disable the behavior for a single recipe that
recipe would need to delVar SOURCE_DATE_EPOCH.  With this set, all sorts
of programs pay attention to the value, if it's blank it may error,
otherwise it falls back to either 0, or another date.)

The documentation is expanded to make it clear how to override the behavior
but it was not desired to explain how to 'disable' the behavior for a
single recipe.

For the externalsrc, the code was moved from the reproducible_build into
the externalsrc and made contingent on the do_unpack being removed.

Mark Hatle (2):
  reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking
  externalsrc: Work with reproducible_build

 meta/classes/externalsrc.bbclass| 10 +
 meta/classes/reproducible_build.bbclass | 53 -
 2 files changed, 44 insertions(+), 19 deletions(-)

-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155878): 
https://lists.openembedded.org/g/openembedded-core/message/155878
Mute This Topic: https://lists.openembedded.org/mt/85500124/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core][PATCH v2 1/2] reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking

2021-09-09 Thread Mark Hatle
From: Mark Hatle 

Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would
fall back and select the default epoch (April 2011), but still perform
the reproducible build actions.  This resulted in binaries that had an
unusually old date.

Simplify the functions and remove the anonymous python as no longer
necessary.

Also improve the documentation to better explain what the class is doing
and how a recipe can override the behavior if necessary.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/reproducible_build.bbclass | 53 -
 1 file changed, 34 insertions(+), 19 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass 
b/meta/classes/reproducible_build.bbclass
index 378121903d..96bb012243 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -1,17 +1,38 @@
 # reproducible_build.bbclass
 #
-# Sets SOURCE_DATE_EPOCH in each component's build environment.
+# Sets the default SOURCE_DATE_EPOCH in each component's build environment.
+# The format is number of seconds since the system epoch.
+#
 # Upstream components (generally) respect this environment variable,
 # using it in place of the "current" date and time.
 # See https://reproducible-builds.org/specs/source-date-epoch/
 #
-# After sources are unpacked but before they are patched, we set a 
reproducible value for SOURCE_DATE_EPOCH.
-# This value should be reproducible for anyone who builds the same revision 
from the same sources.
+# The default value of SOURCE_DATE_EPOCH comes from the function
+# get_source_date_epoch_value which reads from the SDE_FILE, or if the file
+# is not available (or set to 0) will use the fallback of
+# SOURCE_DATE_EPOCH_FALLBACK.
+#
+# The SDE_FILE is normally constructed from the function
+# create_source_date_epoch_stamp which is typically added as a postfuncs to
+# the do_unpack task.  If a recipe does NOT have do_unpack, it should be added
+# to a task that runs after the source is available and before the
+# do_deploy_source_date_epoch task is executed.
+#
+# If a recipe wishes to override the default behavior it should set it's own
+# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task
+# with recipe-specific functionality to write the appropriate
+# SOURCE_DATE_EPOCH into the SDE_FILE.
+#
+# SOURCE_DATE_EPOCH is intended to be a reproducible value.  This value should
+# be reproducible for anyone who builds the same revision from the same
+# sources.
 #
-# There are 4 ways we determine SOURCE_DATE_EPOCH:
+# There are 4 ways the create_source_date_epoch_stamp function determines what
+# becomes SOURCE_DATE_EPOCH:
 #
 # 1. Use the value from __source_date_epoch.txt file if this file exists.
-#This file was most likely created in the previous build by one of the 
following methods 2,3,4.
+#This file was most likely created in the previous build by one of the
+#following methods 2,3,4.
 #Alternatively, it can be provided by a recipe via SRC_URI.
 #
 # If the file does not exist:
@@ -22,20 +43,16 @@
 # 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
 #This works for well-kept repositories distributed via tarball.
 #
-# 4. Use the modification time of the youngest file in the source tree, if 
there is one.
+# 4. Use the modification time of the youngest file in the source tree, if
+#there is one.
 #This will be the newest file from the distribution tarball, if any.
 #
-# 5. Fall back to a fixed timestamp.
+# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK).
 #
-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the 
recipe's SDE_FILE.
-# If none of these mechanisms are suitable, replace the 
do_deploy_source_date_epoch task
-# with recipe-specific functionality to write the appropriate 
SOURCE_DATE_EPOCH into the SDE_FILE.
-#
-# If this file is found by other tasks, the value is exported in the 
SOURCE_DATE_EPOCH variable.
-# SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, 
do_compile, do_package, ...)
+# Once the value is determined, it is stored in the recipe's SDE_FILE.
 
 BUILD_REPRODUCIBLE_BINARIES ??= '1'
-inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 
'reproducible_build_simple', '')}
+inherit reproducible_build_simple
 
 SDE_DIR = "${WORKDIR}/source-date-epoch"
 SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
@@ -92,6 +109,9 @@ python create_source_date_epoch_stamp() {
 os.rename(tmp_file, epochfile)
 }
 
+# Generate the stamp after do_unpack runs
+do_unpack[postfuncs] += "create_source_date_epoch_stamp"
+
 def get_source_date_epoch_value(d):
 cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
 if cached:
@@ -120,8 +140,3 @@ def get_source_date_epoch_value(d):
 
 export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
 BB_HASHBASE_WHITELIST += "SOURCE_DATE_EP

[OE-core][PATCH v2 2/2] externalsrc: Work with reproducible_build

2021-09-09 Thread Mark Hatle
From: Mark Hatle 

Externalsrc removes do_fetch, do_unpack, and do_patch.  The system normally
discovers the correct reproducible date as a postfuncs of do_unpack, so this
date is never found, so it falls back to the default epoch.

Instead we can move the discovery function to a prefuncs on the epoch
deploy task.  This task will run before do_configure, and since the source
is already available can run safely at anytime.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/externalsrc.bbclass | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 54b08adf62..34921b39a2 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -110,6 +110,16 @@ python () {
 continue
 bb.build.deltask(task, d)
 
+if bb.data.inherits_class('reproducible_build', d) and 'do_unpack' in 
d.getVar("SRCTREECOVEREDTASKS").split():
+# The reproducible_build's create_source_date_epoch_stamp function 
must
+# be run after the source is available and before the
+# do_deploy_source_date_epoch task.  In the normal case, it's 
attached
+# to do_unpack as a postfuncs, but since we removed do_unpack 
(above)
+# we need to move the function elsewhere.  The easiest thing to do 
is
+# move it into the prefuncs of the do_deploy_source_date_epoch 
task.
+# This is safe, as externalsrc runs with the source is already 
unpacked.
+d.prependVarFlag('do_deploy_source_date_epoch', 'prefuncs', 
'create_source_date_epoch_stamp ')
+
 d.prependVarFlag('do_compile', 'prefuncs', 
"externalsrc_compile_prefunc ")
 d.prependVarFlag('do_configure', 'prefuncs', 
"externalsrc_configure_prefunc ")
 
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155876): 
https://lists.openembedded.org/g/openembedded-core/message/155876
Mute This Topic: https://lists.openembedded.org/mt/85500122/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core][PATCH 1/2] reproducible_build: Honor BUILD_REPRODUCIBLE_BINARIES

2021-09-08 Thread Mark Hatle
From: Mark Hatle 

A variable BUILD_REPRODUCIBLE_BINARIES is set to '1' by default and was used
to control if the postfuncs were added.  However, it didn't actually disable
the reproducible_build (date) logic.  This resulted in the program falling
back to the default date.

This change fully honors the variable and disables the various pieces
that discover and configure the SOURCE_DATE_EPOCH if the value if not
set to '1'.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/reproducible_build.bbclass | 17 ++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass 
b/meta/classes/reproducible_build.bbclass
index 378121903d..a9c117c3b9 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -47,7 +47,9 @@ TARGET_CC_ARCH:append:class-target = " -Wdate-time"
 # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
 export SOURCE_DATE_EPOCH_FALLBACK ??= "1302044400"
 
-SSTATETASKS += "do_deploy_source_date_epoch"
+# The following are preformed in the anonymous python function, only if
+# BUILD_REPRODUCIBLE_BINARIES == 1
+#SSTATETASKS += "do_deploy_source_date_epoch"
 
 do_deploy_source_date_epoch () {
 mkdir -p ${SDE_DEPLOYDIR}
@@ -73,8 +75,10 @@ python do_deploy_source_date_epoch_setscene () {
 
 do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
 do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
-addtask do_deploy_source_date_epoch_setscene
-addtask do_deploy_source_date_epoch before do_configure after do_patch
+# The following are preformed in the anonymous python function, only if
+# BUILD_REPRODUCIBLE_BINARIES == 1
+#addtask do_deploy_source_date_epoch_setscene
+#addtask do_deploy_source_date_epoch before do_configure after do_patch
 
 python create_source_date_epoch_stamp() {
 import oe.reproducible
@@ -123,5 +127,12 @@ BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH"
 
 python () {
 if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
+# Generate the timestamp with create_source_date_epoch_stamp.
 d.appendVarFlag("do_unpack", "postfuncs", " 
create_source_date_epoch_stamp")
+d.appendVar('SSTATETASKS', " do_deploy_source_date_epoch")
+bb.build.addtask('do_deploy_source_date_epoch_setscene', None, None, d)
+bb.build.addtask('do_deploy_source_date_epoch', 'do_configure', 
'do_patch', d)
+else:
+# If this is set at all, the system components will attempt to use it
+d.delVar('SOURCE_DATE_EPOCH')
 }
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155839): 
https://lists.openembedded.org/g/openembedded-core/message/155839
Mute This Topic: https://lists.openembedded.org/mt/85476713/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core][PATCH 0/2] Fixes for reproducible build and externalsrc

2021-09-08 Thread Mark Hatle
Two fixes, the first one ensures that the BUILD_REPRODUCIBLE_BINARIES variable
actually does something useful.  1 (default) normal behavior, anything else it
is off and regular behavior works.

The second makes the reproducible build work properly with externalsrc.

To test this, I did:

bitbake bash
rpm -qp tmp/deploy/.../bash-...rpm --queryformat '[%{FILENAMES} 
%{FILEMTIMES}\n]'

cleanall the above, and then add:

BUILD_REPRODUCIBLE_BINARIES:pn-bash = '0'

repeat the test...

You can do the same with externalsrc via a bbappen adding externalsrc and repeat
the previous tests and get expected results.

Mark Hatle (2):
  reproducible_build: Honor BUILD_REPRODUCIBLE_BINARIES
  reproducible_build: Work with externalsrc

 meta/classes/reproducible_build.bbclass | 25 +
 1 file changed, 21 insertions(+), 4 deletions(-)

-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155838): 
https://lists.openembedded.org/g/openembedded-core/message/155838
Mute This Topic: https://lists.openembedded.org/mt/85476712/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core][PATCH 2/2] reproducible_build: Work with externalsrc

2021-09-08 Thread Mark Hatle
From: Mark Hatle 

Externalsrc removes do_fetch, do_unpack, and do_patch.  The system normally
discovers the correct reproducible date as a postfuncs on do_unpack, so this
date is never found, so it goes back to the default epoch.

Instead we can move the discovery function to a prefuncs on the epoch
deploy task.  This task will run before do_configure, and since the source
is already available can run anytime safely.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/reproducible_build.bbclass | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes/reproducible_build.bbclass 
b/meta/classes/reproducible_build.bbclass
index a9c117c3b9..ae0723ab21 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -128,7 +128,13 @@ BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH"
 python () {
 if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
 # Generate the timestamp with create_source_date_epoch_stamp.
-d.appendVarFlag("do_unpack", "postfuncs", " 
create_source_date_epoch_stamp")
+# In most cases this will be a postfuncs of do_unpack.
+# If we're running in external source, add 
create_source_date_epoch_stamp
+# to do_deploy_source_date_epoch instead because there is no do_unpack.
+if d.getVar('EXTERNALSRC') and bb.data.inherits_class('externalsrc', 
d):
+d.appendVarFlag('do_deploy_source_date_epoch', 'prefuncs', ' 
create_source_date_epoch_stamp')
+else:
+d.appendVarFlag("do_unpack", "postfuncs", " 
create_source_date_epoch_stamp")
 d.appendVar('SSTATETASKS', " do_deploy_source_date_epoch")
 bb.build.addtask('do_deploy_source_date_epoch_setscene', None, None, d)
 bb.build.addtask('do_deploy_source_date_epoch', 'do_configure', 
'do_patch', d)
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#155840): 
https://lists.openembedded.org/g/openembedded-core/message/155840
Mute This Topic: https://lists.openembedded.org/mt/85476714/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] should recipes not come with their own systemd/user/group info?

2021-08-05 Thread Mark Hatle


On 8/5/21 6:01 AM, Robert P. J. Day wrote:
> 
>   style question here, but i'm perusing an OE/YP (actually wind river,
> but effectively the same thing) layer that was handed to me, and i'm
> puzzled by how the internally-written systemd-controlled services are
> implemented.
> 
>   on the one hand, there are reasonable recipes that each represent
> some service to be managed by systemd, and they all install their
> executables in the right places and so on. however, rather than each
> recipe also providing its own service file, there is a totally
> separate recipe (call it "systemd-services.bb") which contains under
> its files/ directory dozens and dozens of service files for all of
> those recipes, and is responsible for installing all of them.
> 
>   in short, someone decided that -- for whatever reason -- all of the
> systemd service files should be collected all in one place, and
> installed en masse. is this ... normal? i would have thought that it's
> the responsibility of each recipe to mannage all the content related
> to it, and that would of course include the systemd service file to be
> installed for it. i don't know the rationale for this, and it strikes
> me as a recipe for disaster in trying to keep all these things synced
> up. is there some rationale for doing it this way that i'm unaware
> of?

For a 'product' (as in, this is only for one implementation and nobody will ever
re-use it), I've see this type of thing before.  But for anything intended to be
re-used, it's simply broken.

The service files belong with the recipes providing the services.

>   and closely related, the same trick was used when defining all the
> new users and groups that would own these services. rather than each
> service looking after itself, there is a recipe file (call it
> "new_users_and_groups.bb") which is hundreds of lines of USERADD_PARAM
> and GROUPADD_PARAM lines that define precisely (numerically) all the
> new user and group accounts to be associated with all sorts of
> services, whereupon i ask the same question -- isn't it more typical
> that each service look after its own user and group info?

The same is true here, if you are creating a singular product that has a number
of defined users and groups, sure you can do it in a recipe like this.  But it's
not the right way to do it.  Users/groups should be associated with their users.
 If these are just general administrative usages then they COULD be done this
way, but more people do it at image generation time.

>   in this second case, it *might* be that all of these values need to
> be hard-coded to be consistent with, i don't know, something else i'm
> unaware of, but it still looks strange.

If you need to 'adjust' users and groups, you should be using the existing
mechanisms, useradd-staticids for instance.

--Mark

>   thoughts? the above actually works for the time being, but it does
> seem a but unnatural.
> 
> rday
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#154522): 
https://lists.openembedded.org/g/openembedded-core/message/154522
Mute This Topic: https://lists.openembedded.org/mt/84682032/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH 6/9] shadow: update 4.8.1 -> 4.9

2021-08-04 Thread Mark Hatle


On 8/4/21 1:13 PM, Khem Raj wrote:
> 
> 
> On 8/4/21 3:12 AM, Alexander Kanavin wrote:
>> Yes, plaintext passwords can no longer be there, which is a good thing 
>> I'd say? The hashed/salted passwords can still be provided through the 
>> same class, but this needs to be documented, and perhaps tested too.
>>
> 
> Its perhaps fine to discourage plaintext password setting, but it is a 
> user visible feature as it seems. So the documentation should change for 
> sure to not use it and it should also go into migration guide since it 
> has a potential of tripping a lot of folks. I think documenting the 
> intent to move away from plaintext is urgent, then the question is if
> we want to fist deprecate it or delete this option all in one go.

We SHOULD discourage users from any hardcoded passwords!  But, there is little
to no functional difference between specifying a plain text or salted password,
but there is a HUGE developer/user difference in behavior.

So, if we have a way to set a default password for any account, then we really
do need a way to have a plaintext password specified.

>From a security perspective, there is no advantage between a salted or plain
text password.  (Salted passwords can always be reversed through tables, etc!)

If the current implementation of the plain text passwords is not "secure" due to
bad salts, hash types, etc.  Then lets fix that and move to a more secure style.

If it is decided to remove the -P option for plain text passwords, then we need
to document for the user HOW to generate password hashes.  And if we're showing
them how to do it, it SHOULD be trivial to find a way to do the same thing
_using the build system_.

For example

useradd -P 'foobar' user

to

hash=$(echo 'foobar' | openssl passwd -1 -salt mysalt -stdin)
useradd -p $hash user


or

hash=$(python -c "import crypt; print crypt.crypt('foobar')")
useradd -p $hash user


or




but the point is, we SHOULD discourage _ANY_ hard coded passwords, not just
plain text.  However if a user wants to do this, the system should assist the
user in setting a password into their environment.

--Mark


>> Alex
>>
>> On Wed, 4 Aug 2021 at 10:39, Yi Zhao > > wrote:
>>
>>
>> On 7/30/21 7:45 PM, Alexander Kanavin wrote:
>>> Add a couple backports to fix builds.
>>>
>>> Drop 0002-Allow-for-setting-password-in-clear-text.patch;
>>> what it adds is horribly insecure and AB testing didn't reveal any
>>> regressions or use cases for it.
>>
>> Dropping this patch makes the password setting function in
>> extrausers.bbclass unavailable:
>> https://docs.yoctoproject.org/singleindex.html#extrausers-bbclass
>> 
>>
>>
>> //Yi
>>
>>
>>> Drop /etc/default/ tweaks as files are no longer installed there.
>>>
>>> Drop manpage alternatives as manpages are no longer installed.
>>>
>>> Signed-off-by: Alexander Kanavin  
>>> 
>>> ---
>>>   ...01-Disable-use-of-syslog-for-sysroot.patch |  29 +-
>>>   ...builds-with-respect-to-libsubid-incl.patch | 114 +++
>>>   .../0001-libsubid-link-to-PAM-libraries.patch |  31 ++
>>>   ...w-for-setting-password-in-clear-text.patch | 301 --
>>>   ...nexpected-open-failure-in-chroot-env.patch |   6 +-
>>>   meta/recipes-extended/shadow/shadow.inc   |  21 +-
>>>   .../shadow/{shadow_4.8.1.bb    
>>> =>shadow_4.9.bb  } |   0
>>>   7 files changed, 167 insertions(+), 335 deletions(-)
>>>   create mode 100644 
>>> meta/recipes-extended/shadow/files/0001-Fix-out-of-tree-builds-with-respect-to-libsubid-incl.patch
>>>   create mode 100644 
>>> meta/recipes-extended/shadow/files/0001-libsubid-link-to-PAM-libraries.patch
>>>   delete mode 100644 
>>> meta/recipes-extended/shadow/files/0002-Allow-for-setting-password-in-clear-text.patch
>>>   rename meta/recipes-extended/shadow/{shadow_4.8.1.bb  
>>>   =>shadow_4.9.bb  } (100%)
>>>
>>> diff --git 
>>> a/meta/recipes-extended/shadow/files/0001-Disable-use-of-syslog-for-sysroot.patch
>>>  
>>> b/meta/recipes-extended/shadow/files/0001-Disable-use-of-syslog-for-sysroot.patch
>>> index ab317b9aa0..95728bcd3f 100644
>>> --- 
>>> a/meta/recipes-extended/shadow/files/0001-Disable-use-of-syslog-for-sysroot.patch
>>> +++ 
>>> b/meta/recipes-extended/shadow/files/0001-Disable-use-of-syslog-for-sysroot.patch
>>> @@ -1,4 +1,4 @@
>>> -From fa2d9453656641002802d8165e80adb9e6a729d2 Mon Sep 17 00:00:00 2001
>>> +From 30a3906a0a21120fa6bbc918b6258ab9303fbeaa Mon Sep 17 00:00:00 2001
>>>   From: Scott Garman  
>>> 
>>>   Date: Thu, 14 Apr 2016 12:28:57 +0200
>>>   Subject: [PATCH] Disable use of syslog for sysroot
>>> @@ -19,12 +19,12 @@ Signed-off-by: Chen Qi  

Re: [OE-core] Image creation using RPM/DNF and adding a third party repository configuration

2021-07-15 Thread Mark Hatle


On 7/15/21 4:24 PM, Mark Hatle wrote:
> We have the desire to add a 3rd party repository to our standard images
> generated by OE image generation.  We do this by adding a recipe
> (my-external-repo.bb) that creates a package that adds a repository
> configuration file to ${sysconfdir}/yum.repos.d/my_external.repo.
> 
> If the package is added to the INSTALL_IMAGE, then when it processes the
> IMAGE_FEATURES the system will fail with SSL errors trying to access the new
> third party repository.
> 
> What I believe is happening is that the system (first pass) installs all of 
> the
> main packages, including this repository configuration.  It then starts a 
> second
> pass to install -dev, -src or other components.  This second pass fails with 
> the
> SSL errors (like DNF can't access the SSL certification(s) it needs.)  But 
> even
> if the SSL issue wasn't really a problem, it does point out that DNF is trying
> to access the network and could install something from this third party
> repository, which I don't think is desired.
> 
> An alternative could be to add the third party repository via the ROOTFS POST
> install actions, but this has the problem that we won't be able to update the
> repository if something changes (via a package).
> 
> I'm thinking this MIGHT be a bug in the current implementation, that if 
> someone
> injects a config file it can cause the system to use alternative repositories.
> So should we be checking the directory between passes and sanitizing it?
> 
> Looking for suggestions...
> 
> Thanks!
> --Mark
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153896): 
https://lists.openembedded.org/g/openembedded-core/message/153896
Mute This Topic: https://lists.openembedded.org/mt/84235991/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] [PATCH] populate_sdk_ext: Error if trying to generate an eSDK from a mulitconfig

2021-06-29 Thread Mark Hatle
Running a build such as:

  bitbake mc:my_config:core-image-minimal -c populate_sdk-ext

will result in an error like:

ERROR: Task base-files.do_fetch attempted to execute unexpectedly
Task 
.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_packagedata,
 unihash ec5ba0e6b31561daba005fb49c5239c8e46913465b51166b5905f3e5ffcf2741, 
taskhash ec5ba0e6b31561daba005fb49c5239c8e46913465b51166b5905f3e5ffcf2741
Task 
.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package_write_rpm,
 unihash 1c7d7509c2ff6dcf11009fbec444726826214795d60474ec8d3262d89c40a955, 
taskhash 1c7d7509c2ff6dcf11009fbec444726826214795d60474ec8d3262d89c40a955
Task 
.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_populate_sysroot,
 unihash 9cc3672f4fa62491f545b15cf617a64cd77d15a2cfd432b57d4b936bc415f40d, 
taskhash 9cc3672f4fa62491f545b15cf617a64cd77d15a2cfd432b57d4b936bc415f40d
Task 
.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package_qa,
 unihash 8ada5f62092c971df8dda1d71c728e42994e1dcf2bbdab419de43867d77b64cc, 
taskhash 8ada5f62092c971df8dda1d71c728e42994e1dcf2bbdab419de43867d77b64cc
Task 
.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/images/core-image-minimal.bb:do_image_qa,
 unihash 16656a339389e407a5fdca5d64983af845288f3b3cc5582398e5247efb393257, 
taskhash 16656a339389e407a5fdca5d64983af845288f3b3cc5582398e5247efb393257
Task 
.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/images/core-image-minimal.bb:do_image_complete,
 unihash ef88c74a9f4ae4d252c421eb4e399773aa50cea7c51ffbeed9011e5198a16abb, 
taskhash ef88c74a9f4ae4d252c421eb4e399773aa50cea7c51ffbeed9011e5198a16abb
This is usually due to missing setscene tasks. Those missing in this build 
were: 
{'.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package_qa',

Instead of letting the system error, we simply tell the user this is not 
supported.

As long as the eSDK is constructed based on the primary library, it works fine.

Signed-off-by: Mark Hatle 
---
 meta/classes/populate_sdk_ext.bbclass | 5 +
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 517b4e45ff0..4aabafa079b 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -750,6 +750,11 @@ fakeroot python do_populate_sdk_ext() {
 if d.getVar('SDK_ARCH') != d.getVar('BUILD_ARCH'):
 bb.fatal('The extensible SDK can currently only be built for the same 
architecture as the machine being built on - SDK_ARCH is set to %s (likely via 
setting SDKMACHINE) which is different from the architecture of the build 
machine (%s). Unable to continue.' % (d.getVar('SDK_ARCH'), 
d.getVar('BUILD_ARCH')))
 
+# FIXME hopefully we can remove this restriction at some point, but the 
eSDK
+# can only be built for the primary (default) multiconfig
+if d.getVar('BB_CURRENT_MC') != 'default':
+bb.fatal('The extensible SDK can currently only be built for the 
default multiconfig.  Currently trying to build for %s.' % 
d.getVar('BB_CURRENT_MC'))
+
 d.setVar('SDK_INSTALL_TARGETS', get_sdk_install_targets(d))
 if d.getVar('SDK_INCLUDE_BUILDTOOLS') == '1':
 buildtools_fn = get_current_buildtools(d)
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153434): 
https://lists.openembedded.org/g/openembedded-core/message/153434
Mute This Topic: https://lists.openembedded.org/mt/83876060/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] should the same recipe have two different WORKDIRs?

2021-06-16 Thread Mark Hatle
Part of the WORKDIR component is the target (package) architecture.  If the
recipe or one of it's inherits sets PACKAGE_ARCH = "${MACHINE_ARCH}" it will
move the component.

Where I've seen both directories used is when someone is building for multiple
target boards.. and some of the targets use the default PACKAGE_ARCH, and some
use MACHINE_ARCH.  This is very common with OpenGL and hardware acceleration.
Mesa for instance becomes machine dependent on some systems, but generic for 
others.

--Mark

On 6/16/21 11:30 AM, Robert P. J. Day wrote:
> 
>   perhaps i've just never noticed before, but a colleague asked me to
> debug some strangeness with his WRLinux build, and what i noticed was
> that the recipe that generated a package with a single (aarch64)
> executable created WORKDIRs under both directories:
> 
>   * cortexa53...
>   * acme-coyote   [actual target board]
> 
> this surprises me ... i thought that, based on the attributes of any
> recipe, it would have only one WORKDIR in the appropriate place. what
> does the above mean? i'm not sure what to make of it.
> 
> rday
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153028): 
https://lists.openembedded.org/g/openembedded-core/message/153028
Mute This Topic: https://lists.openembedded.org/mt/83584601/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH] mklibs-native: Fix build with gcc 11

2021-05-17 Thread Mark Hatle


On 5/17/21 2:46 PM, Andre McCurdy wrote:
> On Mon, May 17, 2021 at 10:05 AM Jacob Kroon  wrote:
>>
>> In gcc 11 the default mode for C++ is now -std=gnu++17 instead of 
>> -std=gnu++14,
>> in which support for dynamic exception specifications has been removed.
> 
> As much as I'd like to see mklibs fully supported in OE, at some point
> maybe we should just admit that it doesn't work and there's not enough
> collective motivation to fix it.
> 
> For mklibs to work, the build process should generate a PIC .a archive
> for every .so shared library (ie static libs need to be built and
> somehow forced to contain PIC object code - which may mean patching
> Makefiles etc) and then the PIC .a files need to be available
> alongside the .so libs when mklibs runs. We don't have any support for
> doing that, so why keep pretending to support mklibs?

We used to have exactly that.  The fact nobody is complaining it doesn't work
tells me nobody is using this and we should remove it.

The need for this on modern (embedded) systems is significantly less with a
function altnerative to glibc.  (musl)

--Mark

>> See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0003r5.html
>>
>> Signed-off-by: Jacob Kroon 
>> ---
>>  .../no-dynamic-exception-specifications.patch | 420 ++
>>  .../mklibs/mklibs-native_0.1.44.bb|   1 +
>>  2 files changed, 421 insertions(+)
>>  create mode 100644 
>> meta/recipes-devtools/mklibs/files/no-dynamic-exception-specifications.patch
>>
>> diff --git 
>> a/meta/recipes-devtools/mklibs/files/no-dynamic-exception-specifications.patch
>>  
>> b/meta/recipes-devtools/mklibs/files/no-dynamic-exception-specifications.patch
>> new file mode 100644
>> index 00..5989a67c4f
>> --- /dev/null
>> +++ 
>> b/meta/recipes-devtools/mklibs/files/no-dynamic-exception-specifications.patch
>> @@ -0,0 +1,420 @@
>> +In gcc 11 the default mode for C++ is now -std=gnu++17 instead of 
>> -std=gnu++14,
>> +in which support for dynamic exception specifications has been removed.
>> +
>> +See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0003r5.html
>> +
>> +Upstream-Status: Pending
>> +
>> +Signed-off-by: Jacob Kroon 
>> +
>> +Index: mklibs-0.1.44/src/mklibs-readelf/elf.cpp
>> +===
>> +--- mklibs-0.1.44.orig/src/mklibs-readelf/elf.cpp
>>  mklibs-0.1.44/src/mklibs-readelf/elf.cpp
>> +@@ -36,7 +36,7 @@ file::~file () throw ()
>> + delete *it;
>> + }
>> +
>> +-file *file::open (const char *filename) throw (std::bad_alloc, 
>> std::runtime_error)
>> ++file *file::open (const char *filename)
>> + {
>> +   struct stat buf;
>> +   int fd;
>> +@@ -72,7 +72,7 @@ file *file::open (const char *filename)
>> + }
>> +
>> + template
>> +-file *file::open_class(uint8_t *mem, size_t len) throw (std::bad_alloc, 
>> std::runtime_error)
>> ++file *file::open_class(uint8_t *mem, size_t len)
>> + {
>> +   switch (mem[EI_DATA])
>> +   {
>> +@@ -86,7 +86,7 @@ file *file::open_class(uint8_t *mem, siz
>> + }
>> +
>> + template 
>> +-file_data<_class, _data>::file_data(uint8_t *mem, size_t len) throw 
>> (std::bad_alloc, std::runtime_error)
>> ++file_data<_class, _data>::file_data(uint8_t *mem, size_t len)
>> + : file(mem, len)
>> + {
>> +   if (mem[EI_CLASS] != _class::id)
>> +@@ -190,7 +190,7 @@ section_data<_class, _data>::section_dat
>> + }
>> +
>> + template 
>> +-void section_data<_class, _data>::update(const file ) throw 
>> (std::bad_alloc)
>> ++void section_data<_class, _data>::update(const file )
>> + {
>> +   const section_type  =
>> + dynamic_cast 
>> &>(file.get_section(file.get_shstrndx()));
>> +@@ -204,7 +204,7 @@ section_type::~sec
>> + }
>> +
>> + template 
>> +-section_real<_class, _data, section_type_DYNAMIC>::section_real(Shdr 
>> *header, uint8_t *mem) throw (std::bad_alloc)
>> ++section_real<_class, _data, section_type_DYNAMIC>::section_real(Shdr 
>> *header, uint8_t *mem)
>> + : section_data<_class, _data>(header, mem)
>> + {
>> +   if (this->type != SHT_DYNAMIC)
>> +@@ -221,7 +221,7 @@ section_real<_class, _data, section_type
>> + }
>> +
>> + template 
>> +-void section_real<_class, _data, section_type_DYNAMIC>::update(const file 
>> ) throw (std::bad_alloc)
>> ++void section_real<_class, _data, section_type_DYNAMIC>::update(const file 
>> )
>> + {
>> +   section_data<_class, _data>::update(file);
>> +
>> +@@ -243,7 +243,7 @@ section_type::~sect
>> + }
>> +
>> + template 
>> +-section_real<_class, _data, section_type_DYNSYM>::section_real(Shdr 
>> *header, uint8_t *mem) throw (std::bad_alloc)
>> ++section_real<_class, _data, section_type_DYNSYM>::section_real(Shdr 
>> *header, uint8_t *mem)
>> + : section_data<_class, _data>(header, mem)
>> + {
>> +   if (this->type != SHT_DYNSYM)
>> +@@ -260,7 +260,7 @@ section_real<_class, _data, section_type
>> + }
>> +
>> + template 
>> +-void section_real<_class, _data, section_type_DYNSYM>::update(const file 
>> ) throw (std::bad_alloc)
>> ++void 

[OE-core] Multiconfig - sstate-cache re-use and error

2021-05-05 Thread Mark Hatle
I'm cross posting this cause I suspect it's a bitbake bug, but it requires
sstate-cache / setscene stuff which is from the oe-core side of things.  I
really don't know what is causing the problem, but I can at least reproduce it,
even if it ends up taking a while.

The configuration.  I'm building a system that has 4 multiconfigs (plus the
regular configuration), with the output being an SDK (main config), and 4
different rootfs from the multiconfig.  (See below for actual setup details for
the reproducer.)

What is happening.  The system starts building and appears to correctly defer
the building of the duplicate components.  Eventually it will start building the
item that is not deferred, and soon after it appears to change what is deferred
and will attempt to call setscene tasks, but the sstate-cache has not yet been
written to by the first task.  This is occurring on a completely new build, no
existing sstate-cache.

The console-log can show the order in which things are happening for a small
subset, so might be helpful.  In this chunk socat ended up showing this 
behavior:

NOTE: Deferring mc:qemuarm64-one:socat.bb:do_deploy_source_date_epoch after
mc:qemuarm64-four:socat.bb:do_deploy_source_date_epoch
[repeat for do_package]
[repeat for do_package_qa]
[repeat for do_package_write_rpm]
[repeat for do_packagedata]
[repeat for do_populate_lic]
[repeat for do_populate_sysroot]
*** repeat the sequence above for mc:qemuarm64-three which defers for four ***
*** repeat the sequence above for mc:qemuarm64-two which defers for four ***
NOTE: Running task 1362 of 15106 (mc:qemuarm64-four:socat_1.7.4.1.bb:do_fetch)
NOTE: recipe socat-1.7.4.1-r0: task do_fetch: Started
NOTE: recipe socat-1.7.4.1-r0: task do_fetch: Succeeded
NOTE: Running task 1397 of 15106 (mc:qemuarm64-four:socat_1.7.4.1.bb:do_unpack)
NOTE: recipe socat-1.7.4.1-r0: task do_unpack: Started
NOTE: recipe socat-1.7.4.1-r0: task do_unpack: Succeeded
NOTE: Running task 2617 of 15106 (mc:qemuarm64-four:socat_1.7.4.1.bb:do_patch)
NOTE: recipe socat-1.7.4.1-r0: task do_patch: Started
NOTE: recipe socat-1.7.4.1-r0: task do_patch: Succeeded
NOTE: Running task 2711 of 15106
(mc:qemuarm64-four:socat_1.7.4.1.bb:do_deploy_source_date_epoch)
NOTE: recipe socat-1.7.4.1-r0: task do_deploy_source_date_epoch: Started
NOTE: recipe socat-1.7.4.1-r0: task do_deploy_source_date_epoch: Succeeded
(starts deferring three again after one)
NOTE: Deferring mc:qemuarm64-three:socat_1.7.4.1.bb:do_deploy_source_date_epoch
after mc:qemuarm64-one:socat_1.7.4.1.bb:do_deploy_source_date_epoch
[repeat for do_package]
[repeat for do_package_qa]
[repeat for do_package_write_rpm]
[repeat for do_packagedata]
[repeat for do_populate_lic]
[repeat for do_populate_sysroot]
*** repeat the defer sequence above for mc:qemuarm64-two for one ***
NOTE: Running setscene task 1930 of 5424
(mc:qemuarm64-one:socat_1.7.4.1.bb:do_populate_sysroot_setscene)
NOTE: recipe socat-1.7.4.1-r0: task do_populate_sysroot_setscene: Started
ERROR: mc:qemuarm64-one:socat-1.7.4.1-r0 do_populate_sysroot_setscene: No
suitable staging package found
ERROR: Logfile of failure stored in:
...socat/1.7.4.1-r0/temp/log.do_populate_sysroot_setscene.154008
NOTE: recipe socat-1.7.4.1-r0: task do_populate_sysroot_setscene: Failed
WARNING: Setscene task
(mc:qemuarm64-one:socat_1.7.4.1.bb:do_populate_sysroot_setscene) failed with
exit code '1' - real task will be run instead
NOTE: Running task 3183 of 15106
(mc:qemuarm64-four:/socat_1.7.4.1.bb:do_populate_lic)
... now we have components of 'one' and 'four' building ...  eventually two and
three unlock as well and may build at the same time...

(Note, this build happened to be socat first, but other builds I've run have
different things that end up going first -- and it's not always the same items.)

Reproducer:

latest version of poky

$ . ./oe-init-build-env build-mc
$ mkdir conf/multiconfig

add the following files to multiconfig directory:

conf/multiconfig/qemuarm64-one.conf:
MACHINE = "qemuarm64"
SOC_VARIANT = "one"
TMPDIR = "${TOPDIR}/tmp-${MACHINE}-${SOC_VARIANT}-${TCLIBC}"

conf/multiconfig/qemuarm64-two.conf:
MACHINE = "qemuarm64"
SOC_VARIANT = "two"
TMPDIR = "${TOPDIR}/tmp-${MACHINE}-${SOC_VARIANT}-${TCLIBC}"

conf/multiconfig/qemuarm64-three.conf:
MACHINE = "qemuarm64"
SOC_VARIANT = "three"
TMPDIR = "${TOPDIR}/tmp-${MACHINE}-${SOC_VARIANT}-${TCLIBC}"

conf/multiconfig/qemuarm64-four.conf:
MACHINE = "qemuarm64"
SOC_VARIANT = "four"
TMPDIR = "${TOPDIR}/tmp-${MACHINE}-${SOC_VARIANT}-${TCLIBC}"

edit the conf/local.conf and add:

BBMULTICONFIG += "qemuarm64-one"
BBMULTICONFIG += "qemuarm64-two"
BBMULTICONFIG += "qemuarm64-three"
BBMULTICONFIG += "qemuarm64-four"

Start the build:
$ bitbake buildtools-tarball mc:qemuarm64-one:core-image-minimal
mc:qemuarm64-two:core-image-minimal mc:qemuarm64-three:core-image-minimal
mc:qemuarm64-four:core-image-minimal

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#151347): 

Re: [OE-core] master eSDK hash problem with METADATA_REVISION and base-files:do_install

2021-03-29 Thread Mark Hatle
I was using master-next (version was right before I sent the email), and it
still failed.

Looking at things, there doesn't appear to be any METADATA_REVISION set in the
normal build, but there is one set inside of the eSDK.  So it looks like some
corner case may have been missing.

If you notice it says .  Diffsigs I had, I don't know how the 'default'
version should be generated or not -- but it was definitely different inside and
outside the eSDK.

--Mark

On 3/28/21 9:11 PM, Chen Qi wrote:
> Hi Mark,
> 
> I think this has been fixed by the following commit.
> 
> """
> commit ff2ad51b801fd62e2abbc573ba2c9ee8fdc7e012
> Author: Chen Qi 
> Date:   Fri Mar 5 18:10:40 2021 +0800
> 
>     populate_sdk_ext: record METADATA_REVISION
>    
>     As we delete the .git/ directory, it's impossible to get METADATA_REVISION
>     inside eSDK. Because of this, we meet the following warning when 
> installing
> eSDK.
>    
>   WARNING: The base-files:do_install sig is computed to be
> 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
>   but the sig is locked to
> 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in
> SIGGEN_LOCKEDSIGS_t-qemux86-64
>    
>     So we record METADATA_REVISION in eSDK generation time to fix this 
> problem.
>    
>     Signed-off-by: Chen Qi 
>     Signed-off-by: Richard Purdie 
> """
> 
> Best Regards,
> Chen Qi
> 
> On 03/26/2021 07:51 AM, Mark Hatle wrote:
>> Building an eSDK (all stock config):
>>
>> MACHINE=qemuarm bitbake core-image-minimal -c populate_sdk_ext
>>
>>
>> I then install it, and get:
>>
>> Checking sstate mirror object availability: 100%
>> |#|
>> Time: 0:00:00
>> WARNING: The base-files:do_install sig is computed to be
>> 587ccfa9299e26094ae342f1ce3d782bfaf776c639013428a511770c3828d49a, but the 
>> sig is
>> locked to 0c35f266f1e79b1f34260df1b8d677f7eb74a7f7241155601ed73a014c8e286c in
>> SIGGEN_LOCKEDSIGS_t-qemuarm
>> Loading cache: 100%
>> |##|
>> Time: 0:00:00
>> Initialising tasks: 100%
>> |#|
>> Time: 0:00:00
>>
>>
>> Running a bitbake diffsigs on this file, I get:
>>
>> basehash changed from
>> cffec52acb67f62fe3812d0d42d52b1081f9150386f43f0426f5469b4a1a6a25 to
>> 332259212c5e9a4ed2a7fe25b8ea77cb54fb6b82f7deed90179e145f1d0cf1b6
>> Variable METADATA_REVISION value changed from
>> '3210cabe56d6cf6d51638e6f31f93110e40ab2cd' to ''
>>
>>
>> So it appears that some of the recent work on METADATA_REVISION has broken 
>> hash
>> computations in the base-files package due to the value being 'unknown' in 
>> one,
>> and set in the other.
>>
>> --Mark
>>
>>
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#150060): 
https://lists.openembedded.org/g/openembedded-core/message/150060
Mute This Topic: https://lists.openembedded.org/mt/81616842/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH] populate_sdk_ext: Avoid copying and producing .pyc files

2021-03-26 Thread Mark Hatle


On 3/26/21 12:54 AM, Alexander Kanavin wrote:
> Wait, I have to say neither reason is convincing. Pyc files are not
> system-specific, they are the same on all systems. And pseudo issues should be
> dealt with by fixing the issue, not working around the symptoms, no?

These are pyc files that are generated by the host system python.  As such, they
contain versioning information of the particular host they were constructed on.

The next user (of the eSDK) would regenerate the pyc files, assuming their
python was a different version/configuration anyway.

The pseudo issue is simply caused by the generation of these files outside of
the pseudo context.  The three places modified need to run without pseudo
control in order to setup the bitbake (host) environment properly, as well as
verify that the previous files copied/setup are infact correct.

So we're not working around symptoms.. we are indeed fixing the underlying
issues in the implementation.  It just happened to be that I was able to
reproduce the build failure reliably for a time vs it just going away on it's 
own.

--Mark

> Alex
> 
> On Thu, 25 Mar 2021 at 23:44, Mark Hatle  <mailto:mark.ha...@kernel.crashing.org>> wrote:
> 
> Since pyc cache files are really system specific, no real reason to copy 
> or
> generate them during the eSDK build process.  Also generating them has the
> possibility of re-using inodes that pseudo may have been tracking, leading
> a build failure.
> 
> Signed-off-by: Mark Hatle  <mailto:mark.ha...@kernel.crashing.org>>
> ---
>  meta/classes/populate_sdk_ext.bbclass | 4 +++-
>  meta/lib/oe/copy_buildsystem.py       | 6 +++---
>  2 files changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass
> index 9112ab6c5e..14689ec6ac 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -251,7 +251,9 @@ python copy_buildsystem () {
> 
>      # Create a layer for new recipes / appends
>      bbpath = d.getVar('BBPATH')
> -    bb.process.run(['devtool', '--bbpath', bbpath, '--basepath',
> baseoutpath, 'create-workspace', '--create-only', 
> os.path.join(baseoutpath,
> 'workspace')])
> +    env = os.environ.copy()
> +    env['PYTHONDONTWRITEBYTECODE'] = '1'
> +    bb.process.run(['devtool', '--bbpath', bbpath, '--basepath',
> baseoutpath, 'create-workspace', '--create-only', 
> os.path.join(baseoutpath,
> 'workspace')], env=env)
> 
>      # Create bblayers.conf
>      bb.utils.mkdirhier(baseoutpath + '/conf')
> diff --git a/meta/lib/oe/copy_buildsystem.py 
> b/meta/lib/oe/copy_buildsystem.py
> index 31a84f5b06..d97bf9d1b9 100644
> --- a/meta/lib/oe/copy_buildsystem.py
> +++ b/meta/lib/oe/copy_buildsystem.py
> @@ -20,7 +20,7 @@ def _smart_copy(src, dest):
>      mode = os.stat(src).st_mode
>      if stat.S_ISDIR(mode):
>          bb.utils.mkdirhier(dest)
> -        cmd = "tar --exclude='.git' --xattrs --xattrs-include='*' -chf - 
> -C
> %s -p . \
> +        cmd = "tar --exclude='.git' --exclude='__pycache__' --xattrs
> --xattrs-include='*' -chf - -C %s -p . \
>          | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dest)
>          subprocess.check_output(cmd, shell=True, 
> stderr=subprocess.STDOUT)
>      else:
> @@ -259,7 +259,7 @@ def create_locked_sstate_cache(lockedsigs,
> input_sstate_cache, output_sstate_cac
>      bb.note('Generating sstate-cache...')
> 
>      nativelsbstring = d.getVar('NATIVELSBSTRING')
> -    bb.process.run("gen-lockedsig-cache %s %s %s %s %s" % (lockedsigs,
> input_sstate_cache, output_sstate_cache, nativelsbstring, filterfile or 
> ''))
> +    bb.process.run("PYTHONDONTWRITEBYTECODE=1 gen-lockedsig-cache %s %s 
> %s
> %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache,
> nativelsbstring, filterfile or ''))
>      if fixedlsbstring and nativelsbstring != fixedlsbstring:
>          nativedir = output_sstate_cache + '/' + nativelsbstring
>          if os.path.isdir(nativedir):
> @@ -286,7 +286,7 @@ def check_sstate_task_list(d, targets, filteroutfile,
> cmdprefix='', cwd=None, lo
>          logparam = '-l %s' % logfile
>      else:
>          logparam = ''
> -    cmd = "%sBB_SETSCENE_ENFORCE=1 PSEUDO_DISABLED=1 oe-check-sstate %s 
> -s
> -o %s %s" % (cmdprefix, targets, filteroutfile, logparam)
> +    cmd = "%sPYTHONDONTWRITEBYTECODE=1 BB_SETSCENE_ENFORCE=1
> PSEUDO_DISABLED=1 oe-check-

[OE-core] [PATCH 0/1] Enable PR Service support in eSDK

2021-03-25 Thread Mark Hatle
This is the first attempt at enabling the PR service in the eSDK.  It appears
to work for me, but I expect I'm missing corner cases.

Any feedback is appreciatd.

Mark Hatle (1):
  populate_sdk_ext: Add support for PR service

 meta/classes/populate_sdk_ext.bbclass | 25 +
 meta/files/ext-sdk-prepare.py | 17 +
 2 files changed, 42 insertions(+)

-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149964): 
https://lists.openembedded.org/g/openembedded-core/message/149964
Mute This Topic: https://lists.openembedded.org/mt/81617272/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] [PATCH 1/1] populate_sdk_ext: Add support for PR service

2021-03-25 Thread Mark Hatle
From: Mark Hatle 

In the classes/populate_sdk_ext.bbclass the system already copies a number
of configurations, such as the hash equivalency data.  However, the PR
service was being handled.

The new code works by checking if PRSERV_HOST is defined, if it is, use
the existing export functions to write out a conf/prserv.inc file into
the eSDK.  On eSDK install, if a conf/prserv.inc file is present we then
import this file into the system.

This mechanism will work if the PRSERV_HOST is local or remote, as it pulls
the necessary data from the server and then imports it to a local database
on eSDK installation.

Note: the conf/prserv.inc file is not deleted at this time.  It was left
for possible debugging purposes, but removing it is something we could decide
to do in the future.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/populate_sdk_ext.bbclass | 25 +
 meta/files/ext-sdk-prepare.py | 17 +
 2 files changed, 42 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 14689ec6ac..84232ed9f5 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -375,6 +375,10 @@ python copy_buildsystem () {
 # Map gcc-dependent uninative sstate cache for installer usage
 f.write('SSTATE_MIRRORS += " file://universal/(.*) 
file://universal-4.9/\\1 file://universal-4.9/(.*) 
file://universal-4.8/\\1"\n\n')
 
+if d.getVar("PRSERV_HOST"):
+# Override this, we now include PR data, so it should only 
point ot the local database
+f.write('PRSERV_HOST = "localhost:0"\n\n')
+
 # Allow additional config through sdk-extra.conf
 fn = bb.cookerdata.findConfigFile('sdk-extra.conf', d)
 if fn:
@@ -398,6 +402,27 @@ python copy_buildsystem () {
 bb.utils.mkdirhier(os.path.join(baseoutpath, 'cache'))
 shutil.copyfile(builddir + '/cache/bb_unihashes.dat', baseoutpath + 
'/cache/bb_unihashes.dat')
 
+# If PR Service is in use, we need to export this as well
+bb.note('Do we have a pr database?')
+if d.getVar("PRSERV_HOST"):
+bb.note('Writing PR database...')
+# Based on the code in classes/prexport.bbclass
+import oe.prservice
+#dump meta info of tables
+localdata = d.createCopy()
+localdata.setVar('PRSERV_DUMPOPT_COL', "1")
+localdata.setVar('PRSERV_DUMPDIR', os.path.join(baseoutpath, 'conf'))
+localdata.setVar('PRSERV_DUMPFILE', '${PRSERV_DUMPDIR}/prserv.inc')
+
+bb.note('PR Database write to %s' % 
(localdata.getVar('PRSERV_DUMPFILE')))
+
+retval = oe.prservice.prserv_dump_db(localdata)
+if not retval:
+bb.error("prexport_handler: export failed!")
+return
+(metainfo, datainfo) = retval
+oe.prservice.prserv_export_tofile(localdata, metainfo, datainfo, True)
+
 # Use templateconf.cfg file from builddir if exists
 if os.path.exists(builddir + '/conf/templateconf.cfg') and 
use_custom_templateconf == '1':
 shutil.copyfile(builddir + '/conf/templateconf.cfg', baseoutpath + 
'/conf/templateconf.cfg')
diff --git a/meta/files/ext-sdk-prepare.py b/meta/files/ext-sdk-prepare.py
index 163d5e9912..d191e5e19c 100644
--- a/meta/files/ext-sdk-prepare.py
+++ b/meta/files/ext-sdk-prepare.py
@@ -44,6 +44,23 @@ def main():
 sdk_targets = []
 else:
 sdk_targets = ' '.join(sys.argv[2:]).split()
+
+prserv = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'conf', 
'prserv.inc')
+if os.path.isfile(prserv):
+with open(logfile, 'a') as logf:
+logf.write('Importing PR data...\n')
+
+ret = run_command_interruptible('bitbake-prserv-tool import %s' % 
prserv)
+
+lastlog = get_last_consolelog()
+if lastlog:
+with open(lastlog, 'r') as f:
+for line in f:
+logf.write(line)
+if ret:
+print('ERROR: PR data import failed: error log written to %s' 
% logfile)
+return ret
+
 if not sdk_targets:
 # Just do a parse so the cache is primed
 ret = run_command_interruptible('bitbake -p --quiet')
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149963): 
https://lists.openembedded.org/g/openembedded-core/message/149963
Mute This Topic: https://lists.openembedded.org/mt/81617271/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] master eSDK hash problem with METADATA_REVISION and base-files:do_install

2021-03-25 Thread Mark Hatle
Building an eSDK (all stock config):

MACHINE=qemuarm bitbake core-image-minimal -c populate_sdk_ext


I then install it, and get:

Checking sstate mirror object availability: 100%
|#|
Time: 0:00:00
WARNING: The base-files:do_install sig is computed to be
587ccfa9299e26094ae342f1ce3d782bfaf776c639013428a511770c3828d49a, but the sig is
locked to 0c35f266f1e79b1f34260df1b8d677f7eb74a7f7241155601ed73a014c8e286c in
SIGGEN_LOCKEDSIGS_t-qemuarm
Loading cache: 100%
|##|
Time: 0:00:00
Initialising tasks: 100%
|#|
Time: 0:00:00


Running a bitbake diffsigs on this file, I get:

basehash changed from
cffec52acb67f62fe3812d0d42d52b1081f9150386f43f0426f5469b4a1a6a25 to
332259212c5e9a4ed2a7fe25b8ea77cb54fb6b82f7deed90179e145f1d0cf1b6
Variable METADATA_REVISION value changed from
'3210cabe56d6cf6d51638e6f31f93110e40ab2cd' to ''


So it appears that some of the recent work on METADATA_REVISION has broken hash
computations in the base-files package due to the value being 'unknown' in one,
and set in the other.

--Mark

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149961): 
https://lists.openembedded.org/g/openembedded-core/message/149961
Mute This Topic: https://lists.openembedded.org/mt/81616842/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] [PATCH] populate_sdk_ext: Avoid copying and producing .pyc files

2021-03-25 Thread Mark Hatle
Since pyc cache files are really system specific, no real reason to copy or
generate them during the eSDK build process.  Also generating them has the
possibility of re-using inodes that pseudo may have been tracking, leading
a build failure.

Signed-off-by: Mark Hatle 
---
 meta/classes/populate_sdk_ext.bbclass | 4 +++-
 meta/lib/oe/copy_buildsystem.py   | 6 +++---
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 9112ab6c5e..14689ec6ac 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -251,7 +251,9 @@ python copy_buildsystem () {
 
 # Create a layer for new recipes / appends
 bbpath = d.getVar('BBPATH')
-bb.process.run(['devtool', '--bbpath', bbpath, '--basepath', baseoutpath, 
'create-workspace', '--create-only', os.path.join(baseoutpath, 'workspace')])
+env = os.environ.copy()
+env['PYTHONDONTWRITEBYTECODE'] = '1'
+bb.process.run(['devtool', '--bbpath', bbpath, '--basepath', baseoutpath, 
'create-workspace', '--create-only', os.path.join(baseoutpath, 'workspace')], 
env=env)
 
 # Create bblayers.conf
 bb.utils.mkdirhier(baseoutpath + '/conf')
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 31a84f5b06..d97bf9d1b9 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -20,7 +20,7 @@ def _smart_copy(src, dest):
 mode = os.stat(src).st_mode
 if stat.S_ISDIR(mode):
 bb.utils.mkdirhier(dest)
-cmd = "tar --exclude='.git' --xattrs --xattrs-include='*' -chf - -C %s 
-p . \
+cmd = "tar --exclude='.git' --exclude='__pycache__' --xattrs 
--xattrs-include='*' -chf - -C %s -p . \
 | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dest)
 subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
 else:
@@ -259,7 +259,7 @@ def create_locked_sstate_cache(lockedsigs, 
input_sstate_cache, output_sstate_cac
 bb.note('Generating sstate-cache...')
 
 nativelsbstring = d.getVar('NATIVELSBSTRING')
-bb.process.run("gen-lockedsig-cache %s %s %s %s %s" % (lockedsigs, 
input_sstate_cache, output_sstate_cache, nativelsbstring, filterfile or ''))
+bb.process.run("PYTHONDONTWRITEBYTECODE=1 gen-lockedsig-cache %s %s %s %s 
%s" % (lockedsigs, input_sstate_cache, output_sstate_cache, nativelsbstring, 
filterfile or ''))
 if fixedlsbstring and nativelsbstring != fixedlsbstring:
 nativedir = output_sstate_cache + '/' + nativelsbstring
 if os.path.isdir(nativedir):
@@ -286,7 +286,7 @@ def check_sstate_task_list(d, targets, filteroutfile, 
cmdprefix='', cwd=None, lo
 logparam = '-l %s' % logfile
 else:
 logparam = ''
-cmd = "%sBB_SETSCENE_ENFORCE=1 PSEUDO_DISABLED=1 oe-check-sstate %s -s -o 
%s %s" % (cmdprefix, targets, filteroutfile, logparam)
+cmd = "%sPYTHONDONTWRITEBYTECODE=1 BB_SETSCENE_ENFORCE=1 PSEUDO_DISABLED=1 
oe-check-sstate %s -s -o %s %s" % (cmdprefix, targets, filteroutfile, logparam)
 env = dict(d.getVar('BB_ORIGENV', False))
 env.pop('BUILDDIR', '')
 env.pop('BBPATH', '')
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149960): 
https://lists.openembedded.org/g/openembedded-core/message/149960
Mute This Topic: https://lists.openembedded.org/mt/81615695/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] any *compelling* reasons to whitelist env vars for an OE build?

2021-03-25 Thread Mark Hatle


On 3/25/21 9:08 AM, Robert P. J. Day wrote:
> 
>   kind of a philosophical question, but i'm having a discussion with
> some new colleagues about how to customize an OE (actually, wind river
> linux) build, and these colleagues have, until now, used (whitelisted)
> environment variables to pass info to the build, that info being used
> to specify things like a development versus manufacturing build, and
> so on.
> 
>   i suggested that i didn't think there was any really persuasive
> reasons to use environment variables this way, as there are more than
> enough configuration files to customize a build. i mentioned things
> like:
> 
>   * machine config files
>   * distro config files
>   * defining packagegroup files
>   * defining image files
> 
> and on and on and on.
> 
>   my point was that there are more than enough ways to support all the
> customization you might need, that taking advantage of shell
> environment variables strikes me as unnecessarily messy and, for that
> matter, dangerous if you forget so set the environment.
> 
>   thoughts? am i out of line in advising them to use what OE offers
> them, and not extract stuff from the environment?

I _always_ do builds like:

MACHINE=zynqmp-generic DISTRO=petalinux bitbake core-image-minimal

I don't want to have to update any configuration files to pass basic values into
the system.  This is also used heavily on my build systems so we can have a
common configuration for all builds, but built across a wide breadth of 
components.

I also occasionally use BB_ENV_EXTRAWHITE to add additional variables I might
want to override.  For instance, sometimes I want TMPDIR to go to a different
location, instead of having to modify local.conf (and then remember to put it
back the way it was) I can just add to the EXTRAWHITE, and call passing in 
TMPDIR.

--Mark

> rday
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149928): 
https://lists.openembedded.org/g/openembedded-core/message/149928
Mute This Topic: https://lists.openembedded.org/mt/81603196/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] PR Service and eSDK

2021-03-23 Thread Mark Hatle


On 3/23/21 5:45 PM, Richard Purdie wrote:
> On Tue, 2021-03-23 at 17:34 -0500, Mark Hatle wrote:
>> For some reason I thought if PR service was enabled, when the eSDK was 
>> generated
>> it would export the pr service information and include it within the eSDK.
>>
>> I'm not finding the file, or even code that does this.  Am I having a fever
>> dream, or is there code that should be doing this?
>>
>> What I'm trying to build is an eSDK that the user can use devtool and 
>> construct
>> a new version of a package, then they can deploy the new package within their
>> package feed to update their targets.  Without the PR service in the eSDK, 
>> there
>> won't be any way to get a correct PR number.
> 
> I know we copy bb_unihashes.dat into the eSDK with code in 
> populate_sdl_ext.bbclass
> for this reason for hashequiv.
> 
> We do have prserv export and import code but I'm not sure anyone has 
> integrated it
> with eSDK, or if there is integration, I can't spot it atm. I wondered if it 
> copied
> the prserv database from the build directory over to assist with this issue 
> but
> again, I can't spot it.

Ok, this confirms what I am seeing then.  Maybe it was discussed and was never
implemented.  I'll look at where the unihashes.dat is copied over and look to do
the export... and then import in the setup script.

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149851): 
https://lists.openembedded.org/g/openembedded-core/message/149851
Mute This Topic: https://lists.openembedded.org/mt/81563805/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] PR Service and eSDK

2021-03-23 Thread Mark Hatle
For some reason I thought if PR service was enabled, when the eSDK was generated
it would export the pr service information and include it within the eSDK.

I'm not finding the file, or even code that does this.  Am I having a fever
dream, or is there code that should be doing this?

What I'm trying to build is an eSDK that the user can use devtool and construct
a new version of a package, then they can deploy the new package within their
package feed to update their targets.  Without the PR service in the eSDK, there
won't be any way to get a correct PR number.

--Mark

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149849): 
https://lists.openembedded.org/g/openembedded-core/message/149849
Mute This Topic: https://lists.openembedded.org/mt/81563805/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH 1/1] extrausers: Add ability to force password change on first login

2021-03-09 Thread Mark Hatle
On 3/8/21 8:02 PM, Chen Qi wrote:
> Hi Mark,
> 
> Is it something similar to 'passwd-expire' in this extrausers.bbclass?

I wasn't aware of that evening existing.  Yes it looks like it does the same 
thing.

I can withdraw my change then, but we may want to considering adding something
to the documentation about security practices.  For accounts that are created by
the build system, it's best practices to either not make them able to be logged
in with (login locked out '-P *' on the adduser) or force the password to be
reset on next login (using passwd-expire).

--Mark

> Best Regards,
> Chen Qi
> 
> On 03/09/2021 02:08 AM, Mark Hatle wrote:
>> As documented in shadow(5), the third parameter is the last login time.  A
>> special value of '0' is defined which causes the password system to force
>> a password change on next login.
>>
>> Adding the variable "EXTRA_FORCE_PASSWORD_CHANGE", a space separated list of
>> user names, we can use this to adjust the shadow file's third value for the
>> listed users.
>>
>> Note: This does have the same dependencies as other usages of extrausers,
>> specifically base-passwd and shadow.
>>
>> Signed-off-by: Mark Hatle 
>> Signed-off-by: Mark Hatle 
>> ---
>>  meta/classes/extrausers.bbclass | 29 +++--
>>  meta/conf/documentation.conf|  1 +
>>  2 files changed, 28 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/extrausers.bbclass 
>> b/meta/classes/extrausers.bbclass
>> index 90811bfe2a..e9d9358bef 100644
>> --- a/meta/classes/extrausers.bbclass
>> +++ b/meta/classes/extrausers.bbclass
>> @@ -14,10 +14,10 @@
>>  
>>  inherit useradd_base
>>  
>> -PACKAGE_INSTALL_append = " ${@['', 'base-passwd 
>> shadow'][bool(d.getVar('EXTRA_USERS_PARAMS'))]}"
>> +PACKAGE_INSTALL_append = " ${@['', 'base-passwd 
>> shadow'][bool(d.getVar('EXTRA_USERS_PARAMS')) or 
>> bool(d.getVar('EXTRA_FORCE_PASSWORD_CHANGE'))]}"
>>  
>>  # Image level user / group settings
>> -ROOTFS_POSTPROCESS_COMMAND_append = " set_user_group;"
>> +ROOTFS_POSTPROCESS_COMMAND_append = "${@['', ' 
>> set_user_group;'][bool(d.getVar('EXTRA_USERS_PARAMS'))]}"
>>  
>>  # Image level user / group settings
>>  set_user_group () {
>> @@ -66,6 +66,31 @@ set_user_group () {
>>  done
>>  }
>>  
>> +# Image level force a specific user/users to reset their password on first 
>> login
>> +# Note: this requires shadow passwords and login programs that respect the 
>> shadow
>> +# expiration field.
>> +ROOTFS_POSTPROCESS_COMMAND_append = "${@['', '
>> force_password_change;'][bool(d.getVar('EXTRA_FORCE_PASSWORD_CHANGE'))]}"
>> +
>> +# Works by setting 'date of last password change' to 0, which has a special
>> +# meaning of 'user should change her password the next time she will log in 
>> the
>> +# system' See: shadow (5)
>> +force_password_change () {
>> +if [ ! -e ${IMAGE_ROOTFS}/etc/shadow ]; then
>> +bberror "/etc/shadow does not exist in the image, unable to set 
>> password change on login."
>> +return
>> +fi
>> +passwd_change_users="${EXTRA_FORCE_PASSWORD_CHANGE}"
>> +export PSEUDO="${FAKEROOTENV} ${STAGING_DIR_NATIVE}${bindir}/pseudo"
>> +for name in $passwd_change_users; do
>> +if ! grep -q '^'$name':' ${IMAGE_ROOTFS}/etc/shadow ; then
>> +bberror "Unable to find user $name in /etc/shadow, 
>> unable to set password change on login."
>> +fi
>> +bbnote "Set user $name to need a password change on first 
>> login."
>> +cmd="sed -i ${IMAGE_ROOTFS}/etc/shadow -e 
>> 's,^'$name':\\([^:]*\\):[^:]*:,'$name':\\1:0:,'"
>> +eval flock -x ${IMAGE_ROOTFS}${sysconfdir} -c \"$PSEUDO $cmd\" 
>> || true
>> +done
>> +}
>> +
>>  USERADDEXTENSION ?= ""
>>  
>>  inherit ${USERADDEXTENSION}
>> diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
>> index c5a38b0764..d1c5b8b1a3 100644
>> --- a/meta/conf/documentation.conf
>> +++ b/meta/conf/documentation.conf
>> @@ -169,6 +169,7 @@ EXTRA_OESCONS[doc] = "When a recipe inherits the scons 
>> class, this variable spec
>>  EXTRA_QMAKEVARS_POST[doc] = "Configuration variables or options you want to 
>> pass to qmake when the arguments need to be after the .pro file list on the 
>> command line."
&

Re: [OE-core] [PATCH 1/1] extrausers: Add ability to force password change on first login

2021-03-08 Thread Mark Hatle


On 3/8/21 12:50 PM, Khem Raj wrote:
> 
> 
> On 3/8/21 10:08 AM, Mark Hatle wrote:
>> From: Mark Hatle 
>>
>> As documented in shadow(5), the third parameter is the last login time.  A
>> special value of '0' is defined which causes the password system to force
>> a password change on next login.
>>
>> Adding the variable "EXTRA_FORCE_PASSWORD_CHANGE", a space separated list of
>> user names, we can use this to adjust the shadow file's third value for the
>> listed users.
>>
>> Note: This does have the same dependencies as other usages of extrausers,
>> specifically base-passwd and shadow.
>>
> 
> I think it should check for r/w rootfs feature perhaps. unrelated to 

Is there a standard way to check for a r/w roots?  If there is, easy to add.

> this change but it seems it adds a dep on shadow disregarding DISTRO 
> policies where user might have chosen a different login managager, it 
> should perhaps warn about it.

The dep on shadow is the same as any extrauser call.  The dependency sets the
minimum login manager, but any login manager that supports proper shadow
password handling will work.  If it doesn't support shadow password handling
then nothing breaks -- it just won't do anything.  (Really nothing here that can
be enforced in this code block.)

util-linux login + pam for instance used to work.  (I've not tested it though in
a few years.)

--Mark

>> Signed-off-by: Mark Hatle 
>> Signed-off-by: Mark Hatle 
>> ---
>>   meta/classes/extrausers.bbclass | 29 +++--
>>   meta/conf/documentation.conf|  1 +
>>   2 files changed, 28 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/extrausers.bbclass 
>> b/meta/classes/extrausers.bbclass
>> index 90811bfe2a..e9d9358bef 100644
>> --- a/meta/classes/extrausers.bbclass
>> +++ b/meta/classes/extrausers.bbclass
>> @@ -14,10 +14,10 @@
>>   
>>   inherit useradd_base
>>   
>> -PACKAGE_INSTALL_append = " ${@['', 'base-passwd 
>> shadow'][bool(d.getVar('EXTRA_USERS_PARAMS'))]}"
>> +PACKAGE_INSTALL_append = " ${@['', 'base-passwd 
>> shadow'][bool(d.getVar('EXTRA_USERS_PARAMS')) or 
>> bool(d.getVar('EXTRA_FORCE_PASSWORD_CHANGE'))]}"
>>   
>>   # Image level user / group settings
>> -ROOTFS_POSTPROCESS_COMMAND_append = " set_user_group;"
>> +ROOTFS_POSTPROCESS_COMMAND_append = "${@['', ' 
>> set_user_group;'][bool(d.getVar('EXTRA_USERS_PARAMS'))]}"
>>   
>>   # Image level user / group settings
>>   set_user_group () {
>> @@ -66,6 +66,31 @@ set_user_group () {
>>  done
>>   }
>>   
>> +# Image level force a specific user/users to reset their password on first 
>> login
>> +# Note: this requires shadow passwords and login programs that respect the 
>> shadow
>> +# expiration field.
>> +ROOTFS_POSTPROCESS_COMMAND_append = "${@['', ' 
>> force_password_change;'][bool(d.getVar('EXTRA_FORCE_PASSWORD_CHANGE'))]}"
>> +
>> +# Works by setting 'date of last password change' to 0, which has a special
>> +# meaning of 'user should change her password the next time she will log in 
>> the
>> +# system' See: shadow (5)
>> +force_password_change () {
>> +if [ ! -e ${IMAGE_ROOTFS}/etc/shadow ]; then
>> +bberror "/etc/shadow does not exist in the image, unable to set 
>> password change on login."
>> +return
>> +fi
>> +passwd_change_users="${EXTRA_FORCE_PASSWORD_CHANGE}"
>> +export PSEUDO="${FAKEROOTENV} ${STAGING_DIR_NATIVE}${bindir}/pseudo"
>> +for name in $passwd_change_users; do
>> +if ! grep -q '^'$name':' ${IMAGE_ROOTFS}/etc/shadow ; then
>> +bberror "Unable to find user $name in /etc/shadow, 
>> unable to set password change on login."
>> +fi
>> +bbnote "Set user $name to need a password change on first 
>> login."
>> +cmd="sed -i ${IMAGE_ROOTFS}/etc/shadow -e 
>> 's,^'$name':\\([^:]*\\):[^:]*:,'$name':\\1:0:,'"
>> +eval flock -x ${IMAGE_ROOTFS}${sysconfdir} -c \"$PSEUDO $cmd\" 
>> || true
>> +done
>> +}
>> +
>>   USERADDEXTENSION ?= ""
>>   
>>   inherit ${USERADDEXTENSION}
>> diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
>> index c5a38b0764..d1c5b8b1a3 100644
>> --- a/meta/conf/documentation.conf
>> +++ b/meta/conf/documentation.conf
>> @@ -169,6 +169,7 @@ EXTRA_OESCONS[doc] = "When a rec

[OE-core] [PATCH 0/1] Enable the ability to force a password change on boot

2021-03-08 Thread Mark Hatle
As noted in the commit message, the shadow(5) indicates that the third
parameter of the /etc/shadow file, when set to 0, can be used to force
a password change on login.  Note, a login program that supports this
behavior is required.

It was added to extrausers.bbclass as it has the same dependencies as
the other components of extrausers and should often be used in with
adding/creating new accounts.

This was verified by adding the following to the conf/local.conf:

INHERIT += "extrausers"

EXTRA_FORCE_PASSWORD_CHANGE_append = " root"

$ bitbake core-image-minimal
$ runqemu

Login as root, and it should prompt for a password change.

This was further verified by setting a default root password, as well
as adding a new user to the system.  In both cases it worked as expected.

Finally adding an invalid user to the list, and an appropriate error is
generated.


Mark Hatle (1):
  extrausers: Add ability to force password change on first login

 meta/classes/extrausers.bbclass | 29 +++--
 meta/conf/documentation.conf|  1 +
 2 files changed, 28 insertions(+), 2 deletions(-)

-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149114): 
https://lists.openembedded.org/g/openembedded-core/message/149114
Mute This Topic: https://lists.openembedded.org/mt/81180919/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] [PATCH 1/1] extrausers: Add ability to force password change on first login

2021-03-08 Thread Mark Hatle
From: Mark Hatle 

As documented in shadow(5), the third parameter is the last login time.  A
special value of '0' is defined which causes the password system to force
a password change on next login.

Adding the variable "EXTRA_FORCE_PASSWORD_CHANGE", a space separated list of
user names, we can use this to adjust the shadow file's third value for the
listed users.

Note: This does have the same dependencies as other usages of extrausers,
specifically base-passwd and shadow.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/extrausers.bbclass | 29 +++--
 meta/conf/documentation.conf|  1 +
 2 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/meta/classes/extrausers.bbclass b/meta/classes/extrausers.bbclass
index 90811bfe2a..e9d9358bef 100644
--- a/meta/classes/extrausers.bbclass
+++ b/meta/classes/extrausers.bbclass
@@ -14,10 +14,10 @@
 
 inherit useradd_base
 
-PACKAGE_INSTALL_append = " ${@['', 'base-passwd 
shadow'][bool(d.getVar('EXTRA_USERS_PARAMS'))]}"
+PACKAGE_INSTALL_append = " ${@['', 'base-passwd 
shadow'][bool(d.getVar('EXTRA_USERS_PARAMS')) or 
bool(d.getVar('EXTRA_FORCE_PASSWORD_CHANGE'))]}"
 
 # Image level user / group settings
-ROOTFS_POSTPROCESS_COMMAND_append = " set_user_group;"
+ROOTFS_POSTPROCESS_COMMAND_append = "${@['', ' 
set_user_group;'][bool(d.getVar('EXTRA_USERS_PARAMS'))]}"
 
 # Image level user / group settings
 set_user_group () {
@@ -66,6 +66,31 @@ set_user_group () {
done
 }
 
+# Image level force a specific user/users to reset their password on first 
login
+# Note: this requires shadow passwords and login programs that respect the 
shadow
+# expiration field.
+ROOTFS_POSTPROCESS_COMMAND_append = "${@['', ' 
force_password_change;'][bool(d.getVar('EXTRA_FORCE_PASSWORD_CHANGE'))]}"
+
+# Works by setting 'date of last password change' to 0, which has a special
+# meaning of 'user should change her password the next time she will log in the
+# system' See: shadow (5)
+force_password_change () {
+   if [ ! -e ${IMAGE_ROOTFS}/etc/shadow ]; then
+   bberror "/etc/shadow does not exist in the image, unable to set 
password change on login."
+   return
+   fi
+   passwd_change_users="${EXTRA_FORCE_PASSWORD_CHANGE}"
+   export PSEUDO="${FAKEROOTENV} ${STAGING_DIR_NATIVE}${bindir}/pseudo"
+   for name in $passwd_change_users; do
+   if ! grep -q '^'$name':' ${IMAGE_ROOTFS}/etc/shadow ; then
+   bberror "Unable to find user $name in /etc/shadow, 
unable to set password change on login."
+   fi
+   bbnote "Set user $name to need a password change on first 
login."
+   cmd="sed -i ${IMAGE_ROOTFS}/etc/shadow -e 
's,^'$name':\\([^:]*\\):[^:]*:,'$name':\\1:0:,'"
+   eval flock -x ${IMAGE_ROOTFS}${sysconfdir} -c \"$PSEUDO $cmd\" 
|| true
+   done
+}
+
 USERADDEXTENSION ?= ""
 
 inherit ${USERADDEXTENSION}
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index c5a38b0764..d1c5b8b1a3 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -169,6 +169,7 @@ EXTRA_OESCONS[doc] = "When a recipe inherits the scons 
class, this variable spec
 EXTRA_QMAKEVARS_POST[doc] = "Configuration variables or options you want to 
pass to qmake when the arguments need to be after the .pro file list on the 
command line."
 EXTRA_QMAKEVARS_PRE[doc] = "Configuration variables or options you want to 
pass to qmake when the arguments need to be before the .pro file list on the 
command line."
 EXTRA_USERS_PARAMS[doc] = "When a recipe inherits the extrausers class, this 
variable provides image level user and group operations."
+EXTRA_FORCE_PASSWORD_CHANGE[doc] = "When a recipe inherits the extrausers 
class, this variable causes the specified users to require a password change on 
first login."
 
 #F
 
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#149115): 
https://lists.openembedded.org/g/openembedded-core/message/149115
Mute This Topic: https://lists.openembedded.org/mt/81180920/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



[OE-core] Sudo CVE-2021-3156 -- was Re: [yocto-security] OE-core CVE metrics for master on Sun 31 Jan 2021 07:15:01 AM HST

2021-02-05 Thread Mark Hatle
I didn't see Sudo issue CVE-2021-3156 in any of the unpatched lists.

>From a quick look, it appears to be that Master is patched (package is new
enough), but Gatesgarth and older are not.

So with the next set, we should check if it shows up in the unpatched set.

--Mark

On 1/31/21 11:18 AM, Steve Sakoman wrote:
> Branch: master
> 
> New this week:
> 
> Removed this week:
> 
> Full list:  Found 59 unpatched CVEs

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#147711): 
https://lists.openembedded.org/g/openembedded-core/message/147711
Mute This Topic: https://lists.openembedded.org/mt/80414861/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [yocto-security] [PATCH] openssl: drop support for deprecated algorithms

2020-12-21 Thread Mark Hatle


On 12/19/20 12:04 PM, Konrad Weihmann wrote:
> 
> 
> On 19.12.20 18:58, Richard Purdie wrote:
>> On Sat, 2020-12-19 at 18:53 +0100, Konrad Weihmann wrote:
>>> On 19.12.20 18:36, Richard Purdie wrote:
PACKAGECONFIG[cryptodev-linux] = 
 "enable-devcryptoeng,disable-devcryptoeng,cryptodev-linux,,cryptodev-module"
 +PACKAGECONFIG[no-tls1] = "no-tls1"
 +PACKAGECONFIG[no-tls1_1] = "no-tls1_1"

B = "${WORKDIR}/build"
do_configure[cleandirs] = "${B}"
 @@ -52,6 +54,10 @@ EXTRA_OECONF_class-nativesdk = 
 "--with-rand-seed=os,devrandom"
CFLAGS_append_class-native = " -DOPENSSLDIR=/not/builtin 
 -DENGINESDIR=/not/builtin"
CFLAGS_append_class-nativesdk = " -DOPENSSLDIR=/not/builtin 
 -DENGINESDIR=/not/builtin"

 +# Disable deprecated crypto algorithms
 +# Retained for compatibilty - des (curl), dh (python-ssl), dsa (rpm)
 +DEPRECATED_CRYPTO_FLAGS = " no-ssl no-idea no-psk no-rc2 no-rc4 no-rc5 
 no-md2 no-md4 no-srp no-camellia no-bf no-mdc2 no-scrypt no-seed 
 no-siphash no-sm2 no-sm3 no-sm4 no-whirlpool"
 +
>>>   From my perspective this breaks backward compatibility, so I would
>>> rather have them all that as optional PACKAGECONFIG fields (which also
>>> does make it easier for ppl, still relying on one of those algorithms,
>>> for whatever reason, to re-enable them) - with the current approach all
>>> one could do is to override it with a bbappend - and tbh letting ppl
>>> have bbappends for this recipe, doesn't sound like the best idea in the
>>> long run to "enforce" any kind of "security" or "hardening"
>>
>> Having it as a variable does mean you could customise the variable and
>> doesn't mean it has to be done with a bbappend, it can be set from a
>> distro config too.
>>
>> I'm not sure turning each one into a packageconfig is going to be more
>> helpful compared to this in practise...
> 
> I'm not sure I follow, as this is a "hard" assign - if it would (in 
> theory) a ??= assignment, yes then it would be fine. Still that leaves 
> us with a not commonly known variable, while PACKAGECONFIG is more 
> widely accepted in 3rd party layers/distros from my experience.

In the past I had done something similar w/ PACKAGECONFIG, and then had a
comment in the recipe that indicated that certain items were not recommended
with a link to the OpenSSL documentation where it explained that.

It might also be a reasonable idea (security wise, maybe not OE wise) to display
a bb.warn if one of the 'unsafe' crypto algorithms is enabled.

As for someone mentioning that it's unclear if we should be maintaining a list,
in the past there was a list in the OpenSSL documentation.  I would certainly
rely on that list as what should and should not be enabled by default (with
exceptions as appropriate.)

--Mark

>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>> 
>>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#146031): 
https://lists.openembedded.org/g/openembedded-core/message/146031
Mute This Topic: https://lists.openembedded.org/mt/79087117/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] How to create a directory in multiple packages?

2020-12-15 Thread Mark Hatle
On 12/15/20 8:24 AM, Peter Kjellerstedt wrote:
>> -Original Message-
>> From: openembedded-core@lists.openembedded.org > c...@lists.openembedded.org> On Behalf Of Mark Hatle
>> Sent: den 15 december 2020 02:02
>> To: openembedded-core@lists.openembedded.org
>> Subject: Re: [OE-core] How to create a directory in multiple packages?
>>
>> On 12/14/20 11:43 AM, Peter Kjellerstedt wrote:
>>> Say we have a recipe that creates an empty /etc/foo directory. Now we
>>> want to add a new file in that directory /etc/foo/bar and package it as
>>> ${PN}-bar. This means the creation of the /etc/foo directory is moved
>>> from the ${PN} package to the ${PN}-bar package. Is there any way to
>>> make ${PN} continue to create an empty /etc/foo, or is the only
>>> alternative to introduce an /etc/foo/.dummy and package it in ${PN}?
>>
>> try adjust the order of the PACKAGES variant.  Something like:
>>
>> PACKAGES = "${PN}-bar ... ${PN}"
>>
>> FILES_${PN}-bar = "/etc/foo/bar"
>> FILES_${PN} = "/etc/foo"
>>
>> That SHOULD package the file 'bar' in -bar, and the directory in ${PN}.
> 
> Unfortunately that does not work (seems bitbake is too smart). What I have 
> is basically:
> 
> PACKAGE_BEFORE_PN = "${PN}-bar"
> FILES_${PN} += "${sysconfdir}/foo"
> FILES_${PN}-bar = "${sysconfdir}/foo/bar"
> 
> which results in the following in the spec file:
> 
> %files -n foo-bar
> %defattr(-,-,-,-)
> %dir "/etc"
> %dir "/etc/foo"
> "/etc/foo/bar"
> 
> There is nothing else about /etc or /etc/foo in the spec file.

Ya, there definitely should be a way to do this...  If not, it will be
problematic as more or more recipes get split into smaller chunks.

> After delving into the code for at bit, I believe this is due to how 
> populate_packages() works. When it handles the ${PN}-bar package and 
> finds the /etc/foo/bar file, it will also mark /etc and /etc/foo as  
> seen, which means they will not be added to any other package, unless 
> that package has an entry for some other /etc/foo/something file, even 
> if /etc/foo is explicitly listed in, e.g., FILES_${PN}.
> 
> So for now, it seems we will have to resort to using a pkg_postinst_${PN}.

Big problem with pkg_postinst, the system won't be able to track
perms/owner/group for the directory and ensure they are synced like the package
manager itself can.

For RPM, I know the %dir directive can (and should be) used to do this kind of
thing, but I don't know what the equivalencies are in opkg and deb.  If there is
a similar mechanism there to specify a directory (not not it's contents) then I
suspect may want to extend recipe syntax to match.

--Mark

>> (Some of the people commenting this isn't less then optimal, actually
>> it's not that unusual.. Typical case I see is creating a directory in 
>> /var or /usr/share for datafiles and the main package owns the directory, 
>> with sub-packages [or other packages] contributing data into that 
>> directory.)
> 
> Yes, there are reason why the empty directory needs to be installed as 
> part of the image. Otherwise I would just have used a file in 
> "/usr/lib/tmpfiles.d".
> 
>> --Mark
>>
>>> //Peter
> 
> //Peter
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#145758): 
https://lists.openembedded.org/g/openembedded-core/message/145758
Mute This Topic: https://lists.openembedded.org/mt/78956546/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] How to create a directory in multiple packages?

2020-12-14 Thread Mark Hatle


On 12/14/20 11:43 AM, Peter Kjellerstedt wrote:
> Say we have a recipe that creates an empty /etc/foo directory. Now we 
> want to add a new file in that directory /etc/foo/bar and package it as 
> ${PN}-bar. This means the creation of the /etc/foo directory is moved 
> from the ${PN} package to the ${PN}-bar package. Is there any way to 
> make ${PN} continue to create an empty /etc/foo, or is the only 
> alternative to introduce an /etc/foo/.dummy and package it in ${PN}?

try adjust the order of the PACKAGES variant.  Something like:

PACKAGES = "${PN}-bar ... ${PN}"

FILES_${PN}-bar = "/etc/foo/bar"
FILES_${PN} = "/etc/foo"

That SHOULD package the file 'bar' in -bar, and the directory in ${PN}.

(Some of the people commenting this isn't less then optimal, actually it's not
that unusual.. Typical case I see is creating a directory in /var or /usr/share
for datafiles and the main package owns the directory, with sub-packages [or
other packages] contributing data into that directory.)

--Mark

> //Peter
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#145629): 
https://lists.openembedded.org/g/openembedded-core/message/145629
Mute This Topic: https://lists.openembedded.org/mt/78956546/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH] file /etc/ethertypes conflicts between netbase-1:6.2-r0.corei7_64 and ebtables-2.0.10+4-r4.corei7_64

2020-12-14 Thread Mark Hatle
The version in netbase is the correct one.

For a comparison, see the followng for the netbase version:

https://salsa.debian.org/md/netbase/-/blob/master/etc/ethertypes

and see the following for the ebtables version:

http://git.netfilter.org/ebtables/tree/ethertypes?h=ebtables-2.0.10-4

So the correct fix for this would be to remove ethertypes from the ebtables
recipe in meta-oe.

Additionally, it looks like there is a MUCH newer version of ebtables.  (2.0.10
was released 6 year ago, while 2.0.11 was released about 1 year ago.)

(Even 2.0.11 has an older version of the ethertypes file then what netbase has.)

--Mark

On 12/14/20 4:55 AM, Outback Dingo wrote:
> On Mon, Dec 14, 2020 at 5:39 PM Alexander Kanavin
>  wrote:
>>
>> Sorry, but you need to explain why the file from from ebtables (an optional 
>> package from a 3rd party layer) takes precedence over file from netbase 
>> (which is a core item).
> 
> further to the previous note, it conflicts with ebtables in meta-oe
> from what i can tell
> 
> commit c71a08cea8477f69ee3bc511f1f14d99e09c0a49
> Author: Paul Eggleton 
> Date:   Mon Dec 3 15:30:40 2012 +
> 
> ebtables: add from OE-Classic, update and tidy up
> 
> * Update to 2.0.10-4
> * Handle hardcoded paths in initscript
> * Add LIC_FILES_CHKSUM
> * Set SUMMARY (which sets DESCRIPTION)
> * Drop PRIORITY
> * Minor formatting/ordering tweaks
> 
> Based on a patch by Vladimir Redzhepoff 
> 
> Signed-off-by: Paul Eggleton 
> 
> 
>>
>> Alex
>>
>> On Mon, 14 Dec 2020 at 11:33, Outback Dingo  wrote:
>>>
>>> ---
>>>  meta/recipes-core/netbase/netbase_6.2.bb | 1 -
>>>  1 file changed, 1 deletion(-)
>>>
>>> diff --git a/meta/recipes-core/netbase/netbase_6.2.bb 
>>> b/meta/recipes-core/netbase/netbase_6.2.bb
>>> index a54d2e7764..262b2cf1bc 100644
>>> --- a/meta/recipes-core/netbase/netbase_6.2.bb
>>> +++ b/meta/recipes-core/netbase/netbase_6.2.bb
>>> @@ -19,5 +19,4 @@ do_install () {
>>> install -m 0644 ${S}/etc/rpc ${D}${sysconfdir}/rpc
>>> install -m 0644 ${S}/etc/protocols ${D}${sysconfdir}/protocols
>>> install -m 0644 ${S}/etc/services ${D}${sysconfdir}/services
>>> -   install -m 0644 ${S}/etc/ethertypes ${D}${sysconfdir}/ethertypes
>>>  }
>>> --
>>> 2.20.1
>>>
>>>
>>>
>>>
>>>
>>>
>>> 
>>>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#145551): 
https://lists.openembedded.org/g/openembedded-core/message/145551
Mute This Topic: https://lists.openembedded.org/mt/78947734/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH] file /etc/ethertypes conflicts between netbase-1:6.2-r0.corei7_64 and ebtables-2.0.10+4-r4.corei7_64

2020-12-14 Thread Mark Hatle
I agree.  Netbase is the required for all installs.  ebtables is only used in
some installs.

So unless ebtables has a more up-to-date version of the file, it seems like the
bug is that ebtables needs to either remove the file or sync to the netbase 
version.

(You won't get a conflict if both files are identical, md5sum/shasum)

--Mark

On 12/14/20 4:39 AM, Alexander Kanavin wrote:
> Sorry, but you need to explain why the file from from ebtables (an optional
> package from a 3rd party layer) takes precedence over file from netbase (which
> is a core item).
> 
> Alex
> 
> On Mon, 14 Dec 2020 at 11:33, Outback Dingo  > wrote:
> 
> ---
>  meta/recipes-core/netbase/netbase_6.2.bb  | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/meta/recipes-core/netbase/netbase_6.2.bb
>  b/meta/recipes-core/netbase/netbase_6.2.bb
> 
> index a54d2e7764..262b2cf1bc 100644
> --- a/meta/recipes-core/netbase/netbase_6.2.bb 
> +++ b/meta/recipes-core/netbase/netbase_6.2.bb 
> @@ -19,5 +19,4 @@ do_install () {
>         install -m 0644 ${S}/etc/rpc ${D}${sysconfdir}/rpc
>         install -m 0644 ${S}/etc/protocols ${D}${sysconfdir}/protocols
>         install -m 0644 ${S}/etc/services ${D}${sysconfdir}/services
> -       install -m 0644 ${S}/etc/ethertypes ${D}${sysconfdir}/ethertypes
>  }
> -- 
> 2.20.1
> 
> 
> 
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#145542): 
https://lists.openembedded.org/g/openembedded-core/message/145542
Mute This Topic: https://lists.openembedded.org/mt/78947734/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH 7/7] sstate: set mode explicitly when creating directories in sstate-cache

2020-09-28 Thread Mark Hatle
I worry that this could be problematic for other ways.  Security requirements
for an org or even users who want to share the sstate case 0777 etc.

Wouldn't it be better to warn the user that their umask won't allow them to
share this with others, and give them instructions and opening the umask?

--Mark

On 9/28/20 11:19 AM, Ross Burton wrote:
> When creating directories in the sstate-cache, explicitly set the mode
> passed to mkdir to 0775 so that the directories are group writable, as
> otherwise they cannot be shared with other users.
> 
> Signed-off-by: Ross Burton 
> ---
>  meta/classes/sstate.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 66a96a7603..a8ae75101d 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -787,7 +787,7 @@ sstate_create_package () {
>   return
>   fi
>  
> - mkdir -p `dirname ${SSTATE_PKG}`
> + mkdir --mode=0775 -p `dirname ${SSTATE_PKG}`
>   TFILE=`mktemp ${SSTATE_PKG}.`
>  
>   # Use pigz if available
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142868): 
https://lists.openembedded.org/g/openembedded-core/message/142868
Mute This Topic: https://lists.openembedded.org/mt/77177692/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH 2/2] ssh-regen-hostkeys: Add a recipe with pregenerated ssh host keys

2020-09-28 Thread Mark Hatle
I'm worried about this from a product security perspective.

I think this is very valid case for an autobuilder/autotest infrastructure,
however if this ends up in a release product it will lead to huge problems.

Is there a way we can ensure this can only be used for the autobuilder/autotest
infrastructure, and never provided by accident in an image.  (If a user decided
they must do something like this, we can't stop them -- but we should allow it
to happene either by accident or make it look like it's good practice.)

--Mark

On 9/23/20 10:05 AM, Richard Purdie wrote:
> Host keys are getting bigger and taking an ever increasing amount of time
> to generate. Whilst we do need to test that works, we don't need to test
> it in every image. Add a recipe which can be added to images with
> pre-generated keys, allowing us to speed up tests on the autobuilder
> where it makes sense to.
> 
> Signed-off-by: Richard Purdie 
> ---
>  .../ssh-pregen-hostkeys/dropbear_rsa_host_key | Bin 0 -> 805 bytes
>  .../openssh/ssh_host_ecdsa_key|   9 +
>  .../openssh/ssh_host_ecdsa_key.pub|   1 +
>  .../openssh/ssh_host_ed25519_key  |   7 
>  .../openssh/ssh_host_ed25519_key.pub  |   1 +
>  .../openssh/ssh_host_rsa_key  |  38 ++
>  .../openssh/ssh_host_rsa_key.pub  |   1 +
>  .../ssh-pregen-hostkeys_1.0.bb|  19 +
>  8 files changed, 76 insertions(+)
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/dropbear_rsa_host_key
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key.pub
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ed25519_key
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ed25519_key.pub
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_rsa_key
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_rsa_key.pub
>  create mode 100644 
> meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys_1.0.bb
> 
> diff --git 
> a/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/dropbear_rsa_host_key
>  
> b/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/dropbear_rsa_host_key
> new file mode 100644
> index 
> ..30443c94388530f82308f41517839c8932026eec
> GIT binary patch
> literal 805
> zcmV+=1KRum000Mbb7(Dcb724g00RL40RR920RW_9o z#>e=VPY-g{TMU)xikgot*E3d4mq}vnGGMFK&?`3lQuzp%?!~`G;T{U;4Y_oX
> ztW&5pYa=!AY~l?MU+0l28E$@8(~zi5Bd|IC1+@_wEtWbRYFyfC@g&!whp05e8cXIs
> zAO2$|o3V1#D_vFi`9{vpf^~zgpZ#hwyW(^VKuj zr|?tGWY(1vcP3@X_D<(~^_D`?%NDne77p}AN|!y909XYC`_sATu*WrQf(gmixEp2-
> z>$8a#0)WG zP#Ae@xR_Z!^$9gP{n3QwBhy^nPrHKA9b%3xtLoGy16TJlVr!|nt%cHREwUHDBZ)#&
> zIuv0};sB&0b(1XUZ=R#^gKw)AJ->viB+c z@@ZGZ%D<}4p}ve0)Xwh;edfrl=)p~)sWosd`Y;+BqJ~l)2TH)09_+3
> zZ4`?ekchIl_qjZCYr4Lp0K;iIPX5{`t64nTmV(|FuFJ$BAyJg?pxXg zbV^FhSAdt5 z)g%(nv9r;~j;->7$f}g_o>)88b=v%Es_PL7V(*H}r1F5#*9l3)Gfn zS-L{YkFoPvZ(fHzZ3tgU=!A>JlT_mL2YLkdI4|&9vMig5l?U-%Rc`5EN%eoyF
> zuVc{w004mi|Ec zN-Z{|f9K{)ifw^eNk}eKbdX6Z%1|5s(MS`eaRn9_0H4of0ISncPXCoP&
> jnu6P-g&8cZCSpI?z=;2?Sr97OqIjGUAl>AZv%QM*=5vR&
> 
> literal 0
> HcmV?d1
> 
> diff --git 
> a/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key
>  
> b/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key
> new file mode 100644
> index 000..86c2104ec8a
> --- /dev/null
> +++ 
> b/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key
> @@ -0,0 +1,9 @@
> +-BEGIN OPENSSH PRIVATE KEY-
> +b3BlbnNzaC1rZXktdjEABG5vbmUEbm9uZQABaBNlY2RzYS
> +1zaGEyLW5pc3RwMjU2CG5pc3RwMjU2QQRJR6iZxr/NTqQN9NOwV+WPtu42r2eF
> +rJ0xsnlqw5bpmfz6aDR8RQvVHUZjRGQfR/RXPbQ5x+bjjdm176TuXNhHqAoE27MKBN
> +uzE2VjZHNhLXNoYTItbmlzdHAyNTYIbmlzdHAyNTYAAABBBElHqJnGv81OpA30
> +07BX5Y+27javZ4WsnTGyeWrDlumZ/PpoNHxFC9UdRmNEZB9H9Fc9tDnH5uON2bXvpO5c2E
> +cgLiHv/IWhxwosz9BiNILOOPlXaueL5hVTBKUJkpOi48sNcm9vdEBxZW11bWlw
> +cwECAw==
> +-END OPENSSH PRIVATE KEY-
> diff --git 
> a/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key.pub
>  
> b/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key.pub
> new file mode 100644
> index 000..a358aeb88a7
> --- /dev/null
> +++ 
> b/meta/recipes-connectivity/ssh-pregen-hostkeys/ssh-pregen-hostkeys/openssh/ssh_host_ecdsa_key.pub
> @@ -0,0 +1 @@
> +ecdsa-sha2-nistp256 
> 

Re: [OE-core] [PATCH] oe-setup-builddir: create conf/multiconfig/ from TEMPLATECONF

2020-09-17 Thread Mark Hatle
On 9/17/20 1:39 AM, gr embeter wrote:
> Just for the sake of knowing, how do you reference multiconfig directories?
> Do you copy them manually to $BUILDDIR/conf after setup-env is done?
> I mean when I run `bitbake multiconfig:configA:core-image-minimal'
> there should be "$BUILDDIR/conf/multiconfig/configA.conf" file, right?


My layer contains:

my_layer/conf/multiconfig/multiconfig1.conf
my_layer/conf/multiconfig/multiconfig2.conf
my_layer/conf/multiconfig/multiconfig3.conf
my_layer/conf/multiconfig/multiconfig4.conf

My build directory:

conf/bblayers.conf contains:

BBLAYERS ?= " \
  /scratch1/fray/xilinx/poky/meta \
  /scratch1/fray/xilinx/poky/meta-poky \
  /scratch1/fray/xilinx/poky/meta-yocto-bsp \
  /scratch1/fray/xilinx/my_layer \
  "

conf/local.conf contains:

BBMULTICONFIG = "multiconfig1 multiconfig2 multiconfig3 multiconfig4"

...

The user is responsible for providing their own BBMULTICONFIG.  However, if you
want to provide a default one, you can already including the BBMULTICONFIG for
the setting in your layer set.

--Mark


> On Thu, Sep 17, 2020 at 12:42 AM Mark Hatle
>  wrote:
>>
>> In my configurations, we refernce the multiconfig directories inside of our 
>> many
>> layers.  I definitely don't want the copied into the conf/* directory 
>> structure,
>> since I don't want users modifying the prebuilt ones.
>>
>> --Mark
>>
>> On 9/16/20 12:08 PM, gr embeter wrote:
>>>> Why copy this from TEMPLATECONF when they can be supplied by any layer in 
>>>> your BBLAYERS?
>>>
>>> At the stage of creating build directory I think it is logically to
>>> use TEMPLATECONF both for "local.conf" and "multiconfig"
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 
>>>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142638): 
https://lists.openembedded.org/g/openembedded-core/message/142638
Mute This Topic: https://lists.openembedded.org/mt/76889431/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [PATCH] oe-setup-builddir: create conf/multiconfig/ from TEMPLATECONF

2020-09-16 Thread Mark Hatle
In my configurations, we refernce the multiconfig directories inside of our many
layers.  I definitely don't want the copied into the conf/* directory structure,
since I don't want users modifying the prebuilt ones.

--Mark

On 9/16/20 12:08 PM, gr embeter wrote:
>> Why copy this from TEMPLATECONF when they can be supplied by any layer in 
>> your BBLAYERS?
> 
> At the stage of creating build directory I think it is logically to
> use TEMPLATECONF both for "local.conf" and "multiconfig"
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142624): 
https://lists.openembedded.org/g/openembedded-core/message/142624
Mute This Topic: https://lists.openembedded.org/mt/76889431/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



Re: [OE-core] [meta-oe][PATCH 0/5] ARMv8 Tune reorg

2020-09-15 Thread Mark Hatle


On 9/15/20 9:38 AM, Martin Jansa wrote:
> On Mon, Sep 14, 2020 at 06:54:14PM -0400, Jon Mason wrote:
>> On Mon, Sep 14, 2020 at 11:32 AM Martin Jansa  wrote:
>>>
 This reduces the number of files from 12 to 2 for ARMv8a, and that is 
 excluding the 13 I am adding in this series that would otherwise be unique 
 files.
>>>
>>> I don't have a strong opinion on this anymore, but is the number of the 
>>> include files the issue here?
>>
>> This is the reason why I'm doing it.  Arm needs to support all the
>> cortex-a and cortex-m cores.  This would make the majority of the tune
>> files for only Arm cores.  We could move them into
>> meta/conf/machine/include/arm and not make the level down directory so
>> messy, but I think this is cleaner.
>>
>>> I think the issue is the number of possible combinations and using the same 
>>> TUNE_PKGARCH for different tunes (unlike 32bit arm tune files which use 
>>> different). Bundling all these combinations in fewer include files doesn't 
>>> IMHO improve it.
>>
>> The multitude of combinations and redundancy code entries makes it
>> very error prone.  By putting them all in the same file, it makes it
>> much easier to see the differences in entries and catch errors.
> 
> It does make it easier? In my experience it's more difficult to compare

For some of my usecases, it makes it significantly easier.

I need to build multilib configurations that have a variety of tunes in them.
Before I had to include 5-7 different .inc files, and then ignore the warning
about the same files being included multiple times.

With this approach, I can include just one file, the proper superset of them
all, and then do my multilib configurations together.

(Yes, I realize this isn't the typical Linux use-case, but this seems to be
fairly common to do things like this when working with baremetal configurations
or trying to use the Yocto Project to build toolchains that are shared with 
others.)

> very similar long sections in different parts of the same file, than
> comparing very similar files with diff.
> When working on
> https://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/tune-test
> https://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/tune2-test
> https://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/tune3
> https://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/optdefaulttune
> long time ago, I found it really convenient to compare all cortex*.inc
> files just by replacing the number with some placeholder and comparing
> the resulting files (which were in most cases identical) - doing the
> same inside single file just makes it more complicated, probably by
> first splitting the various sections back to separate files again.
> 
>> Longer term, I'd like to see something that removes the redundancies
>> completely, but it will take some work.
> 
> So now all BSPs need to adapt (by changing from the more specific .inc
> file corresponding to the actual CPU in MACHINE they implement to
> generic family .inc while setting one of the possible DEFAULTTUNE,
> instead of leaving the default from .inc file), but all this only
> for the dubious benefit of having fewer .inc files in oe-core, right?

I suggested how a transition could be done, but this was rejected in favor of
make people move their BSPs to the new format.

--Mark

>>> With 32bit arm include files it was useful to be able to compare various 
>>> include files to check that they follow the same "structure" (or to add new 
>>> cortexa* one by copying existing and regex-replace).
>>>
>>> Also AVAILTUNES will now contain all possible tunes from given family 
>>> (which makes it almost useless for aarch64 tunes). So instead of BSP 
>>> including .inc file corresponding with the core used in the MACHINE and 
>>> getting some "sane" default DEFAULTTUNE (plus some other possible options 
>>> listed in AVAILTUNES) we now force every BSP to set more specific 
>>> DEFAULTTUNE and then let user figure out what other DEFAULTTUNEs might be 
>>> compatible with it, e.g. when someone is doing multi MACHINE builds and 
>>> wants to share TUNE_PKGARCH between them (to save build time, package feed 
>>> size etc).
>>
>> You can still set a generic default for all the machines of a given
>> family.  By including armv8a.inc, you still get 'DEFAULTTUNE ?=
>> "armv8a-crc"'.  This will allow for a generic for an entire family.
>> I'm not seeing the issue.
> 
> OK, with e.g. raspberrypi4-64.conf from
> https://github.com/agherzan/meta-raspberrypi/blob/a4c8118676ba8002edab29fc81b4e4edd9fad1f1/conf/machine/raspberrypi4-64.conf
> which just
> require conf/machine/include/tune-cortexa72.inc
> 
> Without your patches:
> DEFAULTTUNE="cortexa72"
> TUNE_PKGARCH="cortexa72"
> TUNE_CCARGS=" -mcpu=cortex-a72+crc+crypto"
> AVAILTUNES=" armv4 armv4t armv4b armv4tb armv5 armv5t armv5-vfp
> armv5t-vfp armv5hf-vfp armv5thf-vfp armv5b armv5tb armv5b-vfp
> armv5tb-vfp armv5hfb-vfp armv5thfb-vfp armv5e 

Re: [OE-core] [meta-oe][RFC 1/6] arm64: set BASE_LIB to lib64

2020-09-10 Thread Mark Hatle


On 9/10/20 10:11 AM, Jon Mason wrote:
> On Thu, Sep 10, 2020 at 12:57 AM Mark Hatle
>  wrote:
>>
>>
>>
>> On 9/9/20 6:29 PM, Jon Mason wrote:
>>> On Wed, Sep 9, 2020 at 6:55 PM Mark Hatle
>>>  wrote:
>>>>
>>>>
>>>>
>>>> On 9/9/20 5:45 PM, Jon Mason wrote:
>>>>> Set BASE_LIB for all arm64 systems to be lib64 by default.  This can be
>>>>> overridden for those that want something else (see tune-cortexa32.inc).
>>>>>
>>>>> Signed-off-by: Jon Mason 
>>>>> ---
>>>>>  meta/conf/machine/include/arm/arch-arm64.inc   | 3 +--
>>>>>  meta/conf/machine/include/arm/arch-armv8-2a.inc| 2 --
>>>>>  meta/conf/machine/include/arm/arch-armv8a.inc  | 4 
>>>>>  meta/conf/machine/include/tune-cortexa32.inc   | 1 +
>>>>>  meta/conf/machine/include/tune-cortexa35.inc   | 2 --
>>>>>  meta/conf/machine/include/tune-cortexa53.inc   | 2 --
>>>>>  meta/conf/machine/include/tune-cortexa55.inc   | 1 -
>>>>>  meta/conf/machine/include/tune-cortexa57-cortexa53.inc | 1 -
>>>>>  meta/conf/machine/include/tune-cortexa57.inc   | 2 --
>>>>>  meta/conf/machine/include/tune-cortexa72-cortexa53.inc | 2 --
>>>>>  meta/conf/machine/include/tune-cortexa72.inc   | 1 -
>>>>>  meta/conf/machine/include/tune-cortexa73-cortexa53.inc | 2 --
>>>>>  12 files changed, 2 insertions(+), 21 deletions(-)
>>>>>
>>>>> diff --git a/meta/conf/machine/include/arm/arch-arm64.inc 
>>>>> b/meta/conf/machine/include/arm/arch-arm64.inc
>>>>> index 6d5b22fff081..8c3764186ec4 100644
>>>>> --- a/meta/conf/machine/include/arm/arch-arm64.inc
>>>>> +++ b/meta/conf/machine/include/arm/arch-arm64.inc
>>>>> @@ -1,4 +1,5 @@
>>>>>  DEFAULTTUNE ?= "aarch64"
>>>>> +BASE_LIB ?= "lib64"
>>>>
>>>> Does this work?  The default multilib behavior is:
>>>>
>>>> multilib.conf:baselib = "${@d.getVar('BASE_LIB_tune-' + 
>>>> (d.getVar('DEFAULTTUNE')
>>>> or 'INVALID')) or d.getVar('BASELIB')}"
>>>>
>>>> bitbake.conf:baselib = "${BASELIB}"
>>>> bitbake.conf:BASELIB = "lib"
>>>> bitbake.conf:BASELIB_powerpc64 = "lib64"
>>>> bitbake.conf:BASELIB_powerpc64le = "lib64"
>>>>
>>>> is what has been defined in the bitbake.conf file.
>>>>
>>>> Idea being in the 'normal' (not-multilib) case, 'baselib' (which what is 
>>>> used)
>>>> gets set to 'BASELIB'.  Only PowerPC 64 has a hard coded 'lib64'.  Every 
>>>> other
>>>> architecture is considered to be variable, unless defined in the tune.  
>>>> (The two
>>>> PowerPC settings really should be in a PPC tune file.)
>>>>
>>>> So IF the multilib.conf is enabled, it's going to look for 
>>>> BASE_LIB_tune-
>>>> and then BASELIB, but it won't look at 'BASE_LIB'.
>>>
>>> It depends on what "work" is defined as :)
>>> I did build, boot, and run testimage on it (with issue).  But I'm
>>> probably not hitting the multilib issue you are describing.  Honestly,
>>> I don't understand it and needed to spend more time than I did (based
>>> on your comments).  I like the removal of lines to simplify it, but I
>>> need to do some more reading on it.  I'll drop this patch and come
>>> back to the underlying problem at some point in the future.
>>
>> (This is from memory, so it may not be right)
>>
>> In your local.conf:
>>
>> MULTILIBS = "multilib:lib32"
>> require conf/multilib.conf
>>
>> DEFAULTTUNE_virtclass-multilib-lib32 = "armv7vethf"
>> MACHINE = "qemuarm64"
>>
>> Then add something like "lib32-glibc" as a dependency to your image.
> 
> This is building, booting, and passing testimage for me.  It's
> possible I screwed something up (weird that a screwup would make
> things work though).

Boot the image and verify that both /lib and /lib32 are there and populated by
the libc.  If they are then it likely works.

--Mark

> Thanks,
> Jon
> 
>> --Mark
>>
>>> Thanks,
>>> Jon
>>>
>>>>>  require conf/machine/include/arm/arch-armv7ve.inc
>>>>>
>>&g

Re: [OE-core] [meta-oe][RFC 1/6] arm64: set BASE_LIB to lib64

2020-09-09 Thread Mark Hatle


On 9/9/20 6:29 PM, Jon Mason wrote:
> On Wed, Sep 9, 2020 at 6:55 PM Mark Hatle
>  wrote:
>>
>>
>>
>> On 9/9/20 5:45 PM, Jon Mason wrote:
>>> Set BASE_LIB for all arm64 systems to be lib64 by default.  This can be
>>> overridden for those that want something else (see tune-cortexa32.inc).
>>>
>>> Signed-off-by: Jon Mason 
>>> ---
>>>  meta/conf/machine/include/arm/arch-arm64.inc   | 3 +--
>>>  meta/conf/machine/include/arm/arch-armv8-2a.inc| 2 --
>>>  meta/conf/machine/include/arm/arch-armv8a.inc  | 4 
>>>  meta/conf/machine/include/tune-cortexa32.inc   | 1 +
>>>  meta/conf/machine/include/tune-cortexa35.inc   | 2 --
>>>  meta/conf/machine/include/tune-cortexa53.inc   | 2 --
>>>  meta/conf/machine/include/tune-cortexa55.inc   | 1 -
>>>  meta/conf/machine/include/tune-cortexa57-cortexa53.inc | 1 -
>>>  meta/conf/machine/include/tune-cortexa57.inc   | 2 --
>>>  meta/conf/machine/include/tune-cortexa72-cortexa53.inc | 2 --
>>>  meta/conf/machine/include/tune-cortexa72.inc   | 1 -
>>>  meta/conf/machine/include/tune-cortexa73-cortexa53.inc | 2 --
>>>  12 files changed, 2 insertions(+), 21 deletions(-)
>>>
>>> diff --git a/meta/conf/machine/include/arm/arch-arm64.inc 
>>> b/meta/conf/machine/include/arm/arch-arm64.inc
>>> index 6d5b22fff081..8c3764186ec4 100644
>>> --- a/meta/conf/machine/include/arm/arch-arm64.inc
>>> +++ b/meta/conf/machine/include/arm/arch-arm64.inc
>>> @@ -1,4 +1,5 @@
>>>  DEFAULTTUNE ?= "aarch64"
>>> +BASE_LIB ?= "lib64"
>>
>> Does this work?  The default multilib behavior is:
>>
>> multilib.conf:baselib = "${@d.getVar('BASE_LIB_tune-' + 
>> (d.getVar('DEFAULTTUNE')
>> or 'INVALID')) or d.getVar('BASELIB')}"
>>
>> bitbake.conf:baselib = "${BASELIB}"
>> bitbake.conf:BASELIB = "lib"
>> bitbake.conf:BASELIB_powerpc64 = "lib64"
>> bitbake.conf:BASELIB_powerpc64le = "lib64"
>>
>> is what has been defined in the bitbake.conf file.
>>
>> Idea being in the 'normal' (not-multilib) case, 'baselib' (which what is 
>> used)
>> gets set to 'BASELIB'.  Only PowerPC 64 has a hard coded 'lib64'.  Every 
>> other
>> architecture is considered to be variable, unless defined in the tune.  (The 
>> two
>> PowerPC settings really should be in a PPC tune file.)
>>
>> So IF the multilib.conf is enabled, it's going to look for BASE_LIB_tune-
>> and then BASELIB, but it won't look at 'BASE_LIB'.
> 
> It depends on what "work" is defined as :)
> I did build, boot, and run testimage on it (with issue).  But I'm
> probably not hitting the multilib issue you are describing.  Honestly,
> I don't understand it and needed to spend more time than I did (based
> on your comments).  I like the removal of lines to simplify it, but I
> need to do some more reading on it.  I'll drop this patch and come
> back to the underlying problem at some point in the future.

(This is from memory, so it may not be right)

In your local.conf:

MULTILIBS = "multilib:lib32"
require conf/multilib.conf

DEFAULTTUNE_virtclass-multilib-lib32 = "armv7vethf"
MACHINE = "qemuarm64"

Then add something like "lib32-glibc" as a dependency to your image.

--Mark

> Thanks,
> Jon
> 
>>>  require conf/machine/include/arm/arch-armv7ve.inc
>>>
>>> @@ -14,8 +15,6 @@ TUNE_FEATURES_tune-aarch64 = "aarch64"
>>>  TUNE_FEATURES_tune-aarch64_be = "${TUNE_FEATURES_tune-aarch64} bigendian"
>>>  TUNE_PKGARCH_64_tune-aarch64 = "aarch64"
>>>  TUNE_PKGARCH_64_tune-aarch64_be = "aarch64_be"
>>> -BASE_LIB_tune-aarch64 = "lib64"
>>> -BASE_LIB_tune-aarch64_be = "lib64"
>>
>> This was originally done like this to enable multilib, as BASE_LIB needs to 
>> be
>> configured per multilib.  (I don't know if it's really necessary any longer.)
>>
>> Note the other 32/64 bit architecture are still implemented like this.
>>
>> There was also talk at the time of different lib dirs for little and big 
>> endian,
>> but I never saw that actually happen.
>>
>> We also wanted a way to build optimized libraries, like some distributions, 
>> that
>> are optimized for specific tunings and place those into non-conflicting
>> directories.  But again, AFAIK nobody actually did it and I' not sure it
>> actually works.
>>
&

Re: [OE-core] [meta-oe][RFC 2/6] arch-armv8-2a.inc: Add Cortex-A55 tunings

2020-09-09 Thread Mark Hatle


On 9/9/20 6:21 PM, Jon Mason wrote:
> On Wed, Sep 9, 2020 at 6:59 PM Mark Hatle
>  wrote:
>>
>> I like the direction of this work, but one comment.. (down below)
>>
>> On 9/9/20 5:45 PM, Jon Mason wrote:
>>> Migrate the settings in tune-cortexa55.inc to arch-armv8-2a.inc.  This
>>> will allow for a single file to include all of the tunings of a family
>>> of processors.  This will reduce the proliferation of unique files per
>>> processor currently occuring in conf/machine/include/
>>>
>>> Signed-off-by: Jon Mason 
>>> ---
>>>  .../machine/include/arm/arch-armv8-2a.inc | 31 ++-
>>>  meta/conf/machine/include/tune-cortexa55.inc  | 12 ---
>>>  2 files changed, 23 insertions(+), 20 deletions(-)
>>>  delete mode 100644 meta/conf/machine/include/tune-cortexa55.inc
>>>
>>> diff --git a/meta/conf/machine/include/arm/arch-armv8-2a.inc 
>>> b/meta/conf/machine/include/arm/arch-armv8-2a.inc
>>> index b40ebf176e43..38564a17d98b 100644
>>> --- a/meta/conf/machine/include/arm/arch-armv8-2a.inc
>>> +++ b/meta/conf/machine/include/arm/arch-armv8-2a.inc
>>> @@ -1,4 +1,19 @@
>>> -DEFAULTTUNE ?= "armv8-2a"
>>> +#
>>> +# Tune Settings for Cortex-A55
>>> +#
>>> +TUNEVALID[cortexa55] = "Enable Cortex-A55 specific processor optimizations"
>>> +TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'cortexa55', ' 
>>> -mcpu=cortex-a55', '', d)}"
>>> +
>>> +# Little Endian base configs
>>> +AVAILTUNES += "cortexa55"
>>> +ARMPKGARCH_tune-cortexa55   = "cortexa55"
>>> +TUNE_FEATURES_tune-cortexa55= "aarch64 cortexa55 
>>> crypto"
>>> +PACKAGE_EXTRA_ARCHS_tune-cortexa55  = 
>>> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto} cortexa55"
>>> +
>>> +#
>>> +# Defaults for ARMv8-a
>>> +#
>>> +DEFAULTTUNE?= "armv8-2a"
>>>
>>>  TUNEVALID[armv8-2a] = "Enable instructions for ARMv8-a"
>>>  TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'armv8-2a', ' 
>>> -march=armv8.2-a', '', d)}"
>>> @@ -8,10 +23,10 @@ MACHINEOVERRIDES =. 
>>> "${@bb.utils.contains('TUNE_FEATURES', 'armv8-2a', 'armv8-2a
>>>  require conf/machine/include/arm/arch-armv8a.inc
>>>
>>>  # Little Endian base configs
>>> -AVAILTUNES += "armv8-2a armv8-2a-crypto"
>>> -ARMPKGARCH_tune-armv8-2a?= "armv8-2a"
>>> -ARMPKGARCH_tune-armv8-2a-crypto ?= "armv8-2a"
>>> -TUNE_FEATURES_tune-armv8-2a  = "aarch64 armv8-2a"
>>> -TUNE_FEATURES_tune-armv8-2a-crypto   = 
>>> "${TUNE_FEATURES_tune-armv8-2a} crypto"
>>> -PACKAGE_EXTRA_ARCHS_tune-armv8-2a= 
>>> "${PACKAGE_EXTRA_ARCHS_tune-armv8a} armv8-2a"
>>> -PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto = 
>>> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a} armv8-2a-crypto"
>>> +AVAILTUNES += "armv8-2a 
>>> armv8-2a-crypto"
>>> +ARMPKGARCH_tune-armv8-2a   ?= "armv8-2a"
>>> +ARMPKGARCH_tune-armv8-2a-crypto?= "armv8-2a"
>>> +TUNE_FEATURES_tune-armv8-2a = "aarch64 armv8-2a"
>>> +TUNE_FEATURES_tune-armv8-2a-crypto  = 
>>> "${TUNE_FEATURES_tune-armv8-2a} crypto"
>>> +PACKAGE_EXTRA_ARCHS_tune-armv8-2a   = 
>>> "${PACKAGE_EXTRA_ARCHS_tune-armv8a} armv8-2a"
>>> +PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto= 
>>> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a} armv8-2a-crypto"
>>
>> It may make sense, instead of deleting it, to have just the 'require
>> conf/machine/include/arm/arch-armv8-2a.inc' file for transition.  (If a
>> transition period is NOT required or desired, then this isn't necessary!)
>>
>> Tricky part is how to notify someone they're in the midst of the transition
>> period, and using the deprecated behavior.
>>
>> (In past work, I've had a class that looked for a variable to exist, if it 
>> did
>> it printed the contents as warning messages to the console after parsing.  
>> I'd
>> love to see something like this as standar

Re: [OE-core] [meta-oe][RFC 2/6] arch-armv8-2a.inc: Add Cortex-A55 tunings

2020-09-09 Thread Mark Hatle
I like the direction of this work, but one comment.. (down below)

On 9/9/20 5:45 PM, Jon Mason wrote:
> Migrate the settings in tune-cortexa55.inc to arch-armv8-2a.inc.  This
> will allow for a single file to include all of the tunings of a family
> of processors.  This will reduce the proliferation of unique files per
> processor currently occuring in conf/machine/include/
> 
> Signed-off-by: Jon Mason 
> ---
>  .../machine/include/arm/arch-armv8-2a.inc | 31 ++-
>  meta/conf/machine/include/tune-cortexa55.inc  | 12 ---
>  2 files changed, 23 insertions(+), 20 deletions(-)
>  delete mode 100644 meta/conf/machine/include/tune-cortexa55.inc
> 
> diff --git a/meta/conf/machine/include/arm/arch-armv8-2a.inc 
> b/meta/conf/machine/include/arm/arch-armv8-2a.inc
> index b40ebf176e43..38564a17d98b 100644
> --- a/meta/conf/machine/include/arm/arch-armv8-2a.inc
> +++ b/meta/conf/machine/include/arm/arch-armv8-2a.inc
> @@ -1,4 +1,19 @@
> -DEFAULTTUNE ?= "armv8-2a"
> +#
> +# Tune Settings for Cortex-A55
> +#
> +TUNEVALID[cortexa55] = "Enable Cortex-A55 specific processor optimizations"
> +TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'cortexa55', ' 
> -mcpu=cortex-a55', '', d)}"
> +
> +# Little Endian base configs
> +AVAILTUNES += "cortexa55"
> +ARMPKGARCH_tune-cortexa55   = "cortexa55"
> +TUNE_FEATURES_tune-cortexa55= "aarch64 cortexa55 
> crypto"
> +PACKAGE_EXTRA_ARCHS_tune-cortexa55  = 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto} cortexa55"
> +
> +#
> +# Defaults for ARMv8-a
> +#
> +DEFAULTTUNE?= "armv8-2a"
>  
>  TUNEVALID[armv8-2a] = "Enable instructions for ARMv8-a"
>  TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'armv8-2a', ' 
> -march=armv8.2-a', '', d)}"
> @@ -8,10 +23,10 @@ MACHINEOVERRIDES =. "${@bb.utils.contains('TUNE_FEATURES', 
> 'armv8-2a', 'armv8-2a
>  require conf/machine/include/arm/arch-armv8a.inc
>  
>  # Little Endian base configs
> -AVAILTUNES += "armv8-2a armv8-2a-crypto"
> -ARMPKGARCH_tune-armv8-2a?= "armv8-2a"
> -ARMPKGARCH_tune-armv8-2a-crypto ?= "armv8-2a"
> -TUNE_FEATURES_tune-armv8-2a  = "aarch64 armv8-2a"
> -TUNE_FEATURES_tune-armv8-2a-crypto   = 
> "${TUNE_FEATURES_tune-armv8-2a} crypto"
> -PACKAGE_EXTRA_ARCHS_tune-armv8-2a= 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8a} armv8-2a"
> -PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto = 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a} armv8-2a-crypto"
> +AVAILTUNES += "armv8-2a 
> armv8-2a-crypto"
> +ARMPKGARCH_tune-armv8-2a   ?= "armv8-2a"
> +ARMPKGARCH_tune-armv8-2a-crypto?= "armv8-2a"
> +TUNE_FEATURES_tune-armv8-2a = "aarch64 armv8-2a"
> +TUNE_FEATURES_tune-armv8-2a-crypto  = 
> "${TUNE_FEATURES_tune-armv8-2a} crypto"
> +PACKAGE_EXTRA_ARCHS_tune-armv8-2a   = 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8a} armv8-2a"
> +PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto= 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a} armv8-2a-crypto"

It may make sense, instead of deleting it, to have just the 'require
conf/machine/include/arm/arch-armv8-2a.inc' file for transition.  (If a
transition period is NOT required or desired, then this isn't necessary!)

Tricky part is how to notify someone they're in the midst of the transition
period, and using the deprecated behavior.

(In past work, I've had a class that looked for a variable to exist, if it did
it printed the contents as warning messages to the console after parsing.  I'd
love to see something like this as standard behavior for future cases like 
this.)

> diff --git a/meta/conf/machine/include/tune-cortexa55.inc 
> b/meta/conf/machine/include/tune-cortexa55.inc
> deleted file mode 100644
> index 099b6d72851a..
> --- a/meta/conf/machine/include/tune-cortexa55.inc
> +++ /dev/null
> @@ -1,12 +0,0 @@
> -DEFAULTTUNE ?= "cortexa55"
> -
> -TUNEVALID[cortexa55] = "Enable Cortex-A55 specific processor optimizations"
> -TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'cortexa55', ' 
> -mcpu=cortex-a55', '', d)}"
> -
> -require conf/machine/include/arm/arch-armv8-2a.inc
> -
> -# Little Endian base configs
> -AVAILTUNES += "cortexa55"
> -ARMPKGARCH_tune-cortexa55 = "cortexa55"
> -TUNE_FEATURES_tune-cortexa55  = "aarch64 cortexa55 crypto"
> -PACKAGE_EXTRA_ARCHS_tune-cortexa55= 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto} cortexa55"
> 
> 
> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#142332): 
https://lists.openembedded.org/g/openembedded-core/message/142332
Mute This Topic: https://lists.openembedded.org/mt/76744241/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: 

Re: [OE-core] [meta-oe][RFC 1/6] arm64: set BASE_LIB to lib64

2020-09-09 Thread Mark Hatle


On 9/9/20 5:45 PM, Jon Mason wrote:
> Set BASE_LIB for all arm64 systems to be lib64 by default.  This can be
> overridden for those that want something else (see tune-cortexa32.inc).
> 
> Signed-off-by: Jon Mason 
> ---
>  meta/conf/machine/include/arm/arch-arm64.inc   | 3 +--
>  meta/conf/machine/include/arm/arch-armv8-2a.inc| 2 --
>  meta/conf/machine/include/arm/arch-armv8a.inc  | 4 
>  meta/conf/machine/include/tune-cortexa32.inc   | 1 +
>  meta/conf/machine/include/tune-cortexa35.inc   | 2 --
>  meta/conf/machine/include/tune-cortexa53.inc   | 2 --
>  meta/conf/machine/include/tune-cortexa55.inc   | 1 -
>  meta/conf/machine/include/tune-cortexa57-cortexa53.inc | 1 -
>  meta/conf/machine/include/tune-cortexa57.inc   | 2 --
>  meta/conf/machine/include/tune-cortexa72-cortexa53.inc | 2 --
>  meta/conf/machine/include/tune-cortexa72.inc   | 1 -
>  meta/conf/machine/include/tune-cortexa73-cortexa53.inc | 2 --
>  12 files changed, 2 insertions(+), 21 deletions(-)
> 
> diff --git a/meta/conf/machine/include/arm/arch-arm64.inc 
> b/meta/conf/machine/include/arm/arch-arm64.inc
> index 6d5b22fff081..8c3764186ec4 100644
> --- a/meta/conf/machine/include/arm/arch-arm64.inc
> +++ b/meta/conf/machine/include/arm/arch-arm64.inc
> @@ -1,4 +1,5 @@
>  DEFAULTTUNE ?= "aarch64"
> +BASE_LIB ?= "lib64"

Does this work?  The default multilib behavior is:

multilib.conf:baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE')
or 'INVALID')) or d.getVar('BASELIB')}"

bitbake.conf:baselib = "${BASELIB}"
bitbake.conf:BASELIB = "lib"
bitbake.conf:BASELIB_powerpc64 = "lib64"
bitbake.conf:BASELIB_powerpc64le = "lib64"

is what has been defined in the bitbake.conf file.

Idea being in the 'normal' (not-multilib) case, 'baselib' (which what is used)
gets set to 'BASELIB'.  Only PowerPC 64 has a hard coded 'lib64'.  Every other
architecture is considered to be variable, unless defined in the tune.  (The two
PowerPC settings really should be in a PPC tune file.)

So IF the multilib.conf is enabled, it's going to look for BASE_LIB_tune-
and then BASELIB, but it won't look at 'BASE_LIB'.

>  require conf/machine/include/arm/arch-armv7ve.inc
>  
> @@ -14,8 +15,6 @@ TUNE_FEATURES_tune-aarch64 = "aarch64"
>  TUNE_FEATURES_tune-aarch64_be = "${TUNE_FEATURES_tune-aarch64} bigendian"
>  TUNE_PKGARCH_64_tune-aarch64 = "aarch64"
>  TUNE_PKGARCH_64_tune-aarch64_be = "aarch64_be"
> -BASE_LIB_tune-aarch64 = "lib64"
> -BASE_LIB_tune-aarch64_be = "lib64"

This was originally done like this to enable multilib, as BASE_LIB needs to be
configured per multilib.  (I don't know if it's really necessary any longer.)

Note the other 32/64 bit architecture are still implemented like this.

There was also talk at the time of different lib dirs for little and big endian,
but I never saw that actually happen.

We also wanted a way to build optimized libraries, like some distributions, that
are optimized for specific tunings and place those into non-conflicting
directories.  But again, AFAIK nobody actually did it and I' not sure it
actually works.

And for the record BASE_LIB is documented in the tune README as:

BASE_LIB_tune- - The "/lib" location for a specific ABI.  This is
used in a multilib configuration to place the libraries in the correct,
non-conflicting locations.

>  PACKAGE_EXTRA_ARCHS_tune-aarch64 = "aarch64"
>  PACKAGE_EXTRA_ARCHS_tune-aarch64_be = "aarch64_be"
> diff --git a/meta/conf/machine/include/arm/arch-armv8-2a.inc 
> b/meta/conf/machine/include/arm/arch-armv8-2a.inc
> index 1c095256d185..b40ebf176e43 100644
> --- a/meta/conf/machine/include/arm/arch-armv8-2a.inc
> +++ b/meta/conf/machine/include/arm/arch-armv8-2a.inc
> @@ -15,5 +15,3 @@ TUNE_FEATURES_tune-armv8-2a  = "aarch64 
> armv8-2a"
>  TUNE_FEATURES_tune-armv8-2a-crypto   = 
> "${TUNE_FEATURES_tune-armv8-2a} crypto"
>  PACKAGE_EXTRA_ARCHS_tune-armv8-2a= 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8a} armv8-2a"
>  PACKAGE_EXTRA_ARCHS_tune-armv8-2a-crypto = 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8-2a} armv8-2a-crypto"
> -BASE_LIB_tune-armv8-2a   = "lib64"
> -BASE_LIB_tune-armv8-2a-crypto= "lib64"
> diff --git a/meta/conf/machine/include/arm/arch-armv8a.inc 
> b/meta/conf/machine/include/arm/arch-armv8a.inc
> index f810a1e8fc98..5584005f7009 100644
> --- a/meta/conf/machine/include/arm/arch-armv8a.inc
> +++ b/meta/conf/machine/include/arm/arch-armv8a.inc
> @@ -24,7 +24,3 @@ PACKAGE_EXTRA_ARCHS_tune-armv8a= "aarch64 
> armv8a"
>  PACKAGE_EXTRA_ARCHS_tune-armv8a-crc= 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8a} armv8a-crc"
>  PACKAGE_EXTRA_ARCHS_tune-armv8a-crypto = 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8a} armv8a-crypto"
>  PACKAGE_EXTRA_ARCHS_tune-armv8a-crc-crypto = 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8a-crc} armv8a-crypto armv8a-crc-crypto"
> -BASE_LIB_tune-armv8a   = "lib64"
> 

Re: [OE-core] [meta-oe][PATCH 3/4] tune-cortexa57-cortexa53.inc: add CRC and set march

2020-09-09 Thread Mark Hatle


On 9/9/20 5:16 PM, Jon Mason wrote:
> Add CRC to the default tuning of big.LITTLE Cortex-A57-A53.  This puts
> it inline with all other ARMv8a tunings.  Also, reference
> PACKAGE_EXTRA_ARCHS_tune-armv8a-crc instead of
> PACKAGE_EXTRA_ARCHS_tune-aarch64, which sets the -march to armv8 and
> enables the CRC.
> 
> Signed-off-by: Jon Mason 
> ---
>  meta/conf/machine/include/tune-cortexa57-cortexa53.inc | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/conf/machine/include/tune-cortexa57-cortexa53.inc 
> b/meta/conf/machine/include/tune-cortexa57-cortexa53.inc
> index ba4b0738529c..5880bf203231 100644
> --- a/meta/conf/machine/include/tune-cortexa57-cortexa53.inc
> +++ b/meta/conf/machine/include/tune-cortexa57-cortexa53.inc
> @@ -10,6 +10,6 @@ require conf/machine/include/arm/arch-armv8a.inc
>  # Little Endian base configs
>  AVAILTUNES += "cortexa57-cortexa53"
>  ARMPKGARCH_tune-cortexa57-cortexa53 = "cortexa57-cortexa53"
> -TUNE_FEATURES_tune-cortexa57-cortexa53 = "aarch64 cortexa57-cortexa53"
> -PACKAGE_EXTRA_ARCHS_tune-cortexa57-cortexa53 = 
> "${PACKAGE_EXTRA_ARCHS_tune-aarch64} cortexa57-cortexa53"
> +TUNE_FEATURES_tune-cortexa57-cortexa53 = "aarch64 crc cortexa57-cortexa53"
> +PACKAGE_EXTRA_ARCHS_tune-cortexa57-cortexa53 = 
> "${PACKAGE_EXTRA_ARCHS_tune-armv8a-crc} cortexa57-cortexa53"

So all a57 and a53 (as well as big.LITTLE) contain the CRC unit?

I use the a72/a53, but I use it as a common intersection for software for
multiple products.  One has an a72 and one has an a53, so it's not a true
bigLittle configuration.  I could see someone try this using this tune as well.

--Mark

>  BASE_LIB_tune-cortexa57-cortexa53 = "lib64"
> 
> 
> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#142323): 
https://lists.openembedded.org/g/openembedded-core/message/142323
Mute This Topic: https://lists.openembedded.org/mt/76743748/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v8 1/1] package.bbclass: hash equivalency and pr service

2020-09-02 Thread Mark Hatle
When the PR service is enabled a number of small changes may happen
to variables.  In the do_package step a call to package_get_auto_pr
will end up setting PRAUTO and modifying PKGV (if AUTOINC is there).

PRAUTO is then used by EXTENDPRAUTO, which is then used to generate
PKGR.

Since this behavior typically happens BEFORE the BB_UNIHASH is
calculated for do_package, we need a way to defer the expansion
until after we have the unihash value.

Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded
to placeholder values is the easiest way to deal with this.  All other
variables are expanded as expected.

In the next task, typically do_packagedata, we will then use the
UNIHASH from the do_package to get the PR (AUTOPR) as well as
generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC).

The do_packagedata then translates the placeholders to the final values
when copying the data from pkgdata to pkgdata-pdata-input.

Also update the prservice test case.  With unihash, just changing the
do_package (via a _append) will not change the PR.  So write the date
to a specific file that is incorporated into the unihash to ensure it
is always different for the test.  Various assert messages were also
updated to make it easier to figure out where/why a problem occured.

Signed-off-by: Mark Hatle 
---
 meta/classes/package.bbclass  | 58 +++
 meta/conf/bitbake.conf|  1 +
 meta/lib/oeqa/selftest/cases/prservice.py |  8 ++--
 3 files changed, 55 insertions(+), 12 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 464ba8dc6f..e6236c0bb2 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -7,7 +7,7 @@
 #
 # There are the following default steps but PACKAGEFUNCS can be extended:
 #
-# a) package_get_auto_pr - get PRAUTO from remote PR service
+# a) package_convert_pr_autoinc - convert AUTOINC in PKGV to 
${PRSERV_PV_AUTOINC}
 #
 # b) perform_packagecopy - Copy D into PKGD
 #
@@ -664,12 +664,20 @@ def runtime_mapping_rename (varname, pkg, d):
 #bb.note("%s after: %s" % (varname, d.getVar(varname)))
 
 #
-# Package functions suitable for inclusion in PACKAGEFUNCS
+# Used by do_packagedata (and possibly other routines post do_package)
 #
 
+package_get_auto_pr[vardepsexclude] = "BB_TASKDEPDATA"
 python package_get_auto_pr() {
 import oe.prservice
-import re
+
+def get_do_package_hash(pn):
+if d.getVar("BB_RUNTASK") != "do_package":
+taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+for dep in taskdepdata:
+if taskdepdata[dep][1] == "do_package" and taskdepdata[dep][0] 
== pn:
+return taskdepdata[dep][6]
+return None
 
 # Support per recipe PRSERV_HOST
 pn = d.getVar('PN')
@@ -681,15 +689,22 @@ python package_get_auto_pr() {
 
 # PR Server not active, handle AUTOINC
 if not d.getVar('PRSERV_HOST'):
-if 'AUTOINC' in pkgv:
-d.setVar("PKGV", pkgv.replace("AUTOINC", "0"))
+d.setVar("PRSERV_PV_AUTOINC", "0")
 return
 
 auto_pr = None
 pv = d.getVar("PV")
 version = d.getVar("PRAUTOINX")
 pkgarch = d.getVar("PACKAGE_ARCH")
-checksum = d.getVar("BB_TASKHASH")
+checksum = get_do_package_hash(pn)
+
+# If do_package isn't in the dependencies, we can't get the checksum...
+if not checksum:
+bb.warn('Task %s requested do_package unihash, but it was not 
available.' % d.getVar('BB_RUNTASK'))
+#taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+#for dep in taskdepdata:
+#bb.warn('%s:%s = %s' % (taskdepdata[dep][0], taskdepdata[dep][1], 
taskdepdata[dep][6]))
+return
 
 if d.getVar('PRSERV_LOCKDOWN'):
 auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or 
d.getVar('PRAUTO_' + version) or None
@@ -707,7 +722,7 @@ python package_get_auto_pr() {
 srcpv = bb.fetch2.get_srcrev(d)
 base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
 value = conn.getPR(base_ver, pkgarch, srcpv)
-d.setVar("PKGV", pkgv.replace("AUTOINC", str(value)))
+d.setVar("PRSERV_PV_AUTOINC", str(value))
 
 auto_pr = conn.getPR(version, pkgarch, checksum)
 except Exception as e:
@@ -717,6 +732,22 @@ python package_get_auto_pr() {
 d.setVar('PRAUTO',str(auto_pr))
 }
 
+#
+# Package functions suitable for inclusion in PACKAGEFUNCS
+#
+
+python package_convert_pr_autoinc() {
+pkgv = d.getVar("PKGV")
+
+# Adjust pkgv as necessary...
+if 'AUTOINC' in pkgv:
+d.setVar("PKGV", pkgv.replace("AUTOINC", "${PRSERV_PV_AUTOINC}"))
+
+# Change PRSERV_PV_AUT

[OE-core] [master][PATCH v8 0/1]

2020-09-02 Thread Mark Hatle
v8:

Only one patch left...  Was found that in some configurations the
prservice oe-selftest (prservice.BitbakePrTests) was failing.  Fix
the issue, by changing the way the test case configures itself to ensure
it will work with both unihash on and off.  (Before it only worked with
it off.)

v7:

Update packagedata_translate_pr_autoinc function to handle when there is no
pkgdata to process.

Exception: bb.process.ExecutionError: Execution of 
'/mnt/b/yoe/master/build/tmp/work/cortexa7t2hf-neon-vfpv4-yoe-linux-gnueabi/libtool-cross/2.4.6-r0/temp/run.packagedata_translate_pr_autoinc.2499440'
 failed with exit code 123:
sed: no input files
WARNING: 
/mnt/b/yoe/master/build/tmp/work/cortexa7t2hf-neon-vfpv4-yoe-linux-gnueabi/libtool-cross/2.4.6-r0/temp/run.packagedata_translate_pr_autoinc.2499440:151
 exit 123 from 'xargs sed -e 's,@PRSERV_PV_AUTOINC@,AUTOINC,g' -e 
's,@EXTENDPRAUTO@,.0,g' -i'

Uses the --no-run-if-empty options for xargs.

v6:

Refactor do_packagedata to use existing copy routine, and then sed "inline" to
translate EXTENDPRAUTO and AUTOINC parts.

v5 (not sent to mailing list):

Refactor do_packagedata and create a custom copy routine.  The routine also
uses sed to translate EXTENDPRAUTO and AUTOINC.  This also introduces changes
to the new package_convert_pr_autoinc to always translate PRSERV_PV_AUTOINC
and EXTENDPRAUTO to @PRSERV_PV_AUTOINC@ and @EXTENDPRAUTO@.  So all users of
the do_package components will see this translated version.

Remove the commit that moved package_get_auto_pr into the packagedata.bbclass.

v4 (not sent to mailing list):

rename package_convert_autoinc to package_convert_pr_autoinc.

Revert the creation of 'exclude_pkgdata_vars', and all of the custom 
processing of the excluded variables.

(prior patch 1 and 2 were merged, so no longer part of this.

v3 (not sent to mailing list):

Address all patch comments from the list, except for the 'sed' refactor
items.

v2:

Most comments have been addressed to create a v2 version.  I've refactored
the commits to make a few things more clear, basically moving code around
and fixing minor issues BEFORE the big patch.

Before there were two patches that together implemented the PR Serv/Hash
work.  This has been combined into a single patch and the oe_nohash stuff
has simply been removed as no longer applicable.

The only comment that was NOT addressed in this was the suggestion to
sed the pkgdata files in do_packagedata.  I'm hesitent to do this as
sed with ${...} in them has proven to be fragile for me in the past.  If
we decide that is necessary, I'd suggest we start with this set and then
sed with ${...} in them has proven to be fragile for me in the past.  If
we decide that is necessary, I'd suggest we start with this set and then
make that change after this proves to work (or not).

v1:

Before PR service didn't work reliably with hash equivalency.  Generally
you ended up with results that may not be reproducible, even if you
started with the same PR service database and hash equivalency database. 

Overtime, intermediate PR values would be created that would cause thing
to get out of sync in the case of certain rebuilds or other corner cases.

The set refactors the PR service to work along side the new hash equiv
system.  It moves the PR and AUTOINC lookup to AFTER the do_package task
is complete.  This allows us to use the do_package unihash for lookup.

Additionally this fixed a small issue with the kernel, where the PR value
could get incremented twice.  The fix is an artifact of the other changes
that cause us to only run the PR service work once per recipe.

This has been tested with the following workflow, which covers one of
the critical corner cases for me:

configure local.conf with:

   BB_HASHSERVE = "auto"
   BB_SIGNATURE_HANDLER = "OEEquivHash"
   
   PRSERV_HOST ??= "localhost:0"
   
   INHERIT += "reproducible_build"
   INHERIT += "buildhistory"

bitbake glibc linux-yocto

# Modify meta/recipes-core/glibc/glibc_2.32.bb, add a comment 
# to the do_patch_append().  This will taint the hash of this
# function.

bitbake glibc linux-yocto

# The system should have detected the output was the same, and
# no proceed past do_package in glibc.  The kernel should not
# have built at all.

# Store/mv the tmp and sstate-cache from that build elsewhere
# repeat the run

bitbake glibc linux-yocto   

# Compare the results of tmp/deploy//* between last
# and current run.
# 
# The contents should be the same (filenames specifically).  
# 
# Also the kernel should be r0.0, not r0.1.

Note: if the hash equivalency database or PR server database (located
in the cache directory) is removed, the values may not be the same
as the previous run.

Additionally while testing the various package_*.bbclass files, it
was noted that package_tar.bbclass was not working the same way as
the others.  This was correct as a standalone patch.

Mark Hatle (1):
  package.bbclass: ha

Re: [OE-core] [master][PATCH v7 3/3] package.bbclass: hash equivalency and pr service

2020-09-01 Thread Mark Hatle
Just an FYI.. To reproduce the failure you HAVE to use the 'poky' distro.  I had
switched to 'nodistro' because the system complained about 
"SANITY_TESTED_DISTROS"..

Working on this now...

--Mark

On 9/1/20 3:44 PM, Mark Hatle wrote:
> 
> 
> On 8/28/20 1:05 AM, Richard Purdie wrote:
>> On Thu, 2020-08-27 at 17:12 -0500, Mark Hatle wrote:
>>> When the PR service is enabled a number of small changes may happen
>>> to variables.  In the do_package step a call to package_get_auto_pr
>>> will end up setting PRAUTO and modifying PKGV (if AUTOINC is there).
>>>
>>> PRAUTO is then used by EXTENDPRAUTO, which is then used to generate
>>> PKGR.
>>>
>>> Since this behavior typically happens BEFORE the BB_UNIHASH is
>>> calculated for do_package, we need a way to defer the expansion
>>> until after we have the unihash value.
>>>
>>> Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded
>>> to placeholder values is the easiest way to deal with this.  All
>>> other
>>> variables are expanded as expected.
>>>
>>> In the next task, typically do_packagedata, we will then use the
>>> UNIHASH from the do_package to get the PR (AUTOPR) as well as
>>> generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC).
>>>
>>> The do_packagedata then translates the placeholders to the final
>>> values
>>> when copying the data from pkgdata to pkgdata-pdata-input.
>>>
>>> Signed-off-by: Mark Hatle 
>>> ---
>>>  meta/classes/package.bbclass | 58 +++---
>>> --
>>>  meta/conf/bitbake.conf   |  1 +
>>>  2 files changed, 51 insertions(+), 8 deletions(-)
>>
>> This looked ok in testing apart from:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1289
>> (and similar)
>>
>> which should reproduce with:
>>
>> oe-selftest -r prservice.BitbakePrTests
>>
>> basically the PRServ tests need updating to match the code changes.
> 
> I see the error in the logs, but I'm not able to reproduce this.  No errors 
> are
> reported on my system.  I added some additional debugging and I'm seeing that
> it's reading from
> 
> .../tmp-glibc/pkgdata/qemux86-64/runtime/m4
> 
> This is the correct one, as it should have the complete value of PKGR as it 
> was
> the one transferred by do_packagedata.
> 
> Is there a standard way to keep the temp files around so that I can see what 
> the
> values actually are?
> 
> --Mark
> 
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>>
>> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#142089): 
https://lists.openembedded.org/g/openembedded-core/message/142089
Mute This Topic: https://lists.openembedded.org/mt/76462561/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [master][PATCH v7 3/3] package.bbclass: hash equivalency and pr service

2020-09-01 Thread Mark Hatle


On 8/28/20 1:05 AM, Richard Purdie wrote:
> On Thu, 2020-08-27 at 17:12 -0500, Mark Hatle wrote:
>> When the PR service is enabled a number of small changes may happen
>> to variables.  In the do_package step a call to package_get_auto_pr
>> will end up setting PRAUTO and modifying PKGV (if AUTOINC is there).
>>
>> PRAUTO is then used by EXTENDPRAUTO, which is then used to generate
>> PKGR.
>>
>> Since this behavior typically happens BEFORE the BB_UNIHASH is
>> calculated for do_package, we need a way to defer the expansion
>> until after we have the unihash value.
>>
>> Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded
>> to placeholder values is the easiest way to deal with this.  All
>> other
>> variables are expanded as expected.
>>
>> In the next task, typically do_packagedata, we will then use the
>> UNIHASH from the do_package to get the PR (AUTOPR) as well as
>> generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC).
>>
>> The do_packagedata then translates the placeholders to the final
>> values
>> when copying the data from pkgdata to pkgdata-pdata-input.
>>
>> Signed-off-by: Mark Hatle 
>> ---
>>  meta/classes/package.bbclass | 58 +++---
>> --
>>  meta/conf/bitbake.conf   |  1 +
>>  2 files changed, 51 insertions(+), 8 deletions(-)
> 
> This looked ok in testing apart from:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1289
> (and similar)
> 
> which should reproduce with:
> 
> oe-selftest -r prservice.BitbakePrTests
> 
> basically the PRServ tests need updating to match the code changes.

I see the error in the logs, but I'm not able to reproduce this.  No errors are
reported on my system.  I added some additional debugging and I'm seeing that
it's reading from

.../tmp-glibc/pkgdata/qemux86-64/runtime/m4

This is the correct one, as it should have the complete value of PKGR as it was
the one transferred by do_packagedata.

Is there a standard way to keep the temp files around so that I can see what the
values actually are?

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#142079): 
https://lists.openembedded.org/g/openembedded-core/message/142079
Mute This Topic: https://lists.openembedded.org/mt/76462561/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v7 3/3] package.bbclass: hash equivalency and pr service

2020-08-27 Thread Mark Hatle
When the PR service is enabled a number of small changes may happen
to variables.  In the do_package step a call to package_get_auto_pr
will end up setting PRAUTO and modifying PKGV (if AUTOINC is there).

PRAUTO is then used by EXTENDPRAUTO, which is then used to generate
PKGR.

Since this behavior typically happens BEFORE the BB_UNIHASH is
calculated for do_package, we need a way to defer the expansion
until after we have the unihash value.

Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded
to placeholder values is the easiest way to deal with this.  All other
variables are expanded as expected.

In the next task, typically do_packagedata, we will then use the
UNIHASH from the do_package to get the PR (AUTOPR) as well as
generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC).

The do_packagedata then translates the placeholders to the final values
when copying the data from pkgdata to pkgdata-pdata-input.

Signed-off-by: Mark Hatle 
---
 meta/classes/package.bbclass | 58 +++-
 meta/conf/bitbake.conf   |  1 +
 2 files changed, 51 insertions(+), 8 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7a36262eb6..2199bb019b 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -7,7 +7,7 @@
 #
 # There are the following default steps but PACKAGEFUNCS can be extended:
 #
-# a) package_get_auto_pr - get PRAUTO from remote PR service
+# a) package_convert_pr_autoinc - convert AUTOINC in PKGV to 
${PRSERV_PV_AUTOINC}
 #
 # b) perform_packagecopy - Copy D into PKGD
 #
@@ -664,12 +664,20 @@ def runtime_mapping_rename (varname, pkg, d):
 #bb.note("%s after: %s" % (varname, d.getVar(varname)))
 
 #
-# Package functions suitable for inclusion in PACKAGEFUNCS
+# Used by do_packagedata (and possibly other routines post do_package)
 #
 
+package_get_auto_pr[vardepsexclude] = "BB_TASKDEPDATA"
 python package_get_auto_pr() {
 import oe.prservice
-import re
+
+def get_do_package_hash(pn):
+if d.getVar("BB_RUNTASK") != "do_package":
+taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+for dep in taskdepdata:
+if taskdepdata[dep][1] == "do_package" and taskdepdata[dep][0] 
== pn:
+return taskdepdata[dep][6]
+return None
 
 # Support per recipe PRSERV_HOST
 pn = d.getVar('PN')
@@ -681,15 +689,22 @@ python package_get_auto_pr() {
 
 # PR Server not active, handle AUTOINC
 if not d.getVar('PRSERV_HOST'):
-if 'AUTOINC' in pkgv:
-d.setVar("PKGV", pkgv.replace("AUTOINC", "0"))
+d.setVar("PRSERV_PV_AUTOINC", "0")
 return
 
 auto_pr = None
 pv = d.getVar("PV")
 version = d.getVar("PRAUTOINX")
 pkgarch = d.getVar("PACKAGE_ARCH")
-checksum = d.getVar("BB_TASKHASH")
+checksum = get_do_package_hash(pn)
+
+# If do_package isn't in the dependencies, we can't get the checksum...
+if not checksum:
+bb.warn('Task %s requested do_package unihash, but it was not 
available.' % d.getVar('BB_RUNTASK'))
+#taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+#for dep in taskdepdata:
+#bb.warn('%s:%s = %s' % (taskdepdata[dep][0], taskdepdata[dep][1], 
taskdepdata[dep][6]))
+return
 
 if d.getVar('PRSERV_LOCKDOWN'):
 auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or 
d.getVar('PRAUTO_' + version) or None
@@ -707,7 +722,7 @@ python package_get_auto_pr() {
 srcpv = bb.fetch2.get_srcrev(d)
 base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
 value = conn.getPR(base_ver, pkgarch, srcpv)
-d.setVar("PKGV", pkgv.replace("AUTOINC", str(value)))
+d.setVar("PRSERV_PV_AUTOINC", str(value))
 
 auto_pr = conn.getPR(version, pkgarch, checksum)
 except Exception as e:
@@ -717,6 +732,22 @@ python package_get_auto_pr() {
 d.setVar('PRAUTO',str(auto_pr))
 }
 
+#
+# Package functions suitable for inclusion in PACKAGEFUNCS
+#
+
+python package_convert_pr_autoinc() {
+pkgv = d.getVar("PKGV")
+
+# Adjust pkgv as necessary...
+if 'AUTOINC' in pkgv:
+d.setVar("PKGV", pkgv.replace("AUTOINC", "${PRSERV_PV_AUTOINC}"))
+
+# Change PRSERV_PV_AUTOINC and EXTENDPRAUTO usage to special values
+d.setVar('PRSERV_PV_AUTOINC', '@PRSERV_PV_AUTOINC@')
+d.setVar('EXTENDPRAUTO', '@EXTENDPRAUTO@')
+}
+
 LOCALEBASEPN ??= "${PN}"
 
 python package_do_split_locales() {
@@ -2335,7 +2366,7 @@ python do_package () {
 package_qa_handle_error("var-undefined", msg, d)
 return
 
-bb.build.exec_func("package_get_auto_pr", d)

[OE-core] [master][PATCH v7 2/3] kernel.bbclass: Move away from calling package_get_auto_pr

2020-08-27 Thread Mark Hatle
...instead we call read_subpackage_metadata.

Calling package_get_auto_pr *should* result in the same PKGV AUTOINC
replacement.  However, it will also end up changing PKGR differently
then do_package as the BB_TASKHASH used will be for the wrong task.

Generally this won't cause any real-world issue, but it could cause
problems.

Moving to read_subpackage_metadata ensures that the values used
in do_package will be read in and used for kernel deployment.

Signed-off-by: Mark Hatle 
---
 meta/classes/kernel.bbclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 7869184b94..14c22da306 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -748,7 +748,10 @@ kernel_do_deploy() {
done
fi
 }
-do_deploy[prefuncs] += "package_get_auto_pr"
+
+# We deploy to filenames that include PKGV and PKGR, read the saved data to
+# ensure we get the right values for both
+do_deploy[prefuncs] += "read_subpackage_metadata"
 
 addtask deploy after do_populate_sysroot do_packagedata
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141927): 
https://lists.openembedded.org/g/openembedded-core/message/141927
Mute This Topic: https://lists.openembedded.org/mt/76462560/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v7 0/3] Allow PR Service and hash equiv together

2020-08-27 Thread Mark Hatle
v7:

Update packagedata_translate_pr_autoinc function to handle when there is no
pkgdata to process.

Exception: bb.process.ExecutionError: Execution of 
'/mnt/b/yoe/master/build/tmp/work/cortexa7t2hf-neon-vfpv4-yoe-linux-gnueabi/libtool-cross/2.4.6-r0/temp/run.packagedata_translate_pr_autoinc.2499440'
 failed with exit code 123:
sed: no input files
WARNING: 
/mnt/b/yoe/master/build/tmp/work/cortexa7t2hf-neon-vfpv4-yoe-linux-gnueabi/libtool-cross/2.4.6-r0/temp/run.packagedata_translate_pr_autoinc.2499440:151
 exit 123 from 'xargs sed -e 's,@PRSERV_PV_AUTOINC@,AUTOINC,g' -e 
's,@EXTENDPRAUTO@,.0,g' -i'

Uses the --no-run-if-empty options for xargs.

v6:

Refactor do_packagedata to use existing copy routine, and then sed "inline" to
translate EXTENDPRAUTO and AUTOINC parts.

v5 (not sent to mailing list):

Refactor do_packagedata and create a custom copy routine.  The routine also
uses sed to translate EXTENDPRAUTO and AUTOINC.  This also introduces changes
to the new package_convert_pr_autoinc to always translate PRSERV_PV_AUTOINC
and EXTENDPRAUTO to @PRSERV_PV_AUTOINC@ and @EXTENDPRAUTO@.  So all users of
the do_package components will see this translated version.

Remove the commit that moved package_get_auto_pr into the packagedata.bbclass.

v4 (not sent to mailing list):

rename package_convert_autoinc to package_convert_pr_autoinc.

Revert the creation of 'exclude_pkgdata_vars', and all of the custom 
processing of the excluded variables.

(prior patch 1 and 2 were merged, so no longer part of this.

v3 (not sent to mailing list):

Address all patch comments from the list, except for the 'sed' refactor
items.

v2:

Most comments have been addressed to create a v2 version.  I've refactored
the commits to make a few things more clear, basically moving code around
and fixing minor issues BEFORE the big patch.

Before there were two patches that together implemented the PR Serv/Hash
work.  This has been combined into a single patch and the oe_nohash stuff
has simply been removed as no longer applicable.

The only comment that was NOT addressed in this was the suggestion to
sed the pkgdata files in do_packagedata.  I'm hesitent to do this as
sed with ${...} in them has proven to be fragile for me in the past.  If
we decide that is necessary, I'd suggest we start with this set and then
sed with ${...} in them has proven to be fragile for me in the past.  If
we decide that is necessary, I'd suggest we start with this set and then
make that change after this proves to work (or not).

v1:

Before PR service didn't work reliably with hash equivalency.  Generally
you ended up with results that may not be reproducible, even if you
started with the same PR service database and hash equivalency database. 

Overtime, intermediate PR values would be created that would cause thing
to get out of sync in the case of certain rebuilds or other corner cases.

The set refactors the PR service to work along side the new hash equiv
system.  It moves the PR and AUTOINC lookup to AFTER the do_package task
is complete.  This allows us to use the do_package unihash for lookup.

Additionally this fixed a small issue with the kernel, where the PR value
could get incremented twice.  The fix is an artifact of the other changes
that cause us to only run the PR service work once per recipe.

This has been tested with the following workflow, which covers one of
the critical corner cases for me:

configure local.conf with:

   BB_HASHSERVE = "auto"
   BB_SIGNATURE_HANDLER = "OEEquivHash"
   
   PRSERV_HOST ??= "localhost:0"
   
   INHERIT += "reproducible_build"
   INHERIT += "buildhistory"

bitbake glibc linux-yocto

# Modify meta/recipes-core/glibc/glibc_2.32.bb, add a comment 
# to the do_patch_append().  This will taint the hash of this
# function.

bitbake glibc linux-yocto

# The system should have detected the output was the same, and
# no proceed past do_package in glibc.  The kernel should not
# have built at all.

# Store/mv the tmp and sstate-cache from that build elsewhere
# repeat the run

bitbake glibc linux-yocto   

# Compare the results of tmp/deploy//* between last
# and current run.
# 
# The contents should be the same (filenames specifically).  
# 
# Also the kernel should be r0.0, not r0.1.

Note: if the hash equivalency database or PR server database (located
in the cache directory) is removed, the values may not be the same
as the previous run.

Additionally while testing the various package_*.bbclass files, it
was noted that package_tar.bbclass was not working the same way as
the others.  This was correct as a standalone patch.

Mark Hatle (3):
  buildhistory.bbclass: Rework to use read_subpackage_metadata
  kernel.bbclass: Move away from calling package_get_auto_pr
  package.bbclass: hash equivalency and pr service

 meta/classes/buildhistory.bbclass | 49 --
 meta/classes/kernel.bbclass   |  5 ++-
 meta/classes/package.

[OE-core] [master][PATCH v7 1/3] buildhistory.bbclass: Rework to use read_subpackage_metadata

2020-08-27 Thread Mark Hatle
Using this mechanism ensures that we have a single point to implement
the loading of the package and subpackage meta data.  This also then
allows the buildhistory class to use the regular datastore vs it's
own custom arrays for processing history items.

Signed-off-by: Mark Hatle 
---
 meta/classes/buildhistory.bbclass | 49 ++-
 1 file changed, 22 insertions(+), 27 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass 
b/meta/classes/buildhistory.bbclass
index 805e976ac5..fbdb1d3161 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -258,20 +258,15 @@ python buildhistory_emit_pkghistory() {
 rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') 
or ""))
 write_recipehistory(rcpinfo, d)
 
-pkgdest = d.getVar('PKGDEST')
+bb.build.exec_func("read_subpackage_metadata", d)
+
 for pkg in packagelist:
-pkgdata = {}
-with open(os.path.join(pkgdata_dir, 'runtime', pkg)) as f:
-for line in f.readlines():
-item = line.rstrip('\n').split(': ', 1)
-key = item[0]
-if key.endswith('_' + pkg):
-key = key[:-len(pkg)-1]
-pkgdata[key] = 
item[1].encode('latin-1').decode('unicode_escape')
-
-pkge = pkgdata.get('PKGE', '0')
-pkgv = pkgdata['PKGV']
-pkgr = pkgdata['PKGR']
+localdata = d.createCopy()
+localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
+
+pkge = localdata.getVar("PKGE") or '0'
+pkgv = localdata.getVar("PKGV")
+pkgr = localdata.getVar("PKGR")
 #
 # Find out what the last version was
 # Make sure the version did not decrease
@@ -288,31 +283,31 @@ python buildhistory_emit_pkghistory() {
 
 pkginfo = PackageInfo(pkg)
 # Apparently the version can be different on a per-package basis (see 
Python)
-pkginfo.pe = pkgdata.get('PE', '0')
-pkginfo.pv = pkgdata['PV']
-pkginfo.pr = pkgdata['PR']
-pkginfo.pkg = pkgdata['PKG']
+pkginfo.pe = localdata.getVar("PE") or '0'
+pkginfo.pv = localdata.getVar("PV")
+pkginfo.pr = localdata.getVar("PR")
+pkginfo.pkg = localdata.getVar("PKG")
 pkginfo.pkge = pkge
 pkginfo.pkgv = pkgv
 pkginfo.pkgr = pkgr
-pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RPROVIDES', "")))
-pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RDEPENDS', "")))
-pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RRECOMMENDS', "")))
-pkginfo.rsuggests = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RSUGGESTS', "")))
-pkginfo.rreplaces = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RREPLACES', "")))
-pkginfo.rconflicts = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RCONFLICTS', "")))
-pkginfo.files = oe.utils.squashspaces(pkgdata.get('FILES', ""))
+pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RPROVIDES") or ""))
+pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RDEPENDS") or ""))
+pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RRECOMMENDS") or ""))
+pkginfo.rsuggests = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RSUGGESTS") or ""))
+pkginfo.replaces = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RREPLACES") or ""))
+pkginfo.rconflicts = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RCONFLICTS") or ""))
+pkginfo.files = oe.utils.squashspaces(localdata.getVar("FILES") or "")
 for filevar in pkginfo.filevars:
-pkginfo.filevars[filevar] = pkgdata.get(filevar, "")
+pkginfo.filevars[filevar] = localdata.getVar(filevar) or ""
 
 # Gather information about packaged files
-val = pkgdata.get('FILES_INFO', '')
+val = localdata.getVar('FILES_INFO') or ''
 dictval = json.loads(val)
 filelist = list(dictval.keys())
 filelist.sort()
 pkginfo.filelist = " ".join([shlex.quote(x) for x in filelist])
 
-pkginfo.size = int(pkgdata['PKGSIZE'])
+pkginfo.size = int(localdata.getVar('PKGSIZE') or '0')
 
 write_pkghistory(pkginfo, d)
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141926): 
https://lists.openembedded.org/g/openembedded-core/message/141926
Mute This Topic: https://lists.openembedded.org/mt/76462559/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v6 0/3] Allow PR Service and hash equiv together

2020-08-27 Thread Mark Hatle
v6:

Refactor do_packagedata to use existing copy routine, and then sed "inline" to
translate EXTENDPRAUTO and AUTOINC parts.

v5 (not sent to mailing list):

Refactor do_packagedata and create a custom copy routine.  The routine also
uses sed to translate EXTENDPRAUTO and AUTOINC.  This also introduces changes
to the new package_convert_pr_autoinc to always translate PRSERV_PV_AUTOINC
and EXTENDPRAUTO to @PRSERV_PV_AUTOINC@ and @EXTENDPRAUTO@.  So all users of
the do_package components will see this translated version.

Remove the commit that moved package_get_auto_pr into the packagedata.bbclass.

v4 (not sent to mailing list):

rename package_convert_autoinc to package_convert_pr_autoinc.

Revert the creation of 'exclude_pkgdata_vars', and all of the custom 
processing of the excluded variables.

(prior patch 1 and 2 were merged, so no longer part of this.

v3 (not sent to mailing list):

Address all patch comments from the list, except for the 'sed' refactor
items.

v2:

Most comments have been addressed to create a v2 version.  I've refactored
the commits to make a few things more clear, basically moving code around
and fixing minor issues BEFORE the big patch.

Before there were two patches that together implemented the PR Serv/Hash
work.  This has been combined into a single patch and the oe_nohash stuff
has simply been removed as no longer applicable.

The only comment that was NOT addressed in this was the suggestion to
sed the pkgdata files in do_packagedata.  I'm hesitent to do this as
sed with ${...} in them has proven to be fragile for me in the past.  If
we decide that is necessary, I'd suggest we start with this set and then
sed with ${...} in them has proven to be fragile for me in the past.  If
we decide that is necessary, I'd suggest we start with this set and then
make that change after this proves to work (or not).

v1:

Before PR service didn't work reliably with hash equivalency.  Generally
you ended up with results that may not be reproducible, even if you
started with the same PR service database and hash equivalency database. 

Overtime, intermediate PR values would be created that would cause thing
to get out of sync in the case of certain rebuilds or other corner cases.

The set refactors the PR service to work along side the new hash equiv
system.  It moves the PR and AUTOINC lookup to AFTER the do_package task
is complete.  This allows us to use the do_package unihash for lookup.

Additionally this fixed a small issue with the kernel, where the PR value
could get incremented twice.  The fix is an artifact of the other changes
that cause us to only run the PR service work once per recipe.

This has been tested with the following workflow, which covers one of
the critical corner cases for me:

configure local.conf with:

   BB_HASHSERVE = "auto"
   BB_SIGNATURE_HANDLER = "OEEquivHash"
   
   PRSERV_HOST ??= "localhost:0"
   
   INHERIT += "reproducible_build"
   INHERIT += "buildhistory"

bitbake glibc linux-yocto

# Modify meta/recipes-core/glibc/glibc_2.32.bb, add a comment 
# to the do_patch_append().  This will taint the hash of this
# function.

bitbake glibc linux-yocto

# The system should have detected the output was the same, and
# no proceed past do_package in glibc.  The kernel should not
# have built at all.

# Store/mv the tmp and sstate-cache from that build elsewhere
# repeat the run

bitbake glibc linux-yocto   

# Compare the results of tmp/deploy//* between last
# and current run.
# 
# The contents should be the same (filenames specifically).  
# 
# Also the kernel should be r0.0, not r0.1.

Note: if the hash equivalency database or PR server database (located
in the cache directory) is removed, the values may not be the same
as the previous run.

Additionally while testing the various package_*.bbclass files, it
was noted that package_tar.bbclass was not working the same way as
the others.  This was correct as a standalone patch.

Mark Hatle (3):
  buildhistory.bbclass: Rework to use read_subpackage_metadata
  kernel.bbclass: Move away from calling package_get_auto_pr
  package.bbclass: hash equivalency and pr service

 meta/classes/buildhistory.bbclass | 49 --
 meta/classes/kernel.bbclass   |  5 ++-
 meta/classes/package.bbclass  | 58 ++-
 meta/conf/bitbake.conf|  1 +
 4 files changed, 77 insertions(+), 36 deletions(-)

-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141917): 
https://lists.openembedded.org/g/openembedded-core/message/141917
Mute This Topic: https://lists.openembedded.org/mt/76458842/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v6 2/3] kernel.bbclass: Move away from calling package_get_auto_pr

2020-08-27 Thread Mark Hatle
...instead we call read_subpackage_metadata.

Calling package_get_auto_pr *should* result in the same PKGV AUTOINC
replacement.  However, it will also end up changing PKGR differently
then do_package as the BB_TASKHASH used will be for the wrong task.

Generally this won't cause any real-world issue, but it could cause
problems.

Moving to read_subpackage_metadata ensures that the values used
in do_package will be read in and used for kernel deployment.

Signed-off-by: Mark Hatle 
---
 meta/classes/kernel.bbclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 7869184b94..14c22da306 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -748,7 +748,10 @@ kernel_do_deploy() {
done
fi
 }
-do_deploy[prefuncs] += "package_get_auto_pr"
+
+# We deploy to filenames that include PKGV and PKGR, read the saved data to
+# ensure we get the right values for both
+do_deploy[prefuncs] += "read_subpackage_metadata"
 
 addtask deploy after do_populate_sysroot do_packagedata
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141919): 
https://lists.openembedded.org/g/openembedded-core/message/141919
Mute This Topic: https://lists.openembedded.org/mt/76458846/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v6 3/3] package.bbclass: hash equivalency and pr service

2020-08-27 Thread Mark Hatle
When the PR service is enabled a number of small changes may happen
to variables.  In the do_package step a call to package_get_auto_pr
will end up setting PRAUTO and modifying PKGV (if AUTOINC is there).

PRAUTO is then used by EXTENDPRAUTO, which is then used to generate
PKGR.

Since this behavior typically happens BEFORE the BB_UNIHASH is
calculated for do_package, we need a way to defer the expansion
until after we have the unihash value.

Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded
to placeholder values is the easiest way to deal with this.  All other
variables are expanded as expected.

In the next task, typically do_packagedata, we will then use the
UNIHASH from the do_package to get the PR (AUTOPR) as well as
generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC).

The do_packagedata then translates the placeholders to the final values
when copying the data from pkgdata to pkgdata-pdata-input.

Signed-off-by: Mark Hatle 
---
 meta/classes/package.bbclass | 58 +++-
 meta/conf/bitbake.conf   |  1 +
 2 files changed, 51 insertions(+), 8 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7a36262eb6..0a4e9d13fa 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -7,7 +7,7 @@
 #
 # There are the following default steps but PACKAGEFUNCS can be extended:
 #
-# a) package_get_auto_pr - get PRAUTO from remote PR service
+# a) package_convert_pr_autoinc - convert AUTOINC in PKGV to 
${PRSERV_PV_AUTOINC}
 #
 # b) perform_packagecopy - Copy D into PKGD
 #
@@ -664,12 +664,20 @@ def runtime_mapping_rename (varname, pkg, d):
 #bb.note("%s after: %s" % (varname, d.getVar(varname)))
 
 #
-# Package functions suitable for inclusion in PACKAGEFUNCS
+# Used by do_packagedata (and possibly other routines post do_package)
 #
 
+package_get_auto_pr[vardepsexclude] = "BB_TASKDEPDATA"
 python package_get_auto_pr() {
 import oe.prservice
-import re
+
+def get_do_package_hash(pn):
+if d.getVar("BB_RUNTASK") != "do_package":
+taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+for dep in taskdepdata:
+if taskdepdata[dep][1] == "do_package" and taskdepdata[dep][0] 
== pn:
+return taskdepdata[dep][6]
+return None
 
 # Support per recipe PRSERV_HOST
 pn = d.getVar('PN')
@@ -681,15 +689,22 @@ python package_get_auto_pr() {
 
 # PR Server not active, handle AUTOINC
 if not d.getVar('PRSERV_HOST'):
-if 'AUTOINC' in pkgv:
-d.setVar("PKGV", pkgv.replace("AUTOINC", "0"))
+d.setVar("PRSERV_PV_AUTOINC", "0")
 return
 
 auto_pr = None
 pv = d.getVar("PV")
 version = d.getVar("PRAUTOINX")
 pkgarch = d.getVar("PACKAGE_ARCH")
-checksum = d.getVar("BB_TASKHASH")
+checksum = get_do_package_hash(pn)
+
+# If do_package isn't in the dependencies, we can't get the checksum...
+if not checksum:
+bb.warn('Task %s requested do_package unihash, but it was not 
available.' % d.getVar('BB_RUNTASK'))
+#taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+#for dep in taskdepdata:
+#bb.warn('%s:%s = %s' % (taskdepdata[dep][0], taskdepdata[dep][1], 
taskdepdata[dep][6]))
+return
 
 if d.getVar('PRSERV_LOCKDOWN'):
 auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or 
d.getVar('PRAUTO_' + version) or None
@@ -707,7 +722,7 @@ python package_get_auto_pr() {
 srcpv = bb.fetch2.get_srcrev(d)
 base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
 value = conn.getPR(base_ver, pkgarch, srcpv)
-d.setVar("PKGV", pkgv.replace("AUTOINC", str(value)))
+d.setVar("PRSERV_PV_AUTOINC", str(value))
 
 auto_pr = conn.getPR(version, pkgarch, checksum)
 except Exception as e:
@@ -717,6 +732,22 @@ python package_get_auto_pr() {
 d.setVar('PRAUTO',str(auto_pr))
 }
 
+#
+# Package functions suitable for inclusion in PACKAGEFUNCS
+#
+
+python package_convert_pr_autoinc() {
+pkgv = d.getVar("PKGV")
+
+# Adjust pkgv as necessary...
+if 'AUTOINC' in pkgv:
+d.setVar("PKGV", pkgv.replace("AUTOINC", "${PRSERV_PV_AUTOINC}"))
+
+# Change PRSERV_PV_AUTOINC and EXTENDPRAUTO usage to special values
+d.setVar('PRSERV_PV_AUTOINC', '@PRSERV_PV_AUTOINC@')
+d.setVar('EXTENDPRAUTO', '@EXTENDPRAUTO@')
+}
+
 LOCALEBASEPN ??= "${PN}"
 
 python package_do_split_locales() {
@@ -2335,7 +2366,7 @@ python do_package () {
 package_qa_handle_error("var-undefined", msg, d)
 return
 
-bb.build.exec_func("package_get_auto_pr", d)

[OE-core] [master][PATCH v6 1/3] buildhistory.bbclass: Rework to use read_subpackage_metadata

2020-08-27 Thread Mark Hatle
Using this mechanism ensures that we have a single point to implement
the loading of the package and subpackage meta data.  This also then
allows the buildhistory class to use the regular datastore vs it's
own custom arrays for processing history items.

Signed-off-by: Mark Hatle 
---
 meta/classes/buildhistory.bbclass | 49 ++-
 1 file changed, 22 insertions(+), 27 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass 
b/meta/classes/buildhistory.bbclass
index 805e976ac5..fbdb1d3161 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -258,20 +258,15 @@ python buildhistory_emit_pkghistory() {
 rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') 
or ""))
 write_recipehistory(rcpinfo, d)
 
-pkgdest = d.getVar('PKGDEST')
+bb.build.exec_func("read_subpackage_metadata", d)
+
 for pkg in packagelist:
-pkgdata = {}
-with open(os.path.join(pkgdata_dir, 'runtime', pkg)) as f:
-for line in f.readlines():
-item = line.rstrip('\n').split(': ', 1)
-key = item[0]
-if key.endswith('_' + pkg):
-key = key[:-len(pkg)-1]
-pkgdata[key] = 
item[1].encode('latin-1').decode('unicode_escape')
-
-pkge = pkgdata.get('PKGE', '0')
-pkgv = pkgdata['PKGV']
-pkgr = pkgdata['PKGR']
+localdata = d.createCopy()
+localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
+
+pkge = localdata.getVar("PKGE") or '0'
+pkgv = localdata.getVar("PKGV")
+pkgr = localdata.getVar("PKGR")
 #
 # Find out what the last version was
 # Make sure the version did not decrease
@@ -288,31 +283,31 @@ python buildhistory_emit_pkghistory() {
 
 pkginfo = PackageInfo(pkg)
 # Apparently the version can be different on a per-package basis (see 
Python)
-pkginfo.pe = pkgdata.get('PE', '0')
-pkginfo.pv = pkgdata['PV']
-pkginfo.pr = pkgdata['PR']
-pkginfo.pkg = pkgdata['PKG']
+pkginfo.pe = localdata.getVar("PE") or '0'
+pkginfo.pv = localdata.getVar("PV")
+pkginfo.pr = localdata.getVar("PR")
+pkginfo.pkg = localdata.getVar("PKG")
 pkginfo.pkge = pkge
 pkginfo.pkgv = pkgv
 pkginfo.pkgr = pkgr
-pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RPROVIDES', "")))
-pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RDEPENDS', "")))
-pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RRECOMMENDS', "")))
-pkginfo.rsuggests = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RSUGGESTS', "")))
-pkginfo.rreplaces = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RREPLACES', "")))
-pkginfo.rconflicts = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RCONFLICTS', "")))
-pkginfo.files = oe.utils.squashspaces(pkgdata.get('FILES', ""))
+pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RPROVIDES") or ""))
+pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RDEPENDS") or ""))
+pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RRECOMMENDS") or ""))
+pkginfo.rsuggests = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RSUGGESTS") or ""))
+pkginfo.replaces = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RREPLACES") or ""))
+pkginfo.rconflicts = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RCONFLICTS") or ""))
+pkginfo.files = oe.utils.squashspaces(localdata.getVar("FILES") or "")
 for filevar in pkginfo.filevars:
-pkginfo.filevars[filevar] = pkgdata.get(filevar, "")
+pkginfo.filevars[filevar] = localdata.getVar(filevar) or ""
 
 # Gather information about packaged files
-val = pkgdata.get('FILES_INFO', '')
+val = localdata.getVar('FILES_INFO') or ''
 dictval = json.loads(val)
 filelist = list(dictval.keys())
 filelist.sort()
 pkginfo.filelist = " ".join([shlex.quote(x) for x in filelist])
 
-pkginfo.size = int(pkgdata['PKGSIZE'])
+pkginfo.size = int(localdata.getVar('PKGSIZE') or '0')
 
 write_pkghistory(pkginfo, d)
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141918): 
https://lists.openembedded.org/g/openembedded-core/message/141918
Mute This Topic: https://lists.openembedded.org/mt/76458843/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [master][PATCH v2 0/6] Allow PR Service and hash equiv togther

2020-08-26 Thread Mark Hatle


On 8/26/20 6:48 AM, Richard Purdie wrote:
> On Wed, 2020-08-26 at 06:27 -0500, Mark Hatle wrote:
>> v2:
>>
>> Most comments have been addressed to create a v2 version.  I've refactored
>> the commits to make a few things more clear, basically moving code around
>> and fixing minor issues BEFORE the big patch.
>>
>> Before there were two patches that together implemented the PR Serv/Hash
>> work.  This has been combined into a single patch and the oe_nohash stuff
>> has simply been removed as no longer applicable.
>>
>> The only comment that was NOT addressed in this was the suggestion to
>> sed the pkgdata files in do_packagedata.  I'm hesitent to do this as
>> sed with ${...} in them has proven to be fragile for me in the past.  If
>> we decide that is necessary, I'd suggest we start with this set and then
>> make that change after this proves to work (or not).
> 
> Sorry to keep on about this but I am worried about code readability and
> trying not to let the code get too unreadable (we have to try).
> 
> Your shell expansion issue is a very valid concern. How about setting
> the variables we expand late to "@@" and then using that as the
> expression to sed later?

This was actually the first thing I tried, and it was a complete failure.  The
problem is that all of the variables that we play with (not just PKGV/PKGR, as
this impacts the RRECOMMENDS, RDEPENDS, etc..) all need this replacement done.

Then they reference PKGR, which references EXTENDPRAUTO, which is a python
variable which references PRAUTO which is what we actually care about.

So really what we should be doing is putting in 'PRAUTO' (@PRAUTO@) as the
replacement value and then replacing that later -- but we can't just do that
since if it's blank then EXTENDPRAUTO works differently then if it's set.

So we would need to wrap @EXTENDPRAUTO@, which "works".. but then we have to be
able to expand it on the reading of the data as well.  Which means modifications
of every routine that reads the variables from the pkgdata store.  But the
various places where the variables are read in don't have access to the list of
variables to expand -- so we might have to assume @...@ is used, which may work
but I'm still worried about how complicated it gets.

With the current usage, the only place I've seen issues (buildhistory) is where
they read the pkgdata directly and it never ends up in a datastore.  Assuming it
goes into a datastore then it's automatically expanded.  (Other then
buildhistory, I think all of the reader routines are located in
lib/oe/packagedata.py -- so we should be able to scan for this stuff in the
general case.)

(I also don't want to move EXTENDPRAUTO into say do_package, because I believe
the implementation of this is going to need to change in the future to support
the use-case where a standard distribution is produced, a user consumes it --
modifies a package and provides a variant of a few patches and just wants to
extend the EXTENDPRAUTO with additional release information.  I.e. initial
version of EXTENDPRAUTO is .1, where the user's customization makes it .1.1...)

> I'm also wondering if we should just inject the @@ change (instead
> of the delVar) into d directly at the start of do_package itself, where
> the existing bb.build.exec_func("package_get_auto_pr", d) call is?
> 
> That would then remove all the confusing localdata changes?

Part of the reason I moved the change to a function as I thought it made it
easier to read and understand the why.

--Mark

> Cheers,
> 
> Richard
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141855): 
https://lists.openembedded.org/g/openembedded-core/message/141855
Mute This Topic: https://lists.openembedded.org/mt/76426267/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [master][PATCH v2 4/6] package.bbclass: Move package_get_auto_pr to packagedata.bbclass

2020-08-26 Thread Mark Hatle


On 8/26/20 6:44 AM, Richard Purdie wrote:
> On Wed, 2020-08-26 at 06:27 -0500, Mark Hatle wrote:
>> Running package_get_auto_pr during do_package is 'too early' when we
>> need to evaluate the do_package output to get it's unihash to look up
>> the right value in the PR database.
>>
>> Move this function packagedata.bbclass (no changes yet), as 
> 
> The "no changes yet" isn't true as the new function is using
> BB_TASKDEPDATA?

Sorry, that's a merge error on my part.  I'll get that fixed.  That move to
taskdepdata was supposed to be in the final patch.

--Mark

> Cheers,
> 
> Richard
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141854): 
https://lists.openembedded.org/g/openembedded-core/message/141854
Mute This Topic: https://lists.openembedded.org/mt/76426263/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v2 2/6] kernel.bbclass: Remove do_install[prefunc] no longer needed

2020-08-26 Thread Mark Hatle
Prior work has refactored the do_install task multiple times, and any
references to PKGV and PKGR (even indirect ones) have been removed.

Signed-off-by: Mark Hatle 
---
 meta/classes/kernel.bbclass | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index cba77daa7a..7869184b94 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -416,7 +416,6 @@ kernel_do_install() {
install -d ${D}${sysconfdir}/modules-load.d
install -d ${D}${sysconfdir}/modprobe.d
 }
-do_install[prefuncs] += "package_get_auto_pr"
 
 # Must be ran no earlier than after do_kernel_checkout or else Makefile won't 
be in ${S}/Makefile
 do_kernel_version_sanity_check() {
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141846): 
https://lists.openembedded.org/g/openembedded-core/message/141846
Mute This Topic: https://lists.openembedded.org/mt/76426264/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v2 4/6] package.bbclass: Move package_get_auto_pr to packagedata.bbclass

2020-08-26 Thread Mark Hatle
Running package_get_auto_pr during do_package is 'too early' when we need
to evaluate the do_package output to get it's unihash to look up the
right value in the PR database.

Move this function packagedata.bbclass (no changes yet), as subsequent
patches will refactor this function and the order in which it's called.

Signed-off-by: Mark Hatle 
---
 meta/classes/package.bbclass | 50 ---
 meta/classes/packagedata.bbclass | 59 
 2 files changed, 59 insertions(+), 50 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7a36262eb6..d44c359e94 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -667,56 +667,6 @@ def runtime_mapping_rename (varname, pkg, d):
 # Package functions suitable for inclusion in PACKAGEFUNCS
 #
 
-python package_get_auto_pr() {
-import oe.prservice
-import re
-
-# Support per recipe PRSERV_HOST
-pn = d.getVar('PN')
-host = d.getVar("PRSERV_HOST_" + pn)
-if not (host is None):
-d.setVar("PRSERV_HOST", host)
-
-pkgv = d.getVar("PKGV")
-
-# PR Server not active, handle AUTOINC
-if not d.getVar('PRSERV_HOST'):
-if 'AUTOINC' in pkgv:
-d.setVar("PKGV", pkgv.replace("AUTOINC", "0"))
-return
-
-auto_pr = None
-pv = d.getVar("PV")
-version = d.getVar("PRAUTOINX")
-pkgarch = d.getVar("PACKAGE_ARCH")
-checksum = d.getVar("BB_TASKHASH")
-
-if d.getVar('PRSERV_LOCKDOWN'):
-auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or 
d.getVar('PRAUTO_' + version) or None
-if auto_pr is None:
-bb.fatal("Can NOT get PRAUTO from lockdown exported file")
-d.setVar('PRAUTO',str(auto_pr))
-return
-
-try:
-conn = d.getVar("__PRSERV_CONN")
-if conn is None:
-conn = oe.prservice.prserv_make_conn(d)
-if conn is not None:
-if "AUTOINC" in pkgv:
-srcpv = bb.fetch2.get_srcrev(d)
-base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
-value = conn.getPR(base_ver, pkgarch, srcpv)
-d.setVar("PKGV", pkgv.replace("AUTOINC", str(value)))
-
-auto_pr = conn.getPR(version, pkgarch, checksum)
-except Exception as e:
-bb.fatal("Can NOT get PRAUTO, exception %s" %  str(e))
-if auto_pr is None:
-bb.fatal("Can NOT get PRAUTO from remote PR service")
-d.setVar('PRAUTO',str(auto_pr))
-}
-
 LOCALEBASEPN ??= "${PN}"
 
 python package_do_split_locales() {
diff --git a/meta/classes/packagedata.bbclass b/meta/classes/packagedata.bbclass
index a903e5cfd2..5475d5c0da 100644
--- a/meta/classes/packagedata.bbclass
+++ b/meta/classes/packagedata.bbclass
@@ -32,3 +32,62 @@ python read_subpackage_metadata () {
 else:
 d.setVar(key, sdata[key], parsing=True)
 }
+
+package_get_auto_pr[vardepsexclude] = "BB_TASKDEPDATA"
+python package_get_auto_pr() {
+import oe.prservice
+import re
+
+def get_do_package_hash(pn):
+if d.getVar("BB_RUNTASK") != "do_package":
+taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+for dep in taskdepdata:
+if taskdepdata[dep][1] == "do_package" and taskdepdata[dep][0] 
== pn:
+return taskdepdata[dep][6]
+return d.getVar("BB_TASKHASH")
+
+# Support per recipe PRSERV_HOST
+pn = d.getVar('PN')
+host = d.getVar("PRSERV_HOST_" + pn)
+if not (host is None):
+d.setVar("PRSERV_HOST", host)
+
+pkgv = d.getVar("PKGV")
+
+# PR Server not active, handle AUTOINC
+if not d.getVar('PRSERV_HOST'):
+if 'AUTOINC' in pkgv:
+d.setVar("PKGV", pkgv.replace("AUTOINC", "0"))
+return
+
+auto_pr = None
+pv = d.getVar("PV")
+version = d.getVar("PRAUTOINX")
+pkgarch = d.getVar("PACKAGE_ARCH")
+checksum = get_do_package_hash(pn)
+
+if d.getVar('PRSERV_LOCKDOWN'):
+auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or 
d.getVar('PRAUTO_' + version) or None
+if auto_pr is None:
+bb.fatal("Can NOT get PRAUTO from lockdown exported file")
+d.setVar('PRAUTO',str(auto_pr))
+return
+
+try:
+conn = d.getVar("__PRSERV_CONN")
+if conn is None:
+conn = oe.prservice.prserv_make_conn(d)
+if conn is not None:
+if "AUTOINC" in pkgv:
+srcpv = bb.fetch2.get_srcrev(d)
+base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
+v

[OE-core] [master][PATCH v2 0/6] Allow PR Service and hash equiv togther

2020-08-26 Thread Mark Hatle
v2:

Most comments have been addressed to create a v2 version.  I've refactored
the commits to make a few things more clear, basically moving code around
and fixing minor issues BEFORE the big patch.

Before there were two patches that together implemented the PR Serv/Hash
work.  This has been combined into a single patch and the oe_nohash stuff
has simply been removed as no longer applicable.

The only comment that was NOT addressed in this was the suggestion to
sed the pkgdata files in do_packagedata.  I'm hesitent to do this as
sed with ${...} in them has proven to be fragile for me in the past.  If
we decide that is necessary, I'd suggest we start with this set and then
make that change after this proves to work (or not).

v1:

Before PR service didn't work reliably with hash equivalency.  Generally
you ended up with results that may not be reproducible, even if you
started with the same PR service database and hash equivalency database.

Overtime, intermediate PR values would be created that would cause thing
to get out of sync in the case of certain rebuilds or other corner cases.

The set refactors the PR service to work along side the new hash equiv
system.  It moves the PR and AUTOINC lookup to AFTER the do_package task
is complete.  This allows us to use the do_package unihash for lookup.

Additionally this fixed a small issue with the kernel, where the PR value
could get incremented twice.  The fix is an artifact of the other changes
that cause us to only run the PR service work once per recipe.

This has been tested with the following workflow, which covers one of
the critical corner cases for me:

configure local.conf with:

   BB_HASHSERVE = "auto"
   BB_SIGNATURE_HANDLER = "OEEquivHash"
   
   PRSERV_HOST ??= "localhost:0"
   
   INHERIT += "reproducible_build"
   INHERIT += "buildhistory"

bitbake glibc linux-yocto

# Modify meta/recipes-core/glibc/glibc_2.32.bb, add a comment
# to the do_patch_append().  This will taint the hash of this
# function.

bitbake glibc linux-yocto

# The system should have detected the output was the same, and
# no proceed past do_package in glibc.  The kernel should not
# have built at all.

# Store/mv the tmp and sstate-cache from that build elsewhere
# repeat the run

bitbake glibc linux-yocto

# Compare the results of tmp/deploy//* between last
# and current run.
#
# The contents should be the same (filenames specifically).
#
# Also the kernel should be r0.0, not r0.1.

Note: if the hash equivalency database or PR server database (located
in the cache directory) is removed, the values may not be the same
as the previous run.

Additionally while testing the various package_*.bbclass files, it
was noted that package_tar.bbclass was not working the same way as
the others.  This was correct as a standalone patch.

Mark Hatle (6):
  package_tar.bbclass: Sync to the other package_* classes
  kernel.bbclass: Remove do_install[prefunc] no longer needed
  buildhistory.bbclass: Rework to use read_subpackage_metadata
  package.bbclass: Move package_get_auto_pr to packagedata.bbclass
  kernel.bbclass: Move away from calling package_get_auto_pr
  package.bbclass: hash equivalency and pr service

 meta/classes/buildhistory.bbclass |  49 ++---
 meta/classes/kernel.bbclass   |   6 +-
 meta/classes/nopackages.bbclass   |   1 +
 meta/classes/package.bbclass  | 111 ++
 meta/classes/package_tar.bbclass  |   6 +-
 meta/classes/packagedata.bbclass  |  68 ++
 meta/conf/bitbake.conf|   1 +
 7 files changed, 148 insertions(+), 94 deletions(-)

-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141848): 
https://lists.openembedded.org/g/openembedded-core/message/141848
Mute This Topic: https://lists.openembedded.org/mt/76426267/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v2 3/6] buildhistory.bbclass: Rework to use read_subpackage_metadata

2020-08-26 Thread Mark Hatle
Using this mechanism ensures that we have a single point to implement
the loading of the package and subpackage meta data.  This also then
allows the buildhistory class to use the regular datastore vs it's
own custom arrays for processing history items.

Signed-off-by: Mark Hatle 
---
 meta/classes/buildhistory.bbclass | 49 ++-
 1 file changed, 22 insertions(+), 27 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass 
b/meta/classes/buildhistory.bbclass
index 805e976ac5..fbdb1d3161 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -258,20 +258,15 @@ python buildhistory_emit_pkghistory() {
 rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') 
or ""))
 write_recipehistory(rcpinfo, d)
 
-pkgdest = d.getVar('PKGDEST')
+bb.build.exec_func("read_subpackage_metadata", d)
+
 for pkg in packagelist:
-pkgdata = {}
-with open(os.path.join(pkgdata_dir, 'runtime', pkg)) as f:
-for line in f.readlines():
-item = line.rstrip('\n').split(': ', 1)
-key = item[0]
-if key.endswith('_' + pkg):
-key = key[:-len(pkg)-1]
-pkgdata[key] = 
item[1].encode('latin-1').decode('unicode_escape')
-
-pkge = pkgdata.get('PKGE', '0')
-pkgv = pkgdata['PKGV']
-pkgr = pkgdata['PKGR']
+localdata = d.createCopy()
+localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
+
+pkge = localdata.getVar("PKGE") or '0'
+pkgv = localdata.getVar("PKGV")
+pkgr = localdata.getVar("PKGR")
 #
 # Find out what the last version was
 # Make sure the version did not decrease
@@ -288,31 +283,31 @@ python buildhistory_emit_pkghistory() {
 
 pkginfo = PackageInfo(pkg)
 # Apparently the version can be different on a per-package basis (see 
Python)
-pkginfo.pe = pkgdata.get('PE', '0')
-pkginfo.pv = pkgdata['PV']
-pkginfo.pr = pkgdata['PR']
-pkginfo.pkg = pkgdata['PKG']
+pkginfo.pe = localdata.getVar("PE") or '0'
+pkginfo.pv = localdata.getVar("PV")
+pkginfo.pr = localdata.getVar("PR")
+pkginfo.pkg = localdata.getVar("PKG")
 pkginfo.pkge = pkge
 pkginfo.pkgv = pkgv
 pkginfo.pkgr = pkgr
-pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RPROVIDES', "")))
-pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RDEPENDS', "")))
-pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RRECOMMENDS', "")))
-pkginfo.rsuggests = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RSUGGESTS', "")))
-pkginfo.rreplaces = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RREPLACES', "")))
-pkginfo.rconflicts = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RCONFLICTS', "")))
-pkginfo.files = oe.utils.squashspaces(pkgdata.get('FILES', ""))
+pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RPROVIDES") or ""))
+pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RDEPENDS") or ""))
+pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RRECOMMENDS") or ""))
+pkginfo.rsuggests = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RSUGGESTS") or ""))
+pkginfo.replaces = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RREPLACES") or ""))
+pkginfo.rconflicts = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RCONFLICTS") or ""))
+pkginfo.files = oe.utils.squashspaces(localdata.getVar("FILES") or "")
 for filevar in pkginfo.filevars:
-pkginfo.filevars[filevar] = pkgdata.get(filevar, "")
+pkginfo.filevars[filevar] = localdata.getVar(filevar) or ""
 
 # Gather information about packaged files
-val = pkgdata.get('FILES_INFO', '')
+val = localdata.getVar('FILES_INFO') or ''
 dictval = json.loads(val)
 filelist = list(dictval.keys())
 filelist.sort()
 pkginfo.filelist = " ".join([shlex.quote(x) for x in filelist])
 
-pkginfo.size = int(pkgdata['PKGSIZE'])
+pkginfo.size = int(localdata.getVar('PKGSIZE') or '0')
 
 write_pkghistory(pkginfo, d)
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141850): 
https://lists.openembedded.org/g/openembedded-core/message/141850
Mute This Topic: https://lists.openembedded.org/mt/76426269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v2 1/6] package_tar.bbclass: Sync to the other package_* classes

2020-08-26 Thread Mark Hatle
Sync up the anonymous python definition with the other package_*.bbclass
files.  This should make future maintenance easier, even though it has
no difference in behavior from what was there.

Additional, there was a missing deltask in the nopackages.bbclass related
to the package_tar which has been corrected.  This could cause problems on
native recipes when package_tar was enabled.

Signed-off-by: Mark Hatle 
---
 meta/classes/nopackages.bbclass  | 1 +
 meta/classes/package_tar.bbclass | 6 ++
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/classes/nopackages.bbclass b/meta/classes/nopackages.bbclass
index 559f5078bd..7a4f632d71 100644
--- a/meta/classes/nopackages.bbclass
+++ b/meta/classes/nopackages.bbclass
@@ -2,6 +2,7 @@ deltask do_package
 deltask do_package_write_rpm
 deltask do_package_write_ipk
 deltask do_package_write_deb
+deltask do_package_write_tar
 deltask do_package_qa
 deltask do_packagedata
 deltask do_package_setscene
diff --git a/meta/classes/package_tar.bbclass b/meta/classes/package_tar.bbclass
index ce3ab4c8e2..d6c1b306fc 100644
--- a/meta/classes/package_tar.bbclass
+++ b/meta/classes/package_tar.bbclass
@@ -57,10 +57,8 @@ python do_package_tar () {
 
 python () {
 if d.getVar('PACKAGES') != '':
-deps = (d.getVarFlag('do_package_write_tar', 'depends') or "").split()
-deps.append('tar-native:do_populate_sysroot')
-deps.append('virtual/fakeroot-native:do_populate_sysroot')
-d.setVarFlag('do_package_write_tar', 'depends', " ".join(deps))
+deps = ' tar-native:do_populate_sysroot 
virtual/fakeroot-native:do_populate_sysroot'
+d.appendVarFlag('do_package_write_tar', 'depends', deps)
 d.setVarFlag('do_package_write_tar', 'fakeroot', "1")
 }
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141847): 
https://lists.openembedded.org/g/openembedded-core/message/141847
Mute This Topic: https://lists.openembedded.org/mt/76426265/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v2 5/6] kernel.bbclass: Move away from calling package_get_auto_pr

2020-08-26 Thread Mark Hatle
...instead we call read_subpackage_metadata.

Calling package_get_auto_pr *should* result in the same PKGV AUTOINC
replacement.  However, it will also end up changing PKGR differently
then do_package as the BB_TASKHASH used will be for the wrong task.

Generally this won't cause any real-world issue, but it could cause
problems.

Moving to read_subpackage_metadata ensures that the values used
in do_package will be read in and used for kernel deployment.

Signed-off-by: Mark Hatle 
---
 meta/classes/kernel.bbclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 7869184b94..14c22da306 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -748,7 +748,10 @@ kernel_do_deploy() {
done
fi
 }
-do_deploy[prefuncs] += "package_get_auto_pr"
+
+# We deploy to filenames that include PKGV and PKGR, read the saved data to
+# ensure we get the right values for both
+do_deploy[prefuncs] += "read_subpackage_metadata"
 
 addtask deploy after do_populate_sysroot do_packagedata
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141851): 
https://lists.openembedded.org/g/openembedded-core/message/141851
Mute This Topic: https://lists.openembedded.org/mt/76426270/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH v2 6/6] package.bbclass: hash equivalency and pr service

2020-08-26 Thread Mark Hatle
When the PR service is enabled a number of small changes may happen
to variables.  In the do_package step a call to package_get_auto_pr
will end up setting PRAUTO and modifying PKGV (if AUTOINC is there).

PRAUTO is then used by EXTENDPRAUTO, which is then used to generate
PKGR.

Since this behavior typically happens BEFORE the BB_UNIHASH is
calculated for do_package, we need a way to defer the expansion
until after we have the unihash value.

Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded
is the easiest way to deal with this.  The majority of items are still
expanded as expected.

In the next task, typically do_packagedata, we will then use the
UNIHASH from the do_package to get the PR (AUTOPR) as well as
generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC).

Then, as required, the standard load routine, read_subpackage_metadata,
will then be used to read the values stored from do_package, and
do_packagedata which provide everything required for the expanded
PKGV and PKGR.

Note, we use read_subpackage_metadata, as PKGV and PKGR may be
set per sub-package with the way things are defined.  Due to this
they are not defined in a ${PN} file, but in the per package
files.

Signed-off-by: Mark Hatle 
---
 meta/classes/package.bbclass | 69 +---
 meta/classes/packagedata.bbclass | 19 ++---
 meta/conf/bitbake.conf   |  1 +
 3 files changed, 69 insertions(+), 20 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index d44c359e94..5570d4954c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -7,7 +7,7 @@
 #
 # There are the following default steps but PACKAGEFUNCS can be extended:
 #
-# a) package_get_auto_pr - get PRAUTO from remote PR service
+# a) package_convert_autoinc - convert AUTOINC in PKGV to ${PRSERV_PV_AUTOINC}
 #
 # b) perform_packagecopy - Copy D into PKGD
 #
@@ -648,11 +648,21 @@ def get_package_additional_metadata (pkg_type, d):
 metadata_fields = [field.strip() for field in oe.data.typed_value(key, 
d)]
 return "\n".join(metadata_fields).strip()
 
+# Remove the excluded variables from a copy of d
+# This ensures that any users will list these variables when expanded
+def exclude_pkgdata_vars(d):
+localdata = d.createCopy()
+for exclude_var in (d.getVar('PKGDATA_VARS_EXCLUDE') or "").split():
+localdata.delVar(exclude_var)
+return localdata
+
 def runtime_mapping_rename (varname, pkg, d):
 #bb.note("%s before: %s" % (varname, d.getVar(varname)))
 
 new_depends = {}
-deps = bb.utils.explode_dep_versions2(d.getVar(varname) or "")
+localdata = exclude_pkgdata_vars(d)
+
+deps = bb.utils.explode_dep_versions2(localdata.getVar(varname) or "")
 for depend, depversions in deps.items():
 new_depend = get_package_mapping(depend, pkg, d, depversions)
 if depend != new_depend:
@@ -667,6 +677,13 @@ def runtime_mapping_rename (varname, pkg, d):
 # Package functions suitable for inclusion in PACKAGEFUNCS
 #
 
+python package_convert_autoinc() {
+pkgv = d.getVar("PKGV")
+
+if 'AUTOINC' in pkgv:
+d.setVar("PKGV", pkgv.replace("AUTOINC", "${PRSERV_PV_AUTOINC}"))
+}
+
 LOCALEBASEPN ??= "${PN}"
 
 python package_do_split_locales() {
@@ -1436,8 +1453,10 @@ python package_fixsymlinks () {
 if found == False:
 bb.note("%s contains dangling symlink to %s" % (pkg, l))
 
+localdata = exclude_pkgdata_vars(d)
+
 for pkg in newrdepends:
-rdepends = bb.utils.explode_dep_versions2(d.getVar('RDEPENDS_' + pkg) 
or "")
+rdepends = bb.utils.explode_dep_versions2(localdata.getVar('RDEPENDS_' 
+ pkg) or "")
 for p in newrdepends[pkg]:
 if p not in rdepends:
 rdepends[p] = []
@@ -1460,6 +1479,12 @@ PKGDESTWORK = "${WORKDIR}/pkgdata"
 
 PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY 
RDEPENDS RPROVIDES RRECOMMENDS RSUGGESTS RREPLACES RCONFLICTS SECTION PKG 
ALLOW_EMPTY FILES CONFFILES FILES_INFO PACKAGE_ADD_METADATA pkg_postinst 
pkg_postrm pkg_preinst pkg_prerm"
 
+# When we write or use PKGDATA_VARS items, components of this may need
+# to be excluded from expansion.
+# Really we only care about PRAUTO, but EXTENDPRAUTO evaluates this, so
+# exclude it instead
+PKGDATA_VARS_EXCLUDE = "EXTENDPRAUTO PRSERV_PV_AUTOINC"
+
 python emit_pkgdata() {
 from glob import glob
 import json
@@ -1498,7 +1523,7 @@ fi
 scriptlet = "set -e\n" + "\n".join(scriptlet_split[0:])
 d.setVar('%s_%s' % (scriptlet_name, pkg), scriptlet)
 
-def write_if_exists(f, pkg, var):
+def write_if_exists(d, f, pkg, var):
 def encode(str):
 import codecs
 c = codecs.ge

Re: [OE-core] [master][PATCH 4/5] package / packagedata bbclass: Change the way PRAUTO and AUTOINC are handled

2020-08-25 Thread Mark Hatle


On 8/25/20 12:04 PM, Richard Purdie wrote:
> On Tue, 2020-08-25 at 11:52 -0500, Mark Hatle wrote:
>>
>> On 8/25/20 9:14 AM, Mark Hatle wrote:
>>>
>>> On 8/25/20 5:56 AM, Richard Purdie wrote:
>>>> On Mon, 2020-08-24 at 18:29 -0500, Mark Hatle wrote:
>>>>> A related change was also needed for the kernel.bbclass.  The
>>>>> kernel
>>>>>
>>>>> needs to get the AUTOINC values used by the various PV/PKGV,
>>>>> but with
>>>>> this new system we don't want to call auto_pr each time --
>>>>> instead
>>>>> call the read_subpackage_metadata to ensure a consistent
>>>>> result.
>>>>>
>>>>> This also fixes an issue in the old version where the kernel
>>>>> sometimes
>>>>> caused the PR to be incremented twice, as the auto_pr was
>>>>> called
>>>>> multiple times from different tasks.
>>>>
>>>> [...]
>>>>
>>>>> diff --git a/meta/classes/kernel.bbclass
>>>>> b/meta/classes/kernel.bbclass
>>>>> index e2ceb6a333..4449452065 100644
>>>>> --- a/meta/classes/kernel.bbclass
>>>>> +++ b/meta/classes/kernel.bbclass
>>>>> @@ -416,7 +416,7 @@ kernel_do_install() {
>>>>>   install -d ${D}${sysconfdir}/modules-load.d
>>>>>   install -d ${D}${sysconfdir}/modprobe.d
>>>>>  }
>>>>> -do_install[prefuncs] += "package_get_auto_pr"
>>>>> +do_install[prefuncs] += "read_subpackage_metadata"
>>
>> I talked with Bruce on this.  We see no reason do_install[prefuncs]
>> is required
>> any longer.
>>
>> It was when that line was written, but when the kernel was refactored
>> to permit multiple kernel builds and such the need for this went
>> away.
> 
> Ok, lets start with a patch to remove that call.
> 
>>>>>  # Must be ran no earlier than after do_kernel_checkout or else
>>>>> Makefile won't be in ${S}/Makefile
>>>>>  do_kernel_version_sanity_check() {
>>>>> @@ -749,7 +749,7 @@ kernel_do_deploy() {
>>>>>   done
>>>>>   fi
>>>>>  }
>>>>> -do_deploy[prefuncs] += "package_get_auto_pr"
>>>>> +do_deploy[prefuncs] += "read_subpackage_metadata"
>>>>>  
>>>>>  addtask deploy after do_populate_sysroot do_packagedata
>>>>
>>>> I don't think this can be right. I understand why the kernel was
>>>> expanding these early but at this point do_package hasn't
>>>> happened so
>>>> reading the subpackage metadata is a null op at best and
>>>> potentially
>>>> changes a lot more in the data store.
>>>>
>>>> I suspect this is not doing the right thing and will be something
>>>> that
>>>> comes back to bite us so we need something more careful here and
>>>> I'd
>>>> like to better understand exactly what change to the variables
>>>> do_install and do_deploy need.
>>>
>>> I don't know exactly why it's doing this.  But I believe it is so
>>> that the
>>> PV/PKGV is expanded (AUTOINC translated).  I don't see any other
>>> reason for it
>>> to be expanded in either do_install or do_deploy.
>>>
>>> Best I can find is:
>>>
>>> kernel-artifact-names.bbclass:KERNEL_ARTIFACT_NAME ?=
>>> "${PKGE}-${PKGV}-${PKGR}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
>>>
>>> (That is then used in other places.)
>>>
>>> So the PKGV replacement is desired to be that early.
>>>
>>> I'll look into it further.
>>
>> In do_deploy, I see multiple usages of the KERNEL_ARTIFACT_NAME
>> (sometimes with
>> alternative variables.. but in the end both PKGV and PKGR are used
>> here.)
>>
>> do_deploy is after do_package/do_packagedata (where the values are
>> assigned and
>> stored).  So it should be safe for us to use the
>> read_subpackage_metadata (or
>> something new if desired) to read back in the stored PKGV (AUTOINC)
>> and PKGR
>> (PRAUTO) values.
> 
> That sounds reasonable. FWIW do_deploy doesn't have to be after
> do_package but we can arrange that in the kernel case.

Still working on the refactor, but the kernel.bbclass "do_deploy" runs after
do_packagedata, and do_packagedata is when the PKGV/PKGR dynamic items are
written.  So it's safe to do it here.

All of the other users I found were after do_package (and didn't care about PKGV
and PKGR) or were after do_packagedata.

> I'd prefer to have a more granular function to call here than
> read_subpackage_metadata, just to make it clear what is happening and
> why.

My concern is that if we don't load all of the subpackage metadata, that the
PKGV and PKGR won't be loaded -exactly- they were they were defined in
(combination of) do_package and do_packagedata.

We can limit the load to just what was written by do_packagedata, but I don't
believe this is significant less efficient or will cause issues... and doing it
this way ensures we have a single routine for future modifications.

--Mark

> Cheers,
> 
> Richard
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141835): 
https://lists.openembedded.org/g/openembedded-core/message/141835
Mute This Topic: https://lists.openembedded.org/mt/76396872/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [master][PATCH 4/5] package / packagedata bbclass: Change the way PRAUTO and AUTOINC are handled

2020-08-25 Thread Mark Hatle


On 8/25/20 9:14 AM, Mark Hatle wrote:
> 
> 
> On 8/25/20 5:56 AM, Richard Purdie wrote:
>> On Mon, 2020-08-24 at 18:29 -0500, Mark Hatle wrote:
>>> A related change was also needed for the kernel.bbclass.  The kernel
>>>
>>> needs to get the AUTOINC values used by the various PV/PKGV, but with
>>> this new system we don't want to call auto_pr each time -- instead
>>> call the read_subpackage_metadata to ensure a consistent result.
>>>
>>> This also fixes an issue in the old version where the kernel
>>> sometimes
>>> caused the PR to be incremented twice, as the auto_pr was called
>>> multiple times from different tasks.
>>
>> [...]
>>
>>>
>>> diff --git a/meta/classes/kernel.bbclass
>>> b/meta/classes/kernel.bbclass
>>> index e2ceb6a333..4449452065 100644
>>> --- a/meta/classes/kernel.bbclass
>>> +++ b/meta/classes/kernel.bbclass
>>> @@ -416,7 +416,7 @@ kernel_do_install() {
>>> install -d ${D}${sysconfdir}/modules-load.d
>>> install -d ${D}${sysconfdir}/modprobe.d
>>>  }
>>> -do_install[prefuncs] += "package_get_auto_pr"
>>> +do_install[prefuncs] += "read_subpackage_metadata"

I talked with Bruce on this.  We see no reason do_install[prefuncs] is required
any longer.

It was when that line was written, but when the kernel was refactored to permit
multiple kernel builds and such the need for this went away.

>>>  # Must be ran no earlier than after do_kernel_checkout or else
>>> Makefile won't be in ${S}/Makefile
>>>  do_kernel_version_sanity_check() {
>>> @@ -749,7 +749,7 @@ kernel_do_deploy() {
>>> done
>>> fi
>>>  }
>>> -do_deploy[prefuncs] += "package_get_auto_pr"
>>> +do_deploy[prefuncs] += "read_subpackage_metadata"
>>>  
>>>  addtask deploy after do_populate_sysroot do_packagedata
>>
>> I don't think this can be right. I understand why the kernel was
>> expanding these early but at this point do_package hasn't happened so
>> reading the subpackage metadata is a null op at best and potentially
>> changes a lot more in the data store.
>>
>> I suspect this is not doing the right thing and will be something that
>> comes back to bite us so we need something more careful here and I'd
>> like to better understand exactly what change to the variables
>> do_install and do_deploy need.
> 
> I don't know exactly why it's doing this.  But I believe it is so that the
> PV/PKGV is expanded (AUTOINC translated).  I don't see any other reason for it
> to be expanded in either do_install or do_deploy.
> 
> Best I can find is:
> 
> kernel-artifact-names.bbclass:KERNEL_ARTIFACT_NAME ?=
> "${PKGE}-${PKGV}-${PKGR}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
> 
> (That is then used in other places.)
> 
> So the PKGV replacement is desired to be that early.
> 
> I'll look into it further.

In do_deploy, I see multiple usages of the KERNEL_ARTIFACT_NAME (sometimes with
alternative variables.. but in the end both PKGV and PKGR are used here.)

do_deploy is after do_package/do_packagedata (where the values are assigned and
stored).  So it should be safe for us to use the read_subpackage_metadata (or
something new if desired) to read back in the stored PKGV (AUTOINC) and PKGR
(PRAUTO) values.

--Mark

> --Mark
> 
>> Cheers,
>>
>> Richard
>>
>>
>> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141829): 
https://lists.openembedded.org/g/openembedded-core/message/141829
Mute This Topic: https://lists.openembedded.org/mt/76396872/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [master][PATCH 2/5] hash equivalency and pr service

2020-08-25 Thread Mark Hatle


On 8/25/20 7:50 AM, Joshua Watt wrote:
> On Mon, Aug 24, 2020 at 6:29 PM Mark Hatle
>  wrote:
>>
>> When the PR service is enabled a number of small changes happen to variables.
>>
>> During the do_package task, EXTENDPRAUTO will get a value placed into it that
>> is intended to be used to extend the PR.  This value will get written to
>> various intermediate files and will result in the computed hash never being
>> equivalent.
>>
>> To resolve this issue, we create a new file with the extension ".oe_nohash".
>> This extension contains various values that need to be passed from task to
>> task, but NOT calculated within the scope of the task hash.  The items
>> placed into the nohash are captured in package.bbclass in PKGDATA_VAR_NOHASH.
>>
>> The PKGDATA_VAR_NOHASH is also used to prevent expansion of the value when
>> users are written to the various pkgdata files.  This expansion is prevented
>> by removeing the variables, resulting in the system printing the literal
>> ${VAR} when used.
>>
>> Additionally package dependencies rely on variables that also eventually
>> include EXTENDPRAUTO.  So instead of resolving the variable at getVar time,
>> we keep them as references by using the same technique as above.
>>
>> Due to this change, buildhistory needs a few minor modifications to
>> know how to deal with EXTENDPRAUTO not always being available for comparison.
>>
>> Signed-off-by: Mark Hatle 
>> ---
>>  meta/classes/buildhistory.bbclass | 29 ++
>>  meta/classes/package.bbclass  | 63 ---
>>  meta/lib/oe/packagedata.py|  8 +++-
>>  meta/lib/oe/sstatesig.py  |  2 +
>>  4 files changed, 80 insertions(+), 22 deletions(-)
>>
>> diff --git a/meta/classes/buildhistory.bbclass 
>> b/meta/classes/buildhistory.bbclass
>> index 805e976ac5..2bccdfc953 100644
>> --- a/meta/classes/buildhistory.bbclass
>> +++ b/meta/classes/buildhistory.bbclass
>> @@ -99,6 +99,7 @@ python buildhistory_emit_pkghistory() {
>>  import json
>>  import shlex
>>  import errno
>> +import oe.packagedata
>>
>>  pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE')
>>  oldpkghistdir = d.getVar('BUILDHISTORY_OLD_DIR_PACKAGE')
>> @@ -127,6 +128,7 @@ python buildhistory_emit_pkghistory() {
>>  self.pkge = ""
>>  self.pkgv = ""
>>  self.pkgr = ""
>> +self.extendprauto = ""
>>  self.size = 0
>>  self.depends = ""
>>  self.rprovides = ""
>> @@ -163,6 +165,8 @@ python buildhistory_emit_pkghistory() {
>>  pkginfo.pkgv = value
>>  elif name == "PKGR":
>>  pkginfo.pkgr = value
>> +elif name == "EXTENDPRAUTO":
>> +pkginfo.extendprauto = value
>>  elif name == "RPROVIDES":
>>  pkginfo.rprovides = value
>>  elif name == "RDEPENDS":
>> @@ -260,18 +264,24 @@ python buildhistory_emit_pkghistory() {
>>
>>  pkgdest = d.getVar('PKGDEST')
>>  for pkg in packagelist:
>> -pkgdata = {}
>> -with open(os.path.join(pkgdata_dir, 'runtime', pkg)) as f:
>> -for line in f.readlines():
>> -item = line.rstrip('\n').split(': ', 1)
>> -key = item[0]
>> -if key.endswith('_' + pkg):
>> -key = key[:-len(pkg)-1]
>> -pkgdata[key] = 
>> item[1].encode('latin-1').decode('unicode_escape')
>> +pkgdata = oe.packagedata.read_pkgdatafile(os.path.join(pkgdata_dir, 
>> 'runtime', pkg))
>> +
>> +npkgdata = {}
>> +for key in pkgdata.keys():
>> +if key.endswith('_' + pkg):
>> +nkey = key[:-len(pkg)-1]
>> +npkgdata[nkey] = pkgdata[key]
>> +
>> +for key in npkgdata.keys():
>> +  pkgdata[key] = npkgdata[key]
>>
>>  pkge = pkgdata.get('PKGE', '0')
>>  pkgv = pkgdata['PKGV']
>>  pkgr = pkgdata['PKGR']
>> +try:
>> +extendprauto = pkgdata['EXTENDPRAUTO']
>> +except IndexError:
>> +extendprauto = d.getVar('EXTENDPRAUTO')
>>  #
>>  # Find out what the last version was
>>  # Make sure the version did not decrease

Re: [OE-core] [master][PATCH 4/5] package / packagedata bbclass: Change the way PRAUTO and AUTOINC are handled

2020-08-25 Thread Mark Hatle


On 8/25/20 5:56 AM, Richard Purdie wrote:
> On Mon, 2020-08-24 at 18:29 -0500, Mark Hatle wrote:
>> A related change was also needed for the kernel.bbclass.  The kernel
>>
>> needs to get the AUTOINC values used by the various PV/PKGV, but with
>> this new system we don't want to call auto_pr each time -- instead
>> call the read_subpackage_metadata to ensure a consistent result.
>>
>> This also fixes an issue in the old version where the kernel
>> sometimes
>> caused the PR to be incremented twice, as the auto_pr was called
>> multiple times from different tasks.
> 
> [...]
> 
>>
>> diff --git a/meta/classes/kernel.bbclass
>> b/meta/classes/kernel.bbclass
>> index e2ceb6a333..4449452065 100644
>> --- a/meta/classes/kernel.bbclass
>> +++ b/meta/classes/kernel.bbclass
>> @@ -416,7 +416,7 @@ kernel_do_install() {
>>  install -d ${D}${sysconfdir}/modules-load.d
>>  install -d ${D}${sysconfdir}/modprobe.d
>>  }
>> -do_install[prefuncs] += "package_get_auto_pr"
>> +do_install[prefuncs] += "read_subpackage_metadata"
>>  
>>  # Must be ran no earlier than after do_kernel_checkout or else
>> Makefile won't be in ${S}/Makefile
>>  do_kernel_version_sanity_check() {
>> @@ -749,7 +749,7 @@ kernel_do_deploy() {
>>  done
>>  fi
>>  }
>> -do_deploy[prefuncs] += "package_get_auto_pr"
>> +do_deploy[prefuncs] += "read_subpackage_metadata"
>>  
>>  addtask deploy after do_populate_sysroot do_packagedata
> 
> I don't think this can be right. I understand why the kernel was
> expanding these early but at this point do_package hasn't happened so
> reading the subpackage metadata is a null op at best and potentially
> changes a lot more in the data store.
> 
> I suspect this is not doing the right thing and will be something that
> comes back to bite us so we need something more careful here and I'd
> like to better understand exactly what change to the variables
> do_install and do_deploy need.

I don't know exactly why it's doing this.  But I believe it is so that the
PV/PKGV is expanded (AUTOINC translated).  I don't see any other reason for it
to be expanded in either do_install or do_deploy.

Best I can find is:

kernel-artifact-names.bbclass:KERNEL_ARTIFACT_NAME ?=
"${PKGE}-${PKGV}-${PKGR}-${MACHINE}${IMAGE_VERSION_SUFFIX}"

(That is then used in other places.)

So the PKGV replacement is desired to be that early.

I'll look into it further.

--Mark

> Cheers,
> 
> Richard
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141822): 
https://lists.openembedded.org/g/openembedded-core/message/141822
Mute This Topic: https://lists.openembedded.org/mt/76396872/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] ✗ patchtest: failure for Allow PR Service and hash equiv together

2020-08-24 Thread Mark Hatle


On 8/24/20 6:32 PM, Patchwork wrote:
> == Series Details ==
> 
> Series: Allow PR Service and hash equiv together
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/25758/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Patch[master,2/5] hash equivalency and pr service
>  Issue Shortlog does not follow expected format 
> [test_shortlog_format] 
>   Suggested fixCommit shortlog (first line of commit message) should 
> follow the format ": "

In this case, I think what I did was acceptable.  If reviewers disagree I'll
change the format.

> * Issue Errors in your Python code were encountered [test_pylint] 
>   Suggested fixCorrect the lines introduced by your patch
>   Output   Please, fix the listed issues:
>meta/lib/oe/packagedata.py does not exist

Now this one is _confusing_, it absolutely exists:

https://git.openembedded.org/openembedded-core/tree/meta/lib/oe/packagedata.py

Is the link engine broken?

> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.

DONE!

--Mark

> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
> 
> ---
> Guidelines: 
> https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141801): 
https://lists.openembedded.org/g/openembedded-core/message/141801
Mute This Topic: https://lists.openembedded.org/mt/76396938/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH 5/5] buildhistory.bbclass: Rework to use read_subpackage_metadata

2020-08-24 Thread Mark Hatle
Using this mechanism ensures that we have a single point to implement
the loading of the package and subpackage meta data.  This also then
allows the buildhistory class to use the regular datastore vs it's
own custom arrays for processing history items.

Without this refactoring PRSERV_PV_AUTOINC and sometimes PRAUTO do not
get expanded properly within the buildhistory which can leave to
error messages about package versions/releases going backwards.

Signed-off-by: Mark Hatle 
---
 meta/classes/buildhistory.bbclass | 60 +++
 1 file changed, 21 insertions(+), 39 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass 
b/meta/classes/buildhistory.bbclass
index 2bccdfc953..5d478f38b5 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -99,7 +99,6 @@ python buildhistory_emit_pkghistory() {
 import json
 import shlex
 import errno
-import oe.packagedata
 
 pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE')
 oldpkghistdir = d.getVar('BUILDHISTORY_OLD_DIR_PACKAGE')
@@ -128,7 +127,6 @@ python buildhistory_emit_pkghistory() {
 self.pkge = ""
 self.pkgv = ""
 self.pkgr = ""
-self.extendprauto = ""
 self.size = 0
 self.depends = ""
 self.rprovides = ""
@@ -165,8 +163,6 @@ python buildhistory_emit_pkghistory() {
 pkginfo.pkgv = value
 elif name == "PKGR":
 pkginfo.pkgr = value
-elif name == "EXTENDPRAUTO":
-pkginfo.extendprauto = value
 elif name == "RPROVIDES":
 pkginfo.rprovides = value
 elif name == "RDEPENDS":
@@ -262,26 +258,15 @@ python buildhistory_emit_pkghistory() {
 rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') 
or ""))
 write_recipehistory(rcpinfo, d)
 
-pkgdest = d.getVar('PKGDEST')
-for pkg in packagelist:
-pkgdata = oe.packagedata.read_pkgdatafile(os.path.join(pkgdata_dir, 
'runtime', pkg))
-
-npkgdata = {}
-for key in pkgdata.keys():
-if key.endswith('_' + pkg):
-nkey = key[:-len(pkg)-1]
-npkgdata[nkey] = pkgdata[key]
+bb.build.exec_func("read_subpackage_metadata", d)
 
-for key in npkgdata.keys():
-  pkgdata[key] = npkgdata[key]
+for pkg in packagelist:
+localdata = d.createCopy()
+localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
 
-pkge = pkgdata.get('PKGE', '0')
-pkgv = pkgdata['PKGV']
-pkgr = pkgdata['PKGR']
-try:
-extendprauto = pkgdata['EXTENDPRAUTO']
-except IndexError:
-extendprauto = d.getVar('EXTENDPRAUTO')
+pkge = localdata.getVar("PKGE") or '0'
+pkgv = localdata.getVar("PKGV")
+pkgr = localdata.getVar("PKGR")
 #
 # Find out what the last version was
 # Make sure the version did not decrease
@@ -298,32 +283,31 @@ python buildhistory_emit_pkghistory() {
 
 pkginfo = PackageInfo(pkg)
 # Apparently the version can be different on a per-package basis (see 
Python)
-pkginfo.pe = pkgdata.get('PE', '0')
-pkginfo.pv = pkgdata['PV']
-pkginfo.pr = pkgdata['PR']
-pkginfo.pkg = pkgdata['PKG']
+pkginfo.pe = localdata.getVar("PE") or '0'
+pkginfo.pv = localdata.getVar("PV")
+pkginfo.pr = localdata.getVar("PR")
+pkginfo.pkg = localdata.getVar("PKG")
 pkginfo.pkge = pkge
 pkginfo.pkgv = pkgv
 pkginfo.pkgr = pkgr
-pkginfo.extendprauto = extendprauto
-pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RPROVIDES', "")))
-pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RDEPENDS', "")))
-pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RRECOMMENDS', "")))
-pkginfo.rsuggests = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RSUGGESTS', "")))
-pkginfo.rreplaces = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RREPLACES', "")))
-pkginfo.rconflicts = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RCONFLICTS', "")))
-pkginfo.files = oe.utils.squashspaces(pkgdata.get('FILES', ""))
+pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RPROVIDES") or ""))
+pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar("RDEPENDS") or ""))
+pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(localdata.getVar

[OE-core] [master][PATCH 0/5] Allow PR Service and hash equiv together

2020-08-24 Thread Mark Hatle
Before PR service didn't work reliably with hash equivalency.  Generally
you ended up with results that may not be reproducible, even if you
started with the same PR service database and hash equivalency database.

Overtime, intermediate PR values would be created that would cause thing
to get out of sync in the case of certain rebuilds or other corner cases.

The set refactors the PR service to work along side the new hash equiv
system.  It moves the PR and AUTOINC lookup to AFTER the do_package task
is complete.  This allows us to use the do_package unihash for lookup.

Additionally this fixed a small issue with the kernel, where the PR value
could get incremented twice.  The fix is an artifact of the other changes
that cause us to only run the PR service work once per recipe.

This has been tested with the following workflow, which covers one of
the critical corner cases for me:

configure local.conf with:

   BB_HASHSERVE = "auto"
   BB_SIGNATURE_HANDLER = "OEEquivHash"
   
   PRSERV_HOST ??= "localhost:0"
   
   INHERIT += "reproducible_build"
   INHERIT += "buildhistory"

bitbake glibc linux-yocto

# Modify meta/recipes-core/glibc/glibc_2.32.bb, add a comment
# to the do_patch_append().  This will taint the hash of this
# function.

bitbake glibc linux-yocto

# The system should have detected the output was the same, and
# no proceed past do_package in glibc.  The kernel should not
# have built at all.

# Store/mv the tmp and sstate-cache from that build elsewhere
# repeat the run

bitbake glibc linux-yocto

# Compare the results of tmp/deploy//* between last
# and current run.
#
# The contents should be the same (filenames specifically).
#
# Also the kernel should be r0.0, not r0.1.

Note: if the hash equivalency database or PR server database (located
in the cache directory) is removed, the values may not be the same
as the previous run.

Additionally while testing the various package_*.bbclass files, it
was noted that package_tar.bbclass was not working the same way as
the others.  This was correct as a standalone patch.

Mark Hatle (5):
  package_tar.bbclass: Sync to the other package_* classes
  hash equivalency and pr service
  package.bbclass: Move package_get_auto_pr to packagedata.bbclass
  package / packagedata bbclass: Change the way PRAUTO and AUTOINC are
handled
  buildhistory.bbclass: Rework to use read_subpackage_metadata

 meta/classes/buildhistory.bbclass |  49 ++--
 meta/classes/kernel.bbclass   |   4 +-
 meta/classes/nopackages.bbclass   |   1 +
 meta/classes/package.bbclass  | 120 +++---
 meta/classes/package_tar.bbclass  |  11 +--
 meta/classes/packagedata.bbclass  |  69 +
 meta/conf/bitbake.conf|   1 +
 meta/lib/oe/packagedata.py|   8 +-
 meta/lib/oe/sstatesig.py  |   2 +
 9 files changed, 169 insertions(+), 96 deletions(-)

-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141793): 
https://lists.openembedded.org/g/openembedded-core/message/141793
Mute This Topic: https://lists.openembedded.org/mt/76396873/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH 4/5] package / packagedata bbclass: Change the way PRAUTO and AUTOINC are handled

2020-08-24 Thread Mark Hatle
During the do_package operation, we need to convert AUTOINC to some other
value, as PKGV is set and stored to the packagedata at this time.  We
create a new variable called "PRSERV_PV_AUTOINC" to handle this case,
with a default value of AUTOINC.

The value of PRSERV_PV_AUTOINC is added to the PKGDATA_VARS_NOHASH so it
does NOT get written as part of the PKGV to the packagedata files, but is
preserved as a bitbake variable for later processing.

As part of this work, we deterined we should not always write out
PKGDATA_VARS_NOHASH, as it may contain incomplete information.  For instance
EXTENDPRAUTO at do_package time is "", while later becomes a ".0" or
similar number when the PR Service is enabled.

do_packagedata was modified to explicitly run package_get_auto_pr, and then
preserve the PRAUTO and PRSERV_PV_AUTOINC values.  We store these in a new
file called ${PN}_prservice.oe_nohash.  Doing it this way will ensure that
subsequent calls to "read_subpackage_metadata", and similar function will
have correct results for the recipes PRAUTO and PRSERV_PV_AUTOINC.

read_subpackage_metadata was modified to read this file as well.

package_get_auto_pr was refactored to add processing to set the
correct PRSERV_PV_AUTOINC values, instead of modifying PKGV directly.

A related change was also needed for the kernel.bbclass.  The kernel

needs to get the AUTOINC values used by the various PV/PKGV, but with
this new system we don't want to call auto_pr each time -- instead
call the read_subpackage_metadata to ensure a consistent result.

This also fixes an issue in the old version where the kernel sometimes
caused the PR to be incremented twice, as the auto_pr was called
multiple times from different tasks.

Signed-off-by: Mark Hatle 
---
 meta/classes/kernel.bbclass  |  4 ++--
 meta/classes/package.bbclass | 27 +++
 meta/classes/packagedata.bbclass | 30 --
 meta/conf/bitbake.conf   |  1 +
 4 files changed, 42 insertions(+), 20 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index e2ceb6a333..4449452065 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -416,7 +416,7 @@ kernel_do_install() {
install -d ${D}${sysconfdir}/modules-load.d
install -d ${D}${sysconfdir}/modprobe.d
 }
-do_install[prefuncs] += "package_get_auto_pr"
+do_install[prefuncs] += "read_subpackage_metadata"
 
 # Must be ran no earlier than after do_kernel_checkout or else Makefile won't 
be in ${S}/Makefile
 do_kernel_version_sanity_check() {
@@ -749,7 +749,7 @@ kernel_do_deploy() {
done
fi
 }
-do_deploy[prefuncs] += "package_get_auto_pr"
+do_deploy[prefuncs] += "read_subpackage_metadata"
 
 addtask deploy after do_populate_sysroot do_packagedata
 
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 003eb71a6e..e822174b8a 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -7,7 +7,7 @@
 #
 # There are the following default steps but PACKAGEFUNCS can be extended:
 #
-# a) package_get_auto_pr - get PRAUTO from remote PR service
+# a) package_convert_autoinc - convert autoinc in PKGV to ${PRSERV_PV_AUTOINC}
 #
 # b) perform_packagecopy - Copy D into PKGD
 #
@@ -672,6 +672,13 @@ def runtime_mapping_rename (varname, pkg, d):
 # Package functions suitable for inclusion in PACKAGEFUNCS
 #
 
+python package_convert_autoinc() {
+pkgv = d.getVar("PKGV")
+
+if 'AUTOINC' in pkgv:
+d.setVar("PKGV", pkgv.replace("AUTOINC", "${PRSERV_PV_AUTOINC}"))
+}
+
 LOCALEBASEPN ??= "${PN}"
 
 python package_do_split_locales() {
@@ -1470,7 +1477,8 @@ PKGDESTWORK = "${WORKDIR}/pkgdata"
 
 PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY 
RDEPENDS RPROVIDES RRECOMMENDS RSUGGESTS RREPLACES RCONFLICTS SECTION PKG 
ALLOW_EMPTY FILES CONFFILES FILES_INFO PACKAGE_ADD_METADATA pkg_postinst 
pkg_postrm pkg_preinst pkg_prerm"
 
-PKGDATA_VARS_NOHASH = "EXTENDPRAUTO"
+# Really we only care about PRAUTO, but EXTENDPRAUTO evaluates this
+PKGDATA_VARS_NOHASH = "EXTENDPRAUTO PRSERV_PV_AUTOINC"
 
 python emit_pkgdata() {
 from glob import glob
@@ -1584,11 +1592,6 @@ fi
 
 subdata_file = pkgdatadir + "/runtime/%s" % pkg
 
-# Write out nohash variables first
-with open("%s.oe_nohash" % subdata_file, 'w') as sf:
-for var in (d.getVar('PKGDATA_VARS_NOHASH') or "").split():
-write_if_exists(d, sf, pkg, var)
-
 # Write out regular variables, but sanitize nohash values
 with open(subdata_file, 'w') as sf:
 localdata = d.createCopy()
@@ -2322,7 +2325,7 @@ python do_package () {
 package_qa_handle_error("var-undefined", msg, d)
 return

[OE-core] [master][PATCH 1/5] package_tar.bbclass: Sync to the other package_* classes

2020-08-24 Thread Mark Hatle
Sync up the task definitions with the other package classes.  This may not
have been strictly necessary but will make overall maintenance easier as
the various package classes are now in sync.

Additional, there was a missing deltask in the nopackages.bbclass related
to the package_tar which has been corrected.  This could cause problems on
native recipes when package_tar was enabled.

Signed-off-by: Mark Hatle 
---
 meta/classes/nopackages.bbclass  |  1 +
 meta/classes/package_tar.bbclass | 11 ++-
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/meta/classes/nopackages.bbclass b/meta/classes/nopackages.bbclass
index 559f5078bd..7a4f632d71 100644
--- a/meta/classes/nopackages.bbclass
+++ b/meta/classes/nopackages.bbclass
@@ -2,6 +2,7 @@ deltask do_package
 deltask do_package_write_rpm
 deltask do_package_write_ipk
 deltask do_package_write_deb
+deltask do_package_write_tar
 deltask do_package_qa
 deltask do_packagedata
 deltask do_package_setscene
diff --git a/meta/classes/package_tar.bbclass b/meta/classes/package_tar.bbclass
index ce3ab4c8e2..8946bc212a 100644
--- a/meta/classes/package_tar.bbclass
+++ b/meta/classes/package_tar.bbclass
@@ -57,10 +57,8 @@ python do_package_tar () {
 
 python () {
 if d.getVar('PACKAGES') != '':
-deps = (d.getVarFlag('do_package_write_tar', 'depends') or "").split()
-deps.append('tar-native:do_populate_sysroot')
-deps.append('virtual/fakeroot-native:do_populate_sysroot')
-d.setVarFlag('do_package_write_tar', 'depends', " ".join(deps))
+deps = ' tar-native:do_populate_sysroot 
virtual/fakeroot-native:do_populate_sysroot'
+d.appendVarFlag('do_package_write_tar', 'depends', deps)
 d.setVarFlag('do_package_write_tar', 'fakeroot', "1")
 }
 
@@ -70,4 +68,7 @@ python do_package_write_tar () {
 bb.build.exec_func("do_package_tar", d)
 }
 do_package_write_tar[dirs] = "${D}"
-addtask package_write_tar before do_build after do_packagedata do_package
+do_package_write_tar[depends] += 
"${@oe.utils.build_depends_string(d.getVar('PACKAGE_WRITE_DEPS'), 
'do_populate_sysroot')}"
+addtask package_write_tar after do_packagedata do_package
+
+do_build[recrdeptask] += "do_package_write_tar"
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141795): 
https://lists.openembedded.org/g/openembedded-core/message/141795
Mute This Topic: https://lists.openembedded.org/mt/76396875/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [master][PATCH 3/5] package.bbclass: Move package_get_auto_pr to packagedata.bbclass

2020-08-24 Thread Mark Hatle
Running package_get_auto_pr during do_package is 'too early' when we need
to evaluate the do_package output to get it's unihash to look up the
right value in the PR database.

Move this function packagedata.bbclass (no changes yet), as subsequent
patches will refactor this function and the order in which it's called.

Signed-off-by: Mark Hatle 
---
 meta/classes/package.bbclass | 50 ---
 meta/classes/packagedata.bbclass | 59 
 2 files changed, 59 insertions(+), 50 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 5fa96eb331..003eb71a6e 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -672,56 +672,6 @@ def runtime_mapping_rename (varname, pkg, d):
 # Package functions suitable for inclusion in PACKAGEFUNCS
 #
 
-python package_get_auto_pr() {
-import oe.prservice
-import re
-
-# Support per recipe PRSERV_HOST
-pn = d.getVar('PN')
-host = d.getVar("PRSERV_HOST_" + pn)
-if not (host is None):
-d.setVar("PRSERV_HOST", host)
-
-pkgv = d.getVar("PKGV")
-
-# PR Server not active, handle AUTOINC
-if not d.getVar('PRSERV_HOST'):
-if 'AUTOINC' in pkgv:
-d.setVar("PKGV", pkgv.replace("AUTOINC", "0"))
-return
-
-auto_pr = None
-pv = d.getVar("PV")
-version = d.getVar("PRAUTOINX")
-pkgarch = d.getVar("PACKAGE_ARCH")
-checksum = d.getVar("BB_TASKHASH")
-
-if d.getVar('PRSERV_LOCKDOWN'):
-auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or 
d.getVar('PRAUTO_' + version) or None
-if auto_pr is None:
-bb.fatal("Can NOT get PRAUTO from lockdown exported file")
-d.setVar('PRAUTO',str(auto_pr))
-return
-
-try:
-conn = d.getVar("__PRSERV_CONN")
-if conn is None:
-conn = oe.prservice.prserv_make_conn(d)
-if conn is not None:
-if "AUTOINC" in pkgv:
-srcpv = bb.fetch2.get_srcrev(d)
-base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
-value = conn.getPR(base_ver, pkgarch, srcpv)
-d.setVar("PKGV", pkgv.replace("AUTOINC", str(value)))
-
-auto_pr = conn.getPR(version, pkgarch, checksum)
-except Exception as e:
-bb.fatal("Can NOT get PRAUTO, exception %s" %  str(e))
-if auto_pr is None:
-bb.fatal("Can NOT get PRAUTO from remote PR service")
-d.setVar('PRAUTO',str(auto_pr))
-}
-
 LOCALEBASEPN ??= "${PN}"
 
 python package_do_split_locales() {
diff --git a/meta/classes/packagedata.bbclass b/meta/classes/packagedata.bbclass
index a903e5cfd2..981c324909 100644
--- a/meta/classes/packagedata.bbclass
+++ b/meta/classes/packagedata.bbclass
@@ -32,3 +32,62 @@ python read_subpackage_metadata () {
 else:
 d.setVar(key, sdata[key], parsing=True)
 }
+
+package_get_auto_pr[vardepsexclude] = "BB_TASKDEPDATA"
+python package_get_auto_pr() {
+import oe.prservice
+import re
+
+def get_do_package_hash(pn):
+if d.getVar("BB_RUNTASK") != "do_package":
+taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+for dep in taskdepdata:
+if taskdepdata[dep][1] == "do_package" and taskdepdata[dep][0] 
== pn:
+return taskdepdata[dep][6]
+return d.getVar("BB_UNIHASH")
+
+# Support per recipe PRSERV_HOST
+pn = d.getVar('PN')
+host = d.getVar("PRSERV_HOST_" + pn)
+if not (host is None):
+d.setVar("PRSERV_HOST", host)
+
+pkgv = d.getVar("PKGV")
+
+# PR Server not active, handle AUTOINC
+if not d.getVar('PRSERV_HOST'):
+if 'AUTOINC' in pkgv:
+d.setVar("PKGV", pkgv.replace("AUTOINC", "0"))
+return
+
+auto_pr = None
+pv = d.getVar("PV")
+version = d.getVar("PRAUTOINX")
+pkgarch = d.getVar("PACKAGE_ARCH")
+checksum = get_do_package_hash(pn)
+
+if d.getVar('PRSERV_LOCKDOWN'):
+auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or 
d.getVar('PRAUTO_' + version) or None
+if auto_pr is None:
+bb.fatal("Can NOT get PRAUTO from lockdown exported file")
+d.setVar('PRAUTO',str(auto_pr))
+return
+
+try:
+conn = d.getVar("__PRSERV_CONN")
+if conn is None:
+conn = oe.prservice.prserv_make_conn(d)
+if conn is not None:
+if "AUTOINC" in pkgv:
+srcpv = bb.fetch2.get_srcrev(d)
+base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
+v

[OE-core] [master][PATCH 2/5] hash equivalency and pr service

2020-08-24 Thread Mark Hatle
When the PR service is enabled a number of small changes happen to variables.

During the do_package task, EXTENDPRAUTO will get a value placed into it that
is intended to be used to extend the PR.  This value will get written to
various intermediate files and will result in the computed hash never being
equivalent.

To resolve this issue, we create a new file with the extension ".oe_nohash".
This extension contains various values that need to be passed from task to
task, but NOT calculated within the scope of the task hash.  The items
placed into the nohash are captured in package.bbclass in PKGDATA_VAR_NOHASH.

The PKGDATA_VAR_NOHASH is also used to prevent expansion of the value when
users are written to the various pkgdata files.  This expansion is prevented
by removeing the variables, resulting in the system printing the literal
${VAR} when used.

Additionally package dependencies rely on variables that also eventually
include EXTENDPRAUTO.  So instead of resolving the variable at getVar time,
we keep them as references by using the same technique as above.

Due to this change, buildhistory needs a few minor modifications to
know how to deal with EXTENDPRAUTO not always being available for comparison.

Signed-off-by: Mark Hatle 
---
 meta/classes/buildhistory.bbclass | 29 ++
 meta/classes/package.bbclass  | 63 ---
 meta/lib/oe/packagedata.py|  8 +++-
 meta/lib/oe/sstatesig.py  |  2 +
 4 files changed, 80 insertions(+), 22 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass 
b/meta/classes/buildhistory.bbclass
index 805e976ac5..2bccdfc953 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -99,6 +99,7 @@ python buildhistory_emit_pkghistory() {
 import json
 import shlex
 import errno
+import oe.packagedata
 
 pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE')
 oldpkghistdir = d.getVar('BUILDHISTORY_OLD_DIR_PACKAGE')
@@ -127,6 +128,7 @@ python buildhistory_emit_pkghistory() {
 self.pkge = ""
 self.pkgv = ""
 self.pkgr = ""
+self.extendprauto = ""
 self.size = 0
 self.depends = ""
 self.rprovides = ""
@@ -163,6 +165,8 @@ python buildhistory_emit_pkghistory() {
 pkginfo.pkgv = value
 elif name == "PKGR":
 pkginfo.pkgr = value
+elif name == "EXTENDPRAUTO":
+pkginfo.extendprauto = value
 elif name == "RPROVIDES":
 pkginfo.rprovides = value
 elif name == "RDEPENDS":
@@ -260,18 +264,24 @@ python buildhistory_emit_pkghistory() {
 
 pkgdest = d.getVar('PKGDEST')
 for pkg in packagelist:
-pkgdata = {}
-with open(os.path.join(pkgdata_dir, 'runtime', pkg)) as f:
-for line in f.readlines():
-item = line.rstrip('\n').split(': ', 1)
-key = item[0]
-if key.endswith('_' + pkg):
-key = key[:-len(pkg)-1]
-pkgdata[key] = 
item[1].encode('latin-1').decode('unicode_escape')
+pkgdata = oe.packagedata.read_pkgdatafile(os.path.join(pkgdata_dir, 
'runtime', pkg))
+
+npkgdata = {}
+for key in pkgdata.keys():
+if key.endswith('_' + pkg):
+nkey = key[:-len(pkg)-1]
+npkgdata[nkey] = pkgdata[key]
+
+for key in npkgdata.keys():
+  pkgdata[key] = npkgdata[key]
 
 pkge = pkgdata.get('PKGE', '0')
 pkgv = pkgdata['PKGV']
 pkgr = pkgdata['PKGR']
+try:
+extendprauto = pkgdata['EXTENDPRAUTO']
+except IndexError:
+extendprauto = d.getVar('EXTENDPRAUTO')
 #
 # Find out what the last version was
 # Make sure the version did not decrease
@@ -295,6 +305,7 @@ python buildhistory_emit_pkghistory() {
 pkginfo.pkge = pkge
 pkginfo.pkgv = pkgv
 pkginfo.pkgr = pkgr
+pkginfo.extendprauto = extendprauto
 pkginfo.rprovides = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RPROVIDES', "")))
 pkginfo.rdepends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RDEPENDS', "")))
 pkginfo.rrecommends = 
sortpkglist(oe.utils.squashspaces(pkgdata.get('RRECOMMENDS', "")))
@@ -398,6 +409,8 @@ def write_pkghistory(pkginfo, d):
 f.write(u"PKGV = %s\n" % pkginfo.pkgv)
 if pkginfo.pkgr != pkginfo.pr:
 f.write(u"PKGR = %s\n" % pkginfo.pkgr)
+if pkginfo.extendprauto:
+f.write(u"EXTENDPRAUTO = %s\n" % pkginfo.extendprauto)
 f.write(u"RPROVIDES = %s\n" %  pkginfo.rprovides)
 f.write(u"RDEPENDS = %s\n" %  pkginfo.rdepends)

[OE-core] hash equivalency and PR service - RFC

2020-08-19 Thread Mark Hatle
I've been working recently to try to get hash equivalency and PR service to work
together properly.  Earlier today I sent a patch to address one of the issues I
found (un-related to PR service).

In order to replicate the condition, I've been using poky master (oe should work
fine) with the following configuration:

BB_HASHSERVE = "auto"
BB_SIGNATURE_HANDLER = "OEEquivHash"

PRSERV_HOST ??= "localhost:0"

PKGR[vardepsexclude] = "EXTENDPRAUTO"

INHERIT += "reproducible_build"

INHERIT += "buildhistory"

I then run:

MACHINE=qemuarm64 bitbake glibc

Go in and edit meta/recipes-core/glibc/glibc_2.32.bb, search for the
do_patch_append and add "# modify hash".

Re-run the build command.  The result SHOULD be the system detects that nothing
changed (in do_package, inspecting log.do_package should show it changed the
unihash.)  And then the rest is accelerated.  However, the PKGR (from the PR and
AUTOPR) cause this to fail, as well as any embedded dependencies that also have
PR/AUTOPR embedded into them.  Both of these are written to pkgdata files, and
when compared to prior builds for hashs will always be different.

I tried to break this by moving the AUTOPR value into a separate file that was
not used to calculate hash information.  For the most part this works, but I ran
into some problems.  (See below).

I've generated a patch:

https://github.com/mhatle/poky/commit/86fbd639261319de4c951708d9805abba6036510

(I'm not entirely happy with the change, but I think it's a reasonable start for
a review to make sure the direction of the change is reasonable.)

The header of the patch explains some of the work that I've done:


When the PR service is enabled a number of small changes happen to variables.

During the do_package task, EXTENDPRAUTO will get a value placed into it that
is intended to be used to extend the PR.  This value will get written to
various intermediate files and will result in the computed hash never being
equivalent.

To resolve this issue, we create a new file with the extension ".oe_nohash".
This extension contains various values that need to be passed from task to
task, but NOT calculated within the scope of the task hash.  The items
placed into the nohash are captured in package.bbclass in PKGDATA_VAR_NOHASH.

The PKGDATA_VAR_NOHASH is also used to prevent expansion of the value when
users are written to the various pkgdata files.  This expansion is prevented
by replacing the variable references ${VAR} with ##{VAR}## in a local
data store.  The pkgdata loading routines then know to load the nohash and
translate the ##{VAR}## back to ${VAR} automatically.

Additionally package dependencies rely on variables that also eventually
include EXTENDPRAUTO.  So instead of resolving the variable at getVar time,
we keep them as references.

The goal is instead of taking:

RRECOMMEND_package-dev = 'package = ${EXTENDPKGV}'

and making it:

RRECOMMEND_package-dev = 'package = 1.0-r0.1'

We want to keep EXTENDPKGV which internally refers to EXTENDPRAUTO
which is already in the special NOHASH category.

NOTE:  The above is not working properly in the case of ${@...} python
functions.  This is because the bb.utils.explode_dep_version2 does not
understand how to expand or deal with ${@ ... } syntax.  (It expects
a comma or space separated list, so the spaces cause problems!)


If anyone has any ideas how to address the RRECOMMEND issue above, I could use
some help.  I'm going to keep working on it, but I'm not sure how to get the
system to expand some but not everything.  (I might need to use the same local
datastore trick in this expansion that I used elsewhere.)

--Mark
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141652): 
https://lists.openembedded.org/g/openembedded-core/message/141652
Mute This Topic: https://lists.openembedded.org/mt/76302328/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [PATCH] package.bbclass: Sort shlib2 output for hash equivalency

2020-08-19 Thread Mark Hatle
The output was unsorted, so different versions of python, different input
ordering could have have changed the files, and thus changed the hashes
making the system think the output was different, even when unmodified.

Signed-off-by: Mark Hatle 
---
 meta/classes/package.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index f8dc1bb468..7a36262eb6 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1936,7 +1936,7 @@ python package_do_shlibs() {
 shlibs_file = os.path.join(shlibswork_dir, pkg + ".list")
 if len(sonames):
 with open(shlibs_file, 'w') as fd:
-for s in sonames:
+for s in sorted(sonames):
 if s[0] in shlib_provider and s[1] in shlib_provider[s[0]]:
 (old_pkg, old_pkgver) = shlib_provider[s[0]][s[1]]
 if old_pkg != pkg:
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141643): 
https://lists.openembedded.org/g/openembedded-core/message/141643
Mute This Topic: https://lists.openembedded.org/mt/76290147/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [bitbake-devel] Backtrace in bitbake with OE archiver

2020-05-27 Thread Mark Hatle
One more note.. Adding the following and then it works fine:

INITRAMFS_IMAGE = "core-image-minimal"

I suspect it works simply because the task processing order has changed in some
way..

On 5/27/20 2:44 PM, Mark Hatle wrote:
> Cross posting cause I'm not sure where the bug is, I suspect bitbake -- but it
> could be in OE's archiver class.  I have been able to reproduce this in both
> Zeus and master.
> 
> When I start a new project, and add the following to the local.conf:
> 
> INHERIT += "archiver"
> ARCHIVER_MODE[src] = "original"
> COPYLEFT_LICENSE_INCLUDE = ""
> COPYLEFT_LICENSE_EXCLUDE = ""
> 
> 
> Then I run it with:
> 
> bitbake core-image-minimal --runall=deploy_archives
> 
> 
> and I get a backtrace in bitbake:
> 
> ERROR: Running idle function
> Traceback (most recent call last):
>   File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 1527, in
> RunQueue.execute_runqueue():
>  try:
> >return self._execute_runqueue()
>  except bb.runqueue.TaskFailure:
>   File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 1444, in
> RunQueue._execute_runqueue():
>  [43, 967, 4, 
> 3,
> 1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3, 
> 37, 244])
> >if self.rqdata.prepare() == 0:
>  self.state = runQueueComplete
>   File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 935, in
> RunQueueData.prepare():
>  for tid in list(runall_tids):
> >mark_active(tid,1)
>  if self.cooker.configuration.force:
>   File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 850, in
> mark_active(tid='/scratch1/fray/poky/meta/recipes-core/images/core-image-minimal.bb:do_deploy_archives',
> depth=1):
>  for depend in depends:
> >mark_active(depend, depth+1)
> 
>   File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 848, in
> mark_active(tid='/scratch1/fray/poky/meta/recipes-core/images/core-image-minimal.bb:do_ar_original',
> depth=2):
> 
> >depends = self.runtaskentries[tid].depends
>  for depend in depends:
> KeyError:
> '/scratch1/fray/poky/meta/recipes-core/images/core-image-minimal.bb:do_ar_original'
> 
> 
> As you see above the system in the mark_active function of runqueue.py is 
> trying
> to reference the key of the task core-image-minimal:do_ar_original, and it 
> can't
> find it and crashes out.
> 
> 
> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138814): 
https://lists.openembedded.org/g/openembedded-core/message/138814
Mute This Topic: https://lists.openembedded.org/mt/74507943/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] Backtrace in bitbake with OE archiver

2020-05-27 Thread Mark Hatle
Cross posting cause I'm not sure where the bug is, I suspect bitbake -- but it
could be in OE's archiver class.  I have been able to reproduce this in both
Zeus and master.

When I start a new project, and add the following to the local.conf:

INHERIT += "archiver"
ARCHIVER_MODE[src] = "original"
COPYLEFT_LICENSE_INCLUDE = ""
COPYLEFT_LICENSE_EXCLUDE = ""


Then I run it with:

bitbake core-image-minimal --runall=deploy_archives


and I get a backtrace in bitbake:

ERROR: Running idle function
Traceback (most recent call last):
  File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 1527, in
RunQueue.execute_runqueue():
 try:
>return self._execute_runqueue()
 except bb.runqueue.TaskFailure:
  File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 1444, in
RunQueue._execute_runqueue():
 [43, 967, 4, 3,
1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3, 37, 
244])
>if self.rqdata.prepare() == 0:
 self.state = runQueueComplete
  File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 935, in
RunQueueData.prepare():
 for tid in list(runall_tids):
>mark_active(tid,1)
 if self.cooker.configuration.force:
  File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 850, in
mark_active(tid='/scratch1/fray/poky/meta/recipes-core/images/core-image-minimal.bb:do_deploy_archives',
depth=1):
 for depend in depends:
>mark_active(depend, depth+1)

  File "/scratch1/fray/poky/bitbake/lib/bb/runqueue.py", line 848, in
mark_active(tid='/scratch1/fray/poky/meta/recipes-core/images/core-image-minimal.bb:do_ar_original',
depth=2):

>depends = self.runtaskentries[tid].depends
 for depend in depends:
KeyError:
'/scratch1/fray/poky/meta/recipes-core/images/core-image-minimal.bb:do_ar_original'


As you see above the system in the mark_active function of runqueue.py is trying
to reference the key of the task core-image-minimal:do_ar_original, and it can't
find it and crashes out.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138812): 
https://lists.openembedded.org/g/openembedded-core/message/138812
Mute This Topic: https://lists.openembedded.org/mt/74507870/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [PATCH v2] sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors

2020-05-27 Thread Mark Hatle
Ping

I noticed this hasn't been integrated or commented on yet.

On 5/13/20 11:12 AM, Mark Hatle wrote:
> From: Mark Hatle 
> 
> Prior to fetching, the system checks if the sstate file is present
> either locally or on the mirror.  If it is, then it goes to the fetch
> stage.  Up to three files can be fetched, sstate, sstate.siginfo and
> sstate.sig (if signature validation is enabled).
> 
> The previous pstaging_fetch function would iterate over these, and if
> a download error occurred would spew forth a great amount of fetcher
> failure messages as well as stop fetching the next item in the set.
> 
> This was resolved by adding a fetcher.checkstatus() call prior to
> the download.  If the file isn't present, then the exception will
> be triggered, and no fetcher failure messages will reach the user.
> 
> The exception handler is then modified to be a pass so that it will
> loop and pull the rest of the files that that are requested.
> 
> Additionally, a check for the existance of the .sig file was added
> to the sstate_installpkg to avoid an error trying to load the .sig
> if it wasn't downloaded.
> 
> Signed-off-by: Mark Hatle 
> Signed-off-by: Mark Hatle 
> ---
> 
> v2 - fix a typo in the error message about signature not being
>  present.
> 
>  meta/classes/sstate.bbclass | 6 +-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index aa9c30b4e1..375196ef21 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -355,6 +355,9 @@ def sstate_installpkg(ss, d):
>  d.setVar('SSTATE_INSTDIR', sstateinst)
>  
>  if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
> +if not os.path.isfile(sstatepkg + '.sig'):
> +bb.warn("No signature file for sstate package %s, skipping 
> acceleration..." % sstatepkg)
> +return False
>  signer = get_signer(d, 'local')
>  if not signer.verify(sstatepkg + '.sig'):
>  bb.warn("Cannot verify signature on sstate package %s, skipping 
> acceleration..." % sstatepkg)
> @@ -733,10 +736,11 @@ def pstaging_fetch(sstatefetch, d):
>  localdata.setVar('SRC_URI', srcuri)
>  try:
>  fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
> +fetcher.checkstatus()
>  fetcher.download()
>  
>  except bb.fetch2.BBFetchException:
> -break
> +pass
>  
>  def sstate_setscene(d):
>  shared_state = sstate_state_fromvars(d)
> 
> 
> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138809): 
https://lists.openembedded.org/g/openembedded-core/message/138809
Mute This Topic: https://lists.openembedded.org/mt/74185410/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [PATCH v2] sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors

2020-05-13 Thread Mark Hatle
From: Mark Hatle 

Prior to fetching, the system checks if the sstate file is present
either locally or on the mirror.  If it is, then it goes to the fetch
stage.  Up to three files can be fetched, sstate, sstate.siginfo and
sstate.sig (if signature validation is enabled).

The previous pstaging_fetch function would iterate over these, and if
a download error occurred would spew forth a great amount of fetcher
failure messages as well as stop fetching the next item in the set.

This was resolved by adding a fetcher.checkstatus() call prior to
the download.  If the file isn't present, then the exception will
be triggered, and no fetcher failure messages will reach the user.

The exception handler is then modified to be a pass so that it will
loop and pull the rest of the files that that are requested.

Additionally, a check for the existance of the .sig file was added
to the sstate_installpkg to avoid an error trying to load the .sig
if it wasn't downloaded.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---

v2 - fix a typo in the error message about signature not being
 present.

 meta/classes/sstate.bbclass | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index aa9c30b4e1..375196ef21 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -355,6 +355,9 @@ def sstate_installpkg(ss, d):
 d.setVar('SSTATE_INSTDIR', sstateinst)
 
 if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
+if not os.path.isfile(sstatepkg + '.sig'):
+bb.warn("No signature file for sstate package %s, skipping 
acceleration..." % sstatepkg)
+return False
 signer = get_signer(d, 'local')
 if not signer.verify(sstatepkg + '.sig'):
 bb.warn("Cannot verify signature on sstate package %s, skipping 
acceleration..." % sstatepkg)
@@ -733,10 +736,11 @@ def pstaging_fetch(sstatefetch, d):
 localdata.setVar('SRC_URI', srcuri)
 try:
 fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+fetcher.checkstatus()
 fetcher.download()
 
 except bb.fetch2.BBFetchException:
-break
+pass
 
 def sstate_setscene(d):
 shared_state = sstate_state_fromvars(d)
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138229): 
https://lists.openembedded.org/g/openembedded-core/message/138229
Mute This Topic: https://lists.openembedded.org/mt/74185410/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [zeus][ 1/2] sstatesig: Optimise get_taskhash for hashequiv

2020-05-13 Thread Mark Hatle
From: Richard Purdie 

With hashequiv the get_taskhash function is called much more regularly
and contains expensive operations. This these don't change based upon
hash in a given build, improve the caching within the function to
reduce overhead.

(From OE-Core rev: de98cfe3cde4b8d5f4b163b5fba3f129651ef06a)

Signed-off-by: Richard Purdie 
Signed-off-by: Mark Hatle 
---
 meta/lib/oe/sstatesig.py | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index b2316b12b8..f1abff0c45 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -151,6 +151,13 @@ class SignatureGeneratorOEBasicHashMixIn(object):
 
 def get_taskhash(self, tid, deps, dataCache):
 h = super(bb.siggen.SignatureGeneratorBasicHash, 
self).get_taskhash(tid, deps, dataCache)
+if tid in self.lockedhashes:
+if self.lockedhashes[tid]:
+return self.lockedhashes[tid]
+else:
+return h
+
+h = super(bb.siggen.SignatureGeneratorBasicHash, 
self).get_taskhash(tid, deps, dataCache)
 
 (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
 
@@ -187,17 +194,19 @@ class SignatureGeneratorOEBasicHashMixIn(object):
   % (recipename, task, h, h_locked, 
var))
 
 return h_locked
+
+self.lockedhashes[tid] = False
 #bb.warn("%s %s %s" % (recipename, task, h))
 return h
 
 def get_unihash(self, tid):
-if tid in self.lockedhashes:
+if tid in self.lockedhashes and self.lockedhashes[tid]:
 return self.lockedhashes[tid]
 return super().get_unihash(tid)
 
 def dump_sigtask(self, fn, task, stampbase, runtime):
 tid = fn + ":" + task
-if tid in self.lockedhashes:
+if tid in self.lockedhashes and self.lockedhashes[tid]:
 return
 super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, 
task, stampbase, runtime)
 
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138227): 
https://lists.openembedded.org/g/openembedded-core/message/138227
Mute This Topic: https://lists.openembedded.org/mt/74184220/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [zeus][ 0/2] Fix for two issues related to sstate-cache

2020-05-13 Thread Mark Hatle
Patch 1 is a backport from master (it's already in Dunfell).  If you have a
locked-sigs file, and the sstate for that item isn't available the .siginfo and
sigdata files were not being written.  Patch 1/2 resolved that with the
fragment in "dump_sigtask".

Patch 2 was sent for master as well, and this is just to get it applied to
Zeus.  Due to problem #1, we were generating sstate-cache that did not have
siginfo files.  Due to this we were getting failures later on using that
sstate-cache in a subsequent build.  This patch ensures if this happens
again (of if someone intentionally strips those) it won't fail.

Mark Hatle (1):
  sstate.bbclass: When siginfo or sig files are missing, stop fetcher
errors

Richard Purdie (1):
  sstatesig: Optimise get_taskhash for hashequiv

 meta/classes/sstate.bbclass |  6 +-
 meta/lib/oe/sstatesig.py| 13 +++--
 2 files changed, 16 insertions(+), 3 deletions(-)

-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138226): 
https://lists.openembedded.org/g/openembedded-core/message/138226
Mute This Topic: https://lists.openembedded.org/mt/74184219/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [zeus][ 2/2] sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors

2020-05-13 Thread Mark Hatle
From: Mark Hatle 

Prior to fetching, the system checks if the sstate file is present
either locally or on the mirror.  If it is, then it goes to the fetch
stage.  Up to three files can be fetched, sstate, sstate.siginfo and
sstate.sig (if signature validation is enabled).

The previous pstaging_fetch function would iterate over these, and if
a download error occurred would spew forth a great amount of fetcher
failure messages as well as stop fetching the next item in the set.

This was resolved by adding a fetcher.checkstatus() call prior to
the download.  If the file isn't present, then the exception will
be triggered, and no fetcher failure messages will reach the user.

The exception handler is then modified to be a pass so that it will
loop and pull the rest of the files that that are requested.

Additionally, a check for the existance of the .sig file was added
to the sstate_installpkg to avoid an error trying to load the .sig
if it wasn't downloaded.

Signed-off-by: Mark Hatle 
---
 meta/classes/sstate.bbclass | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index c0329cd5d1..3528908727 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -333,6 +333,9 @@ def sstate_installpkg(ss, d):
 d.setVar('SSTATE_INSTDIR', sstateinst)
 
 if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
+if not os.path.isfile(sstatepkg + '.sig'):
+bb.warn("Not signature file for sstate package %s, skipping 
acceleration..." % sstatepkg)
+return False
 signer = get_signer(d, 'local')
 if not signer.verify(sstatepkg + '.sig'):
 bb.warn("Cannot verify signature on sstate package %s, skipping 
acceleration..." % sstatepkg)
@@ -703,10 +706,11 @@ def pstaging_fetch(sstatefetch, d):
 localdata.setVar('SRC_URI', srcuri)
 try:
 fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+fetcher.checkstatus()
 fetcher.download()
 
 except bb.fetch2.BBFetchException:
-break
+pass
 
 def sstate_setscene(d):
 shared_state = sstate_state_fromvars(d)
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138228): 
https://lists.openembedded.org/g/openembedded-core/message/138228
Mute This Topic: https://lists.openembedded.org/mt/74184221/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [PATCH] sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors

2020-05-13 Thread Mark Hatle
From: Mark Hatle 

Prior to fetching, the system checks if the sstate file is present
either locally or on the mirror.  If it is, then it goes to the fetch
stage.  Up to three files can be fetched, sstate, sstate.siginfo and
sstate.sig (if signature validation is enabled).

The previous pstaging_fetch function would iterate over these, and if
a download error occurred would spew forth a great amount of fetcher
failure messages as well as stop fetching the next item in the set.

This was resolved by adding a fetcher.checkstatus() call prior to
the download.  If the file isn't present, then the exception will
be triggered, and no fetcher failure messages will reach the user.

The exception handler is then modified to be a pass so that it will
loop and pull the rest of the files that that are requested.

Additionally, a check for the existance of the .sig file was added
to the sstate_installpkg to avoid an error trying to load the .sig
if it wasn't downloaded.

Signed-off-by: Mark Hatle 
Signed-off-by: Mark Hatle 
---
 meta/classes/sstate.bbclass | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index aa9c30b4e1..da8d5e78f4 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -355,6 +355,9 @@ def sstate_installpkg(ss, d):
 d.setVar('SSTATE_INSTDIR', sstateinst)
 
 if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
+if not os.path.isfile(sstatepkg + '.sig'):
+bb.warn("Not signature file for sstate package %s, skipping 
acceleration..." % sstatepkg)
+return False
 signer = get_signer(d, 'local')
 if not signer.verify(sstatepkg + '.sig'):
 bb.warn("Cannot verify signature on sstate package %s, skipping 
acceleration..." % sstatepkg)
@@ -733,10 +736,11 @@ def pstaging_fetch(sstatefetch, d):
 localdata.setVar('SRC_URI', srcuri)
 try:
 fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+fetcher.checkstatus()
 fetcher.download()
 
 except bb.fetch2.BBFetchException:
-break
+pass
 
 def sstate_setscene(d):
 shared_state = sstate_state_fromvars(d)
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138225): 
https://lists.openembedded.org/g/openembedded-core/message/138225
Mute This Topic: https://lists.openembedded.org/mt/74184073/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [oe][yocto][bitbake] Fetching source using different protocols

2020-05-11 Thread Mark Hatle


On 5/11/20 4:17 AM, Quentin Schulz wrote:
> Hi Mohamed,
> 
> On Mon, May 11, 2020 at 11:03:26AM +0200, Dawod wrote:
>> Hello,
>>
>> I need to fetch a git repo using 2 different protocols ( ssh & https )
>> So that when I run bitbake, It will fetch using ssh protocol first and if
>> it fails to fetch, It will try to fetch using https protocol.
>>
> 
> Why? What's the exact use case?
> 
>> can I do some thing like that or I will have to change it manually every
>> time ?
>>
> 
> Maybe you could play with PREMIRROS? (I've never explicitly used that
> variable though):
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#var-PREMIRRORS

This is what I would suggest.  Have the recipe itself set the fallback URI as
the main URL.


SRC_URI = "git://example.com/my/repository/uri;proto=https"

PREMIRRORS_prepend = "git://example.com/my/repository/uri;proto=https
git://example.com/my/otherrepo/uri;proto=ssh \n"

(the \n is literally '\' and 'n')

The above should, when it sees the SRC_URI, try the ssh protocol first.. if that
doesn't work it will fall back to SRC_URI.

--Mark

> I guess the asterisk parts could be removed and the path to more or less your
> source could be used (ssh first, to http path in your SRC_URI)?
> 
> That's a shot in the dark for me but something to test I'd say :)
> 
> Quentin
> 
> 
> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138136): 
https://lists.openembedded.org/g/openembedded-core/message/138136
Mute This Topic: https://lists.openembedded.org/mt/74131887/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [PATCH 2/5] archiver.bbclass: Make do_deploy_archives a recursive dependency

2020-04-01 Thread Mark Hatle


On 4/1/20 9:49 AM, Mark Hatle wrote:
> (I know this is an old thread, but I ran into this as well..)
> 
> ...
> 
> On 3/11/20 6:40 AM, Paul Barker wrote:
>> On Tue, 10 Mar 2020 23:18:33 +
>> Richard Purdie  wrote:
>>
>>> On Mon, 2020-03-09 at 14:21 +, Paul Barker wrote:
>>>> To ensure that archives are captured for all dependencies of a typical
>>>> bitbake build we add do_deploy_archives to the list of recursive
>>>> dependencies of do_build. Without this, archives may be missed for
>>>> recipes such as gcc-source which do not create packages or populate a
>>>> sysroot.
>>>>
>>>> do_deploy_archives is also added to the recursive dependencies of
>>>> do_populate_sdk so that all sources required for an SDK can be captured.
>>>>
>>>> Signed-off-by: Paul Barker 
>>>> ---
>>>>  meta/classes/archiver.bbclass | 4 +++-
>>>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
>>>> index fef7ad4f62..c11d36d708 100644
>>>> --- a/meta/classes/archiver.bbclass
>>>> +++ b/meta/classes/archiver.bbclass
>>>> @@ -604,7 +604,9 @@ addtask do_ar_configured after do_unpack_and_patch
>>>>  addtask do_ar_mirror after do_fetch
>>>>  addtask do_dumpdata
>>>>  addtask do_ar_recipe
>>>> -addtask do_deploy_archives before do_build
>>>> +addtask do_deploy_archives
>>>> +do_build[recrdeptask] += "do_deploy_archives"
>>>> +do_populate_sdk[recrdeptask] += "do_deploy_archives"  
>>>
>>> We implemented the --runall option to bitbake to try and avoid having
>>> recrdeptask versions of most tasks. Does that not work here? It should
>>> also work for the SDK I think?
> 
> You can't use a --runall operation on a task target, such as:
> 
> bitbake core-image-minimal -c populate_sdk
> 
> I can't find any way to get all of the archiver sources for the populate_sdk
> target.  The above should resolve this, and as Paul said below it will ensure
> that populate_sdk follows the same behavior as regular image generation as 
> well.
> 
> I'm wondering if do_populate_sdk_ext needs it as well.

I verified, adding do_populate_sdk_ext is not needed.. the do_populate_sdk
appears to capture everything in the eSDK, at least in my configuration.

--Mark

>> If the archiver is enabled, its tasks should be in the dependency tree of
>> whatever you're building so that you don't need to invoke bitbake twice to
>> produce the required artifacts. For images that's the way the archiver has
>> always worked, if it's enabled then you just need to do `bitbake image` to
>> build the image and deploy the source archives. This change just extends that
>> behaviour to cover other things we can build and ensures that we don't miss
>> sources for recipes like gcc-source.
>>
>>
>> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136934): 
https://lists.openembedded.org/g/openembedded-core/message/136934
Mute This Topic: https://lists.openembedded.org/mt/72395532/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [PATCH 2/5] archiver.bbclass: Make do_deploy_archives a recursive dependency

2020-04-01 Thread Mark Hatle
(I know this is an old thread, but I ran into this as well..)

...

On 3/11/20 6:40 AM, Paul Barker wrote:
> On Tue, 10 Mar 2020 23:18:33 +
> Richard Purdie  wrote:
> 
>> On Mon, 2020-03-09 at 14:21 +, Paul Barker wrote:
>>> To ensure that archives are captured for all dependencies of a typical
>>> bitbake build we add do_deploy_archives to the list of recursive
>>> dependencies of do_build. Without this, archives may be missed for
>>> recipes such as gcc-source which do not create packages or populate a
>>> sysroot.
>>>
>>> do_deploy_archives is also added to the recursive dependencies of
>>> do_populate_sdk so that all sources required for an SDK can be captured.
>>>
>>> Signed-off-by: Paul Barker 
>>> ---
>>>  meta/classes/archiver.bbclass | 4 +++-
>>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
>>> index fef7ad4f62..c11d36d708 100644
>>> --- a/meta/classes/archiver.bbclass
>>> +++ b/meta/classes/archiver.bbclass
>>> @@ -604,7 +604,9 @@ addtask do_ar_configured after do_unpack_and_patch
>>>  addtask do_ar_mirror after do_fetch
>>>  addtask do_dumpdata
>>>  addtask do_ar_recipe
>>> -addtask do_deploy_archives before do_build
>>> +addtask do_deploy_archives
>>> +do_build[recrdeptask] += "do_deploy_archives"
>>> +do_populate_sdk[recrdeptask] += "do_deploy_archives"  
>>
>> We implemented the --runall option to bitbake to try and avoid having
>> recrdeptask versions of most tasks. Does that not work here? It should
>> also work for the SDK I think?

You can't use a --runall operation on a task target, such as:

bitbake core-image-minimal -c populate_sdk

I can't find any way to get all of the archiver sources for the populate_sdk
target.  The above should resolve this, and as Paul said below it will ensure
that populate_sdk follows the same behavior as regular image generation as well.

I'm wondering if do_populate_sdk_ext needs it as well.

> If the archiver is enabled, its tasks should be in the dependency tree of
> whatever you're building so that you don't need to invoke bitbake twice to
> produce the required artifacts. For images that's the way the archiver has
> always worked, if it's enabled then you just need to do `bitbake image` to
> build the image and deploy the source archives. This change just extends that
> behaviour to cover other things we can build and ensures that we don't miss
> sources for recipes like gcc-source.
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136926): 
https://lists.openembedded.org/g/openembedded-core/message/136926
Mute This Topic: https://lists.openembedded.org/mt/72395532/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [v2 PATCH] base.bbclass: Add COMPATIBLE_OS, useful for non-Linux target recipes

2020-03-26 Thread Mark Hatle
I talked with RP off the list and got this working.

Below is what I ended up doing to prevent the default packages from working with
baremetal, unless they were configured to.

In my baremetal distro.conf:

COMPATOS = ""
COMPATOS_class-target = ".*-linux${LIBCEXTENSION}${ABIEXTENSION}"
COMPATIBLE_HOST ?= "${COMPATOS}"

Then in each of the baremetal -only- recipes:

COMPATIBLE_HOST = ".*-elf"
COMPATIBLE_HOST_arm = "[^-]*-[^-]*-eabi"

In the recipes for BOTH baremetal and Linux:

COMPATIBLE_HOST = "${HOST_SYS}"

(Setting to "" should work as well..)

--Mark

On 3/26/20 12:05 PM, Mark Hatle wrote:
>> On Thu, 2020-03-26 at 10:25 -0500, Mark Hatle wrote:
>>> Add the ability to generate non-Linux based components, such as
>>> baremetal, FreeRTOS, Zypher, etc, exist.  There is a need for a way
>>> to take components as specific to a given OS.
>>>
>>> This is especially important in a multiconfig build where you may be
>>> targeting different OSes which different sets of recipes in a common
>>> set of layers.
>>>
>>> Setting a default of (matching bitbake.conf's default TARGET_OS):
>>>
>>>COMPATIBLE_OS ?= "linux${LIBCEXTENSION}${ABIEXTENSION}"
>>>
>>> will allow all existing recipes to be tagged as Linux specific, so
>>> only non-Linux recipes would need to be tagged with specific
>>> compatibility.
>>>
>>> For a baremetal recipes, the following was used to test this code:
>>>
>>>COMPATIBLE_OS = "elf"
>>>COMPATIBLE_OS_arm = "eabi"
>>>
>>> Signed-off-by: Mark Hatle 
>>> ---
>>> V2:
>>> Only difference to V1 is typograhic fixes to the commit message.
>>>
>>>  meta/classes/base.bbclass| 7 +++
>>>  meta/conf/documentation.conf | 1 +
>>>  2 files changed, 8 insertions(+)
>>>
>>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>>> index 45f9435fd8..c123c5dc50 100644
>>> --- a/meta/classes/base.bbclass
>>> +++ b/meta/classes/base.bbclass
>>> @@ -509,6 +509,13 @@ python () {
>>>  d.setVarFlag('do_devshell', 'fakeroot', '1')
>>>  d.appendVarFlag('do_devshell', 'depends', '
>>> virtual/fakeroot-native:do_populate_sysroot')
>>>
>>> +need_os = d.getVar('COMPATIBLE_OS')
>>> +if need_os and not d.getVar('PARSE_ALL_RECIPES', False):
>>> +import re
>>> +this_os = d.getVar('TARGET_OS')
>>> +if not re.match(need_os, this_os):
>>> +raise bb.parse.SkipRecipe("incompatible with os %s (not
>>> in COMPATIBLE_OS)" % this_os)
>>> +
>>>  need_machine = d.getVar('COMPATIBLE_MACHINE')
>>>  if need_machine and not d.getVar('PARSE_ALL_RECIPES', False):
>>>  import re
>>> diff --git a/meta/conf/documentation.conf
>>> b/meta/conf/documentation.conf
>>> index 0b21d1f63e..7a87540f43 100644
>>> --- a/meta/conf/documentation.conf
>>> +++ b/meta/conf/documentation.conf
>>> @@ -111,6 +111,7 @@ COMBINED_FEATURES[doc] = "A set of features
>>> common between MACHINE_FEATURES and
>>>  COMMON_LICENSE_DIR[doc] = "Points to meta/files/common-licenses in
>>> the Source Directory, which is where generic license files reside."
>>>  COMPATIBLE_HOST[doc] = "A regular expression that resolves to one or
>>> more hosts (when the recipe is native) or one or more targets (when
>>> the recipe is non-native) with which a recipe is compatible."
>>
>> How does COMPATIBLE_OS compare to COMPATIBLE_HOST (documented above)?
> 
> HOST_SYS="aarch64-oe-linux"
> 
> So the equivalent to the COMPATIBLE_OS would be:
> 
> COMPATIBLE_HOST ?= ".*-linux.*"
> 
> So it COULD be used in most cases.
> 
> When I tried this before though, I could get it to work in a multiconfig
> setting.
> 
> Specifically what I tried was setting, in my local.conf:
> 
> COMPATIBLE_HOST ?= ".*-linux${LIBCEXTENSION}${ABIEXTENSION}"
> 
> And then in the recipes:
> 
> COMPATIBLE_HOST = ".*-elf"
> COMPATIBLE_HOST_arm = ".*-eabi"
> 
> But then the second one matches the linux ABIEXTENSION and broke..
> 
> So really the issue is that I -only- ever want to match on the OS part of
> the "SYS" string.  And done to the multiple natures of the '-', it's
> difficult to get right.
> 
> I didn't try something complex like: [^-]-[^-]-eabi
> 
> --Mark
> 
>> Cheers
>>
>> Richard
>>
> 
> 
> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136777): 
https://lists.openembedded.org/g/openembedded-core/message/136777
Mute This Topic: https://lists.openembedded.org/mt/72566050/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [v2 PATCH] base.bbclass: Add COMPATIBLE_OS, useful for non-Linux target recipes

2020-03-26 Thread Mark Hatle
> On Thu, 2020-03-26 at 10:25 -0500, Mark Hatle wrote:
>> Add the ability to generate non-Linux based components, such as
>> baremetal, FreeRTOS, Zypher, etc, exist.  There is a need for a way
>> to take components as specific to a given OS.
>>
>> This is especially important in a multiconfig build where you may be
>> targeting different OSes which different sets of recipes in a common
>> set of layers.
>>
>> Setting a default of (matching bitbake.conf's default TARGET_OS):
>>
>>COMPATIBLE_OS ?= "linux${LIBCEXTENSION}${ABIEXTENSION}"
>>
>> will allow all existing recipes to be tagged as Linux specific, so
>> only non-Linux recipes would need to be tagged with specific
>> compatibility.
>>
>> For a baremetal recipes, the following was used to test this code:
>>
>>COMPATIBLE_OS = "elf"
>>COMPATIBLE_OS_arm = "eabi"
>>
>> Signed-off-by: Mark Hatle 
>> ---
>> V2:
>> Only difference to V1 is typograhic fixes to the commit message.
>>
>>  meta/classes/base.bbclass| 7 +++
>>  meta/conf/documentation.conf | 1 +
>>  2 files changed, 8 insertions(+)
>>
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index 45f9435fd8..c123c5dc50 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -509,6 +509,13 @@ python () {
>>  d.setVarFlag('do_devshell', 'fakeroot', '1')
>>  d.appendVarFlag('do_devshell', 'depends', '
>> virtual/fakeroot-native:do_populate_sysroot')
>>
>> +need_os = d.getVar('COMPATIBLE_OS')
>> +if need_os and not d.getVar('PARSE_ALL_RECIPES', False):
>> +import re
>> +this_os = d.getVar('TARGET_OS')
>> +if not re.match(need_os, this_os):
>> +raise bb.parse.SkipRecipe("incompatible with os %s (not
>> in COMPATIBLE_OS)" % this_os)
>> +
>>  need_machine = d.getVar('COMPATIBLE_MACHINE')
>>  if need_machine and not d.getVar('PARSE_ALL_RECIPES', False):
>>  import re
>> diff --git a/meta/conf/documentation.conf
>> b/meta/conf/documentation.conf
>> index 0b21d1f63e..7a87540f43 100644
>> --- a/meta/conf/documentation.conf
>> +++ b/meta/conf/documentation.conf
>> @@ -111,6 +111,7 @@ COMBINED_FEATURES[doc] = "A set of features
>> common between MACHINE_FEATURES and
>>  COMMON_LICENSE_DIR[doc] = "Points to meta/files/common-licenses in
>> the Source Directory, which is where generic license files reside."
>>  COMPATIBLE_HOST[doc] = "A regular expression that resolves to one or
>> more hosts (when the recipe is native) or one or more targets (when
>> the recipe is non-native) with which a recipe is compatible."
>
> How does COMPATIBLE_OS compare to COMPATIBLE_HOST (documented above)?

HOST_SYS="aarch64-oe-linux"

So the equivalent to the COMPATIBLE_OS would be:

COMPATIBLE_HOST ?= ".*-linux.*"

So it COULD be used in most cases.

When I tried this before though, I could get it to work in a multiconfig
setting.

Specifically what I tried was setting, in my local.conf:

COMPATIBLE_HOST ?= ".*-linux${LIBCEXTENSION}${ABIEXTENSION}"

And then in the recipes:

COMPATIBLE_HOST = ".*-elf"
COMPATIBLE_HOST_arm = ".*-eabi"

But then the second one matches the linux ABIEXTENSION and broke..

So really the issue is that I -only- ever want to match on the OS part of
the "SYS" string.  And done to the multiple natures of the '-', it's
difficult to get right.

I didn't try something complex like: [^-]-[^-]-eabi

--Mark

> Cheers
>
> Richard
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136761): 
https://lists.openembedded.org/g/openembedded-core/message/136761
Mute This Topic: https://lists.openembedded.org/mt/72566050/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [v2 PATCH] base.bbclass: Add COMPATIBLE_OS, useful for non-Linux target recipes

2020-03-26 Thread Mark Hatle
Add the ability to generate non-Linux based components, such as
baremetal, FreeRTOS, Zypher, etc, exist.  There is a need for a way
to take components as specific to a given OS.

This is especially important in a multiconfig build where you may be
targeting different OSes which different sets of recipes in a common
set of layers.

Setting a default of (matching bitbake.conf's default TARGET_OS):

   COMPATIBLE_OS ?= "linux${LIBCEXTENSION}${ABIEXTENSION}"

will allow all existing recipes to be tagged as Linux specific, so
only non-Linux recipes would need to be tagged with specific
compatibility.

For a baremetal recipes, the following was used to test this code:

   COMPATIBLE_OS = "elf"
   COMPATIBLE_OS_arm = "eabi"

Signed-off-by: Mark Hatle 
---
V2:
Only difference to V1 is typograhic fixes to the commit message.

 meta/classes/base.bbclass| 7 +++
 meta/conf/documentation.conf | 1 +
 2 files changed, 8 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 45f9435fd8..c123c5dc50 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -509,6 +509,13 @@ python () {
 d.setVarFlag('do_devshell', 'fakeroot', '1')
 d.appendVarFlag('do_devshell', 'depends', ' 
virtual/fakeroot-native:do_populate_sysroot')
 
+need_os = d.getVar('COMPATIBLE_OS')
+if need_os and not d.getVar('PARSE_ALL_RECIPES', False):
+import re
+this_os = d.getVar('TARGET_OS')
+if not re.match(need_os, this_os):
+raise bb.parse.SkipRecipe("incompatible with os %s (not in 
COMPATIBLE_OS)" % this_os)
+
 need_machine = d.getVar('COMPATIBLE_MACHINE')
 if need_machine and not d.getVar('PARSE_ALL_RECIPES', False):
 import re
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 0b21d1f63e..7a87540f43 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -111,6 +111,7 @@ COMBINED_FEATURES[doc] = "A set of features common between 
MACHINE_FEATURES and
 COMMON_LICENSE_DIR[doc] = "Points to meta/files/common-licenses in the Source 
Directory, which is where generic license files reside."
 COMPATIBLE_HOST[doc] = "A regular expression that resolves to one or more 
hosts (when the recipe is native) or one or more targets (when the recipe is 
non-native) with which a recipe is compatible."
 COMPATIBLE_MACHINE[doc] = "A regular expression that resolves to one or more 
target machines with which a recipe is compatible."
+COMPATIBLE_OS[doc] = "A regular expression that resolves to one or more target 
oses which a recipe is compatible."
 COMPLEMENTARY_GLOB[doc] = "Defines wildcards to match when installing a list 
of complementary packages for all the packages installed in an image."
 CONFFILES[doc] = "Identifies editable or configurable files that are part of a 
package."
 CONFIG_SITE[doc] = "A list of files that contains autoconf test results 
relevant to the current build. This variable is used by the Autotools utilities 
when running configure."
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136755): 
https://lists.openembedded.org/g/openembedded-core/message/136755
Mute This Topic: https://lists.openembedded.org/mt/72566050/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


[OE-core] [PATCH] base.bbclass: Add COMPATIBLE_OS, useful to for non-Linux target recipes

2020-03-26 Thread Mark Hatle
As the ability to generate non-Linux based components, such as
baremetal, FreeRTOS, Zypher, etc, exist.  There is a need for a way
to take components as specific to a given OS.

This is especially important in a multiconfig build where you may be
targeting different OSes which different sets of recipes in a common
set of layers.

Setting a default of (matching bitbake.conf's default TARGET_OS):

   COMPATIBLE_OS ?= "linux${LIBCEXTENSION}${ABIEXTENSION}"

Will allow all existing recipes to be tagged as Linux specific, so
only non-Linux recipes would need to be tagged with specific
compatibility.

For a baremetal recipes, the following was used to test this code:

   COMPATIBLE_OS = "elf"
   COMPATIBLE_OS_arm = "eabi"

Signed-off-by: Mark Hatle 
---
 meta/classes/base.bbclass| 7 +++
 meta/conf/documentation.conf | 1 +
 2 files changed, 8 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 45f9435fd8..c123c5dc50 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -509,6 +509,13 @@ python () {
 d.setVarFlag('do_devshell', 'fakeroot', '1')
 d.appendVarFlag('do_devshell', 'depends', ' 
virtual/fakeroot-native:do_populate_sysroot')
 
+need_os = d.getVar('COMPATIBLE_OS')
+if need_os and not d.getVar('PARSE_ALL_RECIPES', False):
+import re
+this_os = d.getVar('TARGET_OS')
+if not re.match(need_os, this_os):
+raise bb.parse.SkipRecipe("incompatible with os %s (not in 
COMPATIBLE_OS)" % this_os)
+
 need_machine = d.getVar('COMPATIBLE_MACHINE')
 if need_machine and not d.getVar('PARSE_ALL_RECIPES', False):
 import re
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 0b21d1f63e..7a87540f43 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -111,6 +111,7 @@ COMBINED_FEATURES[doc] = "A set of features common between 
MACHINE_FEATURES and
 COMMON_LICENSE_DIR[doc] = "Points to meta/files/common-licenses in the Source 
Directory, which is where generic license files reside."
 COMPATIBLE_HOST[doc] = "A regular expression that resolves to one or more 
hosts (when the recipe is native) or one or more targets (when the recipe is 
non-native) with which a recipe is compatible."
 COMPATIBLE_MACHINE[doc] = "A regular expression that resolves to one or more 
target machines with which a recipe is compatible."
+COMPATIBLE_OS[doc] = "A regular expression that resolves to one or more target 
oses which a recipe is compatible."
 COMPLEMENTARY_GLOB[doc] = "Defines wildcards to match when installing a list 
of complementary packages for all the packages installed in an image."
 CONFFILES[doc] = "Identifies editable or configurable files that are part of a 
package."
 CONFIG_SITE[doc] = "A list of files that contains autoconf test results 
relevant to the current build. This variable is used by the Autotools utilities 
when running configure."
-- 
2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136754): 
https://lists.openembedded.org/g/openembedded-core/message/136754
Mute This Topic: https://lists.openembedded.org/mt/72565485/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [OE-core] [PATCH] mesa-gl: The purpose of mesa-gl is to provide for X11 usage

2020-03-26 Thread Mark Hatle
> On Wed, Mar 25, 2020 at 02:27:37PM -0500, Mark Hatle wrote:
>> > To be honest, I would just take the entire recipe out. It's causing
>> > trouble
>> > during updates, isn't being tested neither for builds nor at runtime,
>> and
>> > is supposed to provide some specific configuration which as this
>> > discussion
>> > makes clear, nobody seems to quite understand.
>>
>> With the abomination that is libmali (and similar), it is still needed.
>> It's the only way to support GL on a primarily GLES compatible system.
>>
>> The problem is the way they do this seems to be a custom version of
>> libdrm, which then conflicts with the mesa version.  Thus the issues.
>>
>> I'm happy to continue testing my particular needs now and the future
>> (thus
>> the patch against master.)
>>...
>
> Stupid question:
>
> Is
>   PREFERRED_PROVIDER_virtual/mesa = "mesa-gl"
>   PREFERRED_PROVIDER_virtual/libgl = "mesa-gl"
> equivalent to
>   PACKAGECONFIG_pn-mesa = "opengl dri x11"
> ?

I don't know.  I didn't write this.

However, doing some investigation, the big difference here is the overall
capabilities.  There are common distributions where a user may want to use
mesa (no external libdrm) as well as distress where they want to use an
external drm and the mesa-gl version only.

So 'mesa' and 'external drm + mesa-gl' are equivalent in functionality for
a given distro.  Only difference being optimization.

You may ask why would I use one vs the other.  In my case, we have a
family of SoC parts.  Some of the family have a Mali graphics chip on
them, while others don't have any optimized graphics so everything has to
be software rendered.  The only difference (from a software point of view)
is if the Mali is on-board or not.  So using a common (binary)
distribution, and being able to just swap those parts is highly desirable.
 (Does that ACTUALLY work right now, I'm not sure.. but that is what I am
working on.)

>> --Mark
>
> cu
> Adrian
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#136752): 
https://lists.openembedded.org/g/openembedded-core/message/136752
Mute This Topic: https://lists.openembedded.org/mt/72547327/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-


  1   2   3   4   5   6   7   8   9   10   >