Re: [OE-core] [meta-oe][PATCH] package.bbclass: provide debug sources in externalsrc -dbg packages

[Vignesh Rajendran (RBEI/ECF3)]

Hello Khem,

Long-wait for any feedback !!!

Mit freundlichen Grüßen / Best regards

Rajendran Vignesh
RBEI/ECF33

Tel. +91 422 67-65103

From: Khem Raj [mailto:raj.k...@gmail.com]
Sent: Friday, July 06, 2018 10:21 AM
To: Vignesh Rajendran (RBEI/ECF3) 
Cc: Patches and discussions about the oe-core layer 
; Paul Eggleton 

Subject: Re: [OE-core] [meta-oe][PATCH] package.bbclass: provide debug sources 
in externalsrc -dbg packages

Yes, you can try that


On Thu, Jul 5, 2018 at 9:29 PM Vignesh Rajendran (RBEI/ECF3) 
mailto:vignesh.rajend...@in.bosch.com>> wrote:
Hello Khem,

Can you clarify your idea and my question?

Mit freundlichen Grüßen / Best regards

Rajendran Vignesh
RBEI/ECF33

Tel. +91 422 67-65103


-Original Message-
From: Vignesh Rajendran (RBEI/ECF3)
Sent: Tuesday, June 19, 2018 10:24 AM
To: 'Khem Raj' mailto:raj.k...@gmail.com>>; Paul Eggleton 
mailto:paul.eggle...@linux.intel.com>>
Cc: Patches and discussions about the oe-core layer 
mailto:openembedded-core@lists.openembedded.org>>
Subject: RE: [OE-core] [meta-oe][PATCH] package.bbclass: provide debug sources 
in externalsrc -dbg packages

Hello Khem,

>> perhaps setting DEBUG_PREFIX_MAP relative to S instead of WORKDIR is
>> going to take care of this

Do you mean to change it at bitbake.conf ?

Mit freundlichen Grüßen / Best regards

Rajendran Vignesh
RBEI/ECF33

Tel. +91 422 67-65103


-Original Message-
From: Khem Raj [mailto:raj.k...@gmail.com]
Sent: Monday, June 18, 2018 11:13 PM
To: Paul Eggleton 
mailto:paul.eggle...@linux.intel.com>>
Cc: Vignesh Rajendran (RBEI/ECF3) 
mailto:vignesh.rajend...@in.bosch.com>>; 
Patches and discussions about the oe-core layer 
mailto:openembedded-core@lists.openembedded.org>>
Subject: Re: [OE-core] [meta-oe][PATCH] package.bbclass: provide debug sources 
in externalsrc -dbg packages

On Sun, May 20, 2018 at 3:26 PM Paul Eggleton
mailto:paul.eggle...@linux.intel.com>> wrote:
>
> Hi Vignesh,
>
> Thanks for the patch, a question below.
>
> On Thursday, 17 May 2018 5:50:00 PM NZST Vignesh Rajendran wrote:
> > The recipes which are using externalsrc method doesn't contain the
> > debug sources in their -dbg packages.
> >
> > Add a handling in copydebugsources function to copy the debugsources for
> > externalsrc -dbg packages.
> >
> > Signed-off-by: Vignesh Rajendran 
> > mailto:vignesh.rajend...@in.bosch.com>>
> > ---
> >  meta/classes/externalsrc.bbclass |  3 +++
> >  meta/classes/package.bbclass | 12 ++--
> >  2 files changed, 13 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/externalsrc.bbclass 
> > b/meta/classes/externalsrc.bbclass
> > index c9f5cf7..135670d 100644
> > --- a/meta/classes/externalsrc.bbclass
> > +++ b/meta/classes/externalsrc.bbclass
> > @@ -129,6 +129,9 @@ python () {
> >  d.setVar('CONFIGURESTAMPFILE', configstamp)
> >  d.setVar('STAMP', 
> > '${STAMPS_DIR}/work-shared/${PN}/${EXTENDPE}${PV}-${PR}')
> >  d.setVar('STAMPCLEAN', '${STAMPS_DIR}/work-shared/${PN}/*-*')
> > +
> > + # set DEBUG_PREFIX_MAP to external source path
> > + d.appendVar("DEBUG_PREFIX_MAP", 
> > "-fdebug-prefix-map=${EXTERNALSRC}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}")
> >  }
> >
> >  python externalsrc_configure_prefunc() {
> > diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> > index edeffa9..f071a2e 100644
> > --- a/meta/classes/package.bbclass
> > +++ b/meta/classes/package.bbclass
> > @@ -420,6 +420,7 @@ def copydebugsources(debugsrcdir, d):
> >  workdir = d.getVar("WORKDIR")
> >  workparentdir = os.path.dirname(os.path.dirname(workdir))
> >  workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + 
> > os.path.basename(workdir)
> > +externalsrc = d.getVar('EXTERNALSRC', True)
> >
> >  # If build path exists in sourcefile, it means toolchain did not 
> > use
> >  # -fdebug-prefix-map to compile
> > @@ -444,9 +445,16 @@ def copydebugsources(debugsrcdir, d):
> >  processdebugsrc += "fgrep -zw '%s' | "
> >  # Remove prefix in the source paths
> >  processdebugsrc += "sed 's#%s##g' | "
> > -processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner 
> > '%s%s' 2>/dev/null)"
> > +# handle external sources to provide sources in -dbg packages
> > +if externalsrc:
> > +localsrc_prefix = "/usr/src/debug/"
> > +processdebugsrc += "sed 's#%s/##g' | "
> > +processdebugsrc += "(cd '%s' ; cpio -pd0mlL 
> > --no-preserve-owner '%s%s/%s' 2>/dev/null)"
> > +cmd = processdebugsrc % (sourcefile, workbasedir, 
> > localsrc_prefix, workbasedir, externalsrc, dvar, debugsrcdir, workbasedir)
> > +else:
> > +processdebugsrc += "(cd '%s' ; cpio -pd0mlL 
> > --no-preserve-owner '%s%s' 2>/dev/null)"
> > +cmd = processdebugsrc % (sourcefile, workbasedir, 
> > localsrc_prefix, workparentdir, dvar, 

Re: [OE-core] [rocko][PATCH] glibc: Avoid multilibbing on wordsize.h

[akuster808]

On 08/09/2018 10:10 AM, Daniel Díaz wrote:
> Hello!
>
>
> On 26 July 2018 at 09:38, Daniel Díaz  wrote:
>> Once another header #includes , there is a
>> potential recursion going on because the
>> multilib_header_wrapper.h #includes  again!
>>
>> This should not happen because an __arm__ (32-bits) or an
>> __aarch64__ (64-bits) environment guarantees that we will
>> be getting the correct definition, but when building against
>> a different target (like BPF), recursion is what happens.
>>
>> This can be seen, for instance, when building eBPF programs
>> from the kernel with `clang -target bpf', such as the ones
>> located in linux/tools/testing/selftests/bpf/.
>>
>> (From OE-Core rev: a74c77d6168101e88c3a3bce7130f4f52cfab95d)
>>
>> Signed-off-by: Daniel Díaz 
>> Signed-off-by: Aníbal Limón 
>> Signed-off-by: Richard Purdie 
>> ---
>>  meta/recipes-core/glibc/glibc-package.inc | 3 +--
>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/meta/recipes-core/glibc/glibc-package.inc 
>> b/meta/recipes-core/glibc/glibc-package.inc
>> index b6d8074..c360dd9 100644
>> --- a/meta/recipes-core/glibc/glibc-package.inc
>> +++ b/meta/recipes-core/glibc/glibc-package.inc
>> @@ -136,8 +136,7 @@ do_install_append_armeb () {
>>  }
>>
>>  do_install_armmultilib () {
>> -
>> -   oe_multilib_header bits/endian.h bits/fcntl.h bits/fenv.h 
>> bits/fp-fast.h bits/hwcap.h bits/ipc.h bits/link.h bits/wordsize.h
>> +   oe_multilib_header bits/endian.h bits/fcntl.h bits/fenv.h 
>> bits/fp-fast.h bits/hwcap.h bits/ipc.h bits/link.h
>> oe_multilib_header bits/local_lim.h bits/mman.h bits/msq.h 
>> bits/pthreadtypes.h bits/pthreadtypes-arch.h  bits/sem.h  bits/semaphore.h 
>> bits/setjmp.h
>> oe_multilib_header bits/shm.h bits/sigstack.h bits/stat.h 
>> bits/statfs.h bits/typesizes.h
>>
>> --
>> 2.7.4
>
> Please disregard this patch. An alternative will be posted shortly.
ok, thanks for the heads up. its in my mut branch. I will dropped shortly.

- armin
>
> Greetings!
>
> Daniel Díaz
> daniel.d...@linaro.org

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

[Anuj Mittal]

On 08/03/2018 04:55 AM, Alejandro Enedino Hernandez Samaniego wrote:
> We can give it a try but do you have a specific example of how it fails
> for you?
> Because I know theres lots of nested sccs on the yocto kernel cache, but
> in that case
> it wouldn't be a problem since this bug is specifically introduced by
> devtool when it
> copies local files from SRC_URI to a oe-local-files directory, it misses
> the corresponding cfg files (or patch files)
> since they're not listed in SRC_URI and it fails to build, in the case
> of the yocto kernel cache, it
> does not contain local files, so they wont be moved hence why it
> wouldn't be an issue.
> 
> From the kernel dev manual:
> https://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html#recipe-space-metadata
> :
> It is only necessary to specify the |.scc| files on the |SRC_URI|
> .

You are ignoring the line right next to it: "BitBake parses them and
fetches any files referenced in the .scc files by the include, patch, or
kconf commands." All files specified using 'include' will be parsed too
even if they are not in SRC_URI and this includes other scc files.

This change is good because it fixes something that is definitely
broken. My suggestion was meant to improve it further. I had provided
the test case in my reply to original patch.

And, I just tested it again using this change applied and I do see
errors when using a nested scc. Here it is again:

│   ├── linux
│   │   ├── linux-intel
│   │   │   ├── abc.scc
│   │   │   ├── def.scc

abc.scc only includes def.scc. And, def.scc includes a patch. SRC_URI
includes only abc.scc.

> 
> It specifies that only the scc files need to be on SRC_URI, which is why
> I'm saying the script
> will eventually run into them from the list.
> 
> I want to be clear that I'm not against doing this recursively, I just
> want to understand your testcase better.

I think if you'd not like to consider this use case as part of this
change, that should also be fine. But, do consider adding a new bug in
that case.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] libjpeg-turbo: fix timezone of reproducible build timestamp

[Christopher Clark]

Avoids producing different build results in different timezones.
Uses UTC with SOURCE_DATE_EPOCH.

Signed-off-by: Christopher Clark 
---
 meta/recipes-graphics/jpeg/libjpeg-turbo_1.5.3.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-graphics/jpeg/libjpeg-turbo_1.5.3.bb 
b/meta/recipes-graphics/jpeg/libjpeg-turbo_1.5.3.bb
index 15353b5..6be6aa5 100644
--- a/meta/recipes-graphics/jpeg/libjpeg-turbo_1.5.3.bb
+++ b/meta/recipes-graphics/jpeg/libjpeg-turbo_1.5.3.bb
@@ -46,7 +46,7 @@ DEBUG_OPTIMIZATION_append_armv5 = " 
${@bb.utils.contains('TUNE_CCARGS', '-mthumb
 def get_build_time(d):
 if d.getVar('SOURCE_DATE_EPOCH') != None:
 import datetime
-return " --with-build-date="+ 
datetime.datetime.fromtimestamp(float(d.getVar('SOURCE_DATE_EPOCH'))).strftime("%Y%m%d")
+return " --with-build-date="+ 
datetime.datetime.utcfromtimestamp(float(d.getVar('SOURCE_DATE_EPOCH'))).strftime("%Y%m%d")
 return ""
 
 EXTRA_OECONF_append_class-target = "${@get_build_time(d)}"
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3] devtool-source.bbclass: Support kernel-fragments/patch not in SRC_URI

[Jaewon Lee]

When using a recipe space kernel-meta, scc files are added through
SRC_URI, but they may include corresponding kernel fragments or patches
that are not necessarily in SRC_URI.

For bitbake, this is not a problem because the kernel-yocto class adds
the path where the .scc file was found to includes which consequentially
makes the .cfg, .patch file available to the kernel build.

However, when using devtool, only files specified in SRC_URI are copied
to oe-local-files in devtool's workspace. So if the cfg/patch file is not in
SRC_URI, it won't be copied, causing a kernel build failure when trying
to find it.

This fix parses local .scc files in SRC_URI, copies the corresponding
.cfg/.patch file to devtool's workdir, and also adds it to local_files
so it is available when doing a devtool build for the kernel.

[YOCTO #12858]

v2: also supporting patch not in SRC_URI
v3: fix spacing issues

Signed-off-by: Jaewon Lee 
Signed-off-by: Alejandro Enedino Hernandez Samaniego 
---
 meta/classes/devtool-source.bbclass | 12 
 1 file changed, 12 insertions(+)

diff --git a/meta/classes/devtool-source.bbclass 
b/meta/classes/devtool-source.bbclass
index 56882a4..67cd0ba 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -90,11 +90,23 @@ python devtool_post_unpack() {
 fname in files])
 return ret
 
+is_kernel_yocto = bb.data.inherits_class('kernel-yocto', d)
 # Move local source files into separate subdir
 recipe_patches = [os.path.basename(patch) for patch in
 oe.recipeutils.get_recipe_patches(d)]
 local_files = oe.recipeutils.get_recipe_local_files(d)
 
+if is_kernel_yocto:
+for key in local_files.copy():
+if key.endswith('scc'):
+sccfile = open(local_files[key], 'r')
+for l in sccfile:
+line = l.split()
+if line and line[0] in ('kconf', 'patch'):
+local_files[line[-1]] = 
os.path.join(os.path.dirname(local_files[key]), line[-1])
+
shutil.copy2(os.path.join(os.path.dirname(local_files[key]), line[-1]), workdir)
+sccfile.close()
+
 # Ignore local files with subdir={BP}
 srcabspath = os.path.abspath(srcsubdir)
 local_files = [fname for fname in local_files if
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 16/16] sstate: Implement hash equivalence sstate

[Joshua Watt]

Converts sstate so that it can use a hash equivalence server to
determine if a task really needs to be rebuilt, or if it can be restored
from a different (equivalent) sstate object.

The dependency IDs are cached persistently using persist_data. This has
a number of advantages:
 1) Dependency IDs can be cached between invocations of bitbake to
prevent needing to contact the server every time (which is slow)
 2) The value of each tasks dependency ID can easily be synchronized
between different threads, which will be useful if bitbake is
updated to do on the fly task re-hashing.

Signed-off-by: Joshua Watt 
---
 meta/classes/sstate.bbclass | 100 --
 meta/conf/bitbake.conf  |   4 +-
 meta/lib/oe/sstatesig.py| 166 
 3 files changed, 261 insertions(+), 9 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 28a64315b60..e956bd40b25 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -11,7 +11,7 @@ def generate_sstatefn(spec, hash, d):
 SSTATE_PKGARCH= "${PACKAGE_ARCH}"
 SSTATE_PKGSPEC= 
"sstate:${PN}:${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}:${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:"
 SSTATE_SWSPEC = "sstate:${PN}::${PV}:${PR}::${SSTATE_VERSION}:"
-SSTATE_PKGNAME= 
"${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC'), 
d.getVar('BB_TASKHASH'), d)}"
+SSTATE_PKGNAME= 
"${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC'), 
d.getVar('BB_DEPID'), d)}"
 SSTATE_PKG= "${SSTATE_DIR}/${SSTATE_PKGNAME}"
 SSTATE_EXTRAPATH   = ""
 SSTATE_EXTRAPATHWILDCARD = ""
@@ -75,6 +75,23 @@ SSTATE_SIG_PASSPHRASE ?= ""
 # Whether to verify the GnUPG signatures when extracting sstate archives
 SSTATE_VERIFY_SIG ?= "0"
 
+SSTATE_HASHEQUIV_METHOD ?= "OEOuthashBasic"
+SSTATE_HASHEQUIV_METHOD[doc] = "The function used to calculate the output hash 
\
+for a task, which in turn is used to determine equivalency. \
+"
+
+SSTATE_HASHEQUIV_SERVER ?= ""
+SSTATE_HASHEQUIV_SERVER[doc] = "The hash equivalence sever. For example, \
+'http://192.168.0.1:5000'. Do not include a trailing slash \
+"
+
+SSTATE_HASHEQUIV_REPORT_TASKDATA ?= "0"
+SSTATE_HASHEQUIV_REPORT_TASKDATA[doc] = "Report additional useful data to the \
+hash equivalency server, such as PN, PV, taskname, etc. This information \
+is very useful for developers looking at task data, but may leak sensitive 
\
+data if the equivalence server is public. \
+"
+
 python () {
 if bb.data.inherits_class('native', d):
 d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH', False))
@@ -626,7 +643,7 @@ def sstate_package(ss, d):
 return
 
 for f in (d.getVar('SSTATECREATEFUNCS') or '').split() + \
- ['sstate_create_package', 'sstate_sign_package'] + \
+ ['sstate_report_depid', 'sstate_create_package', 
'sstate_sign_package'] + \
  (d.getVar('SSTATEPOSTCREATEFUNCS') or '').split():
 # All hooks should run in SSTATE_BUILDDIR.
 bb.build.exec_func(f, d, (sstatebuild,))
@@ -750,6 +767,73 @@ python sstate_sign_package () {
d.getVar('SSTATE_SIG_PASSPHRASE'), armor=False)
 }
 
+def OEOuthashBasic(path, sigfile, task, d):
+import hashlib
+import stat
+
+def update_hash(s):
+s = s.encode('utf-8')
+h.update(s)
+if sigfile:
+sigfile.write(s)
+
+h = hashlib.sha1()
+prev_dir = os.getcwd()
+
+try:
+os.chdir(path)
+
+update_hash("OEOuthashBasic\n")
+
+# It is only currently useful to get equivalent hashes for things that
+# can be restored from sstate. Since the sstate object is named using
+# SSTATE_PKGSPEC and the task name, those should be included in the
+# output hash calculation.
+update_hash("SSTATE_PKGSPEC=%s\n" % d.getVar('SSTATE_PKGSPEC'))
+update_hash("task=%s\n" % task)
+
+for root, dirs, files in os.walk('.', topdown=True):
+# Sort directories and files to ensure consistent ordering
+dirs.sort()
+files.sort()
+
+for f in files:
+path = os.path.join(root, f)
+s = os.lstat(path)
+
+# Hash file path
+update_hash(path + '\n')
+
+# Hash file mode
+update_hash("\tmode=0x%x\n" % stat.S_IMODE(s.st_mode))
+update_hash("\ttype=0x%x\n" % stat.S_IFMT(s.st_mode))
+
+if stat.S_ISBLK(s.st_mode) or stat.S_ISBLK(s.st_mode):
+# Hash device major and minor
+update_hash("\tdev=%d,%d\n" % (os.major(s.st_rdev), 
os.minor(s.st_rdev)))
+elif stat.S_ISLNK(s.st_mode):
+# Hash symbolic link
+update_hash("\tsymlink=%s\n" % os.readlink(path))
+else:
+fh = 

[OE-core] [RFC v2 15/16] bitbake: hashserv: Add hash equivalence reference server

[Joshua Watt]

Implements a reference implementation of the hash equivalence server.
This server has minimal dependencies (and no dependencies outside of the
standard Python library), and implements the minimum required to be a
conforming hash equivalence server.

Signed-off-by: Joshua Watt 
---
 bitbake/bin/bitbake-hashserv |  67 ++
 bitbake/bin/bitbake-selftest |   2 +
 bitbake/lib/hashserv/__init__.py | 152 +++
 bitbake/lib/hashserv/tests.py| 141 
 4 files changed, 362 insertions(+)
 create mode 100755 bitbake/bin/bitbake-hashserv
 create mode 100644 bitbake/lib/hashserv/__init__.py
 create mode 100644 bitbake/lib/hashserv/tests.py

diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv
new file mode 100755
index 000..c49397b73a5
--- /dev/null
+++ b/bitbake/bin/bitbake-hashserv
@@ -0,0 +1,67 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2018 Garmin Ltd.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+import os
+import sys
+import logging
+import argparse
+import sqlite3
+
+sys.path.insert(0, 
os.path.join(os.path.dirname(os.path.dirname(__file__)),'lib'))
+
+import hashserv
+
+VERSION = "1.0.0"
+
+DEFAULT_HOST = ''
+DEFAULT_PORT = 8686
+
+def main():
+parser = argparse.ArgumentParser(description='HTTP Equivalence Reference 
Server. Version=%s' % VERSION)
+parser.add_argument('--address', default=DEFAULT_HOST, help='Bind address 
(default "%(default)s")')
+parser.add_argument('--port', type=int, default=DEFAULT_PORT, help='Bind 
port (default %(default)d)')
+parser.add_argument('--prefix', default='', help='HTTP path prefix 
(default "%(default)s")')
+parser.add_argument('--database', default='./hashserv.db', help='Database 
file (default "%(default)s")')
+parser.add_argument('--log', default='WARNING', help='Set logging level')
+
+args = parser.parse_args()
+
+logger = logging.getLogger('hashserv')
+
+level = getattr(logging, args.log.upper(), None)
+if not isinstance(level, int):
+raise ValueError('Invalid log level: %s' % args.log)
+
+logger.setLevel(level)
+console = logging.StreamHandler()
+console.setLevel(level)
+logger.addHandler(console)
+
+db = sqlite3.connect(args.database)
+
+server = hashserv.create_server((args.address, args.port), db, args.prefix)
+server.serve_forever()
+return 0
+
+if __name__ == '__main__':
+try:
+ret = main()
+except Exception:
+ret = 1
+import traceback
+traceback.print_exc()
+sys.exit(ret)
+
diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest
index 06a1c9a78dd..de1f8f74dda 100755
--- a/bitbake/bin/bitbake-selftest
+++ b/bitbake/bin/bitbake-selftest
@@ -22,6 +22,7 @@ sys.path.insert(0, 
os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib
 import unittest
 try:
 import bb
+import hashserv
 import layerindexlib
 except RuntimeError as exc:
 sys.exit(str(exc))
@@ -34,6 +35,7 @@ tests = ["bb.tests.codeparser",
  "bb.tests.parse",
  "bb.tests.persist_data",
  "bb.tests.utils",
+ "hashserv.tests",
  "layerindexlib.tests.layerindexobj",
  "layerindexlib.tests.restapi",
  "layerindexlib.tests.cooker"]
diff --git a/bitbake/lib/hashserv/__init__.py b/bitbake/lib/hashserv/__init__.py
new file mode 100644
index 000..cde030cb88e
--- /dev/null
+++ b/bitbake/lib/hashserv/__init__.py
@@ -0,0 +1,152 @@
+# Copyright (C) 2018 Garmin Ltd.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from http.server import BaseHTTPRequestHandler, HTTPServer
+import contextlib
+import urllib.parse
+import sqlite3
+import json
+import traceback
+import logging
+from datetime import 

[OE-core] [RFC v2 13/16] bitbake: runqueue: Pass dependency ID to hash validate

[Joshua Watt]

If the dependency ID is being used to track task dependencies, the hash
validation function needs to know about it in order to properly validate
the hash.

TODO: This currently isn't going to be backward compatible with older
hashvalidate functions. Is that necessary, and if so are there any
suggestions for a good approach?

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/runqueue.py | 14 +-
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 909fa3aec22..707baed2125 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1548,6 +1548,7 @@ class RunQueue:
 valid = []
 sq_hash = []
 sq_hashfn = []
+sq_depid = []
 sq_fn = []
 sq_taskname = []
 sq_task = []
@@ -1566,15 +1567,16 @@ class RunQueue:
 sq_fn.append(fn)
 sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
 sq_hash.append(self.rqdata.runtaskentries[tid].hash)
+sq_depid.append(self.rqdata.runtaskentries[tid].depid)
 sq_taskname.append(taskname)
 sq_task.append(tid)
-locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : 
sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
+locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : 
sq_hash, "sq_hashfn" : sq_hashfn, "sq_depid" : sq_depid, "d" : self.cooker.data 
}
 try:
-call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, 
d, siginfo=True)"
+call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, 
sq_depid, d, siginfo=True)"
 valid = bb.utils.better_eval(call, locs)
 # Handle version with no siginfo parameter
 except TypeError:
-call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, 
d)"
+call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, 
sq_depid, d)"
 valid = bb.utils.better_eval(call, locs)
 for v in valid:
 valid_new.add(sq_task[v])
@@ -2286,6 +2288,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
 if self.rq.hashvalidate:
 sq_hash = []
 sq_hashfn = []
+sq_depid = []
 sq_fn = []
 sq_taskname = []
 sq_task = []
@@ -2317,13 +2320,14 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
 sq_fn.append(fn)
 sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
 sq_hash.append(self.rqdata.runtaskentries[tid].hash)
+sq_depid.append(self.rqdata.runtaskentries[tid].depid)
 sq_taskname.append(taskname)
 sq_task.append(tid)
 
 self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT", 
len(stamppresent))
 
-call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash, 
sq_hashfn, d)"
-locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : 
sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
+call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash, 
sq_hashfn, sq_depid, d)"
+locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : 
sq_hash, "sq_hashfn" : sq_hashfn, "sq_depid": sq_depid, "d" : self.cooker.data }
 valid = bb.utils.better_eval(call, locs)
 
 self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 14/16] classes/sstate: Handle depid in hash check

[Joshua Watt]

Handles the argument that passes task dependency IDs in the hash check
function, as it is now required by bitbake

Signed-off-by: Joshua Watt 
---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 6743becf071..28a64315b60 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -766,7 +766,7 @@ sstate_unpack_package () {
 
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
-def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
+def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, sq_depid, d, 
siginfo=False):
 
 ret = []
 missed = []
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 12/16] bitbake: runqueue: Pass dependency ID to task

[Joshua Watt]

The dependency ID is now passed to the task in the BB_DEPID variable

Signed-off-by: Joshua Watt 
---
 bitbake/bin/bitbake-worker |  7 ---
 bitbake/lib/bb/runqueue.py | 10 ++
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index 41ef6d848ac..9650c954359 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -136,7 +136,7 @@ def sigterm_handler(signum, frame):
 os.killpg(0, signal.SIGTERM)
 sys.exit()
 
-def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, 
taskhash, appends, taskdepdata, extraconfigdata, quieterrors=False, 
dry_run_exec=False):
+def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, 
taskhash, depid, appends, taskdepdata, extraconfigdata, quieterrors=False, 
dry_run_exec=False):
 # We need to setup the environment BEFORE the fork, since
 # a fork() or exec*() activates PSEUDO...
 
@@ -235,6 +235,7 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, 
task, taskname, taskha
 
 the_data = bb_cache.loadDataFull(fn, appends)
 the_data.setVar('BB_TASKHASH', taskhash)
+the_data.setVar('BB_DEPID', depid)
 
 bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN"), 
taskname.replace("do_", "")))
 
@@ -425,10 +426,10 @@ class BitbakeWorker(object):
 sys.exit(0)
 
 def handle_runtask(self, data):
-fn, task, taskname, taskhash, quieterrors, appends, taskdepdata, 
dry_run_exec = pickle.loads(data)
+fn, task, taskname, taskhash, depid, quieterrors, appends, 
taskdepdata, dry_run_exec = pickle.loads(data)
 workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
 
-pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, 
self.databuilder, self.workerdata, fn, task, taskname, taskhash, appends, 
taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
+pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, 
self.databuilder, self.workerdata, fn, task, taskname, taskhash, depid, 
appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
 
 self.build_pids[pid] = task
 self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 3ae12d8e69f..909fa3aec22 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -2028,6 +2028,7 @@ class RunQueueExecuteTasks(RunQueueExecute):
 
 taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
 taskhash = self.rqdata.get_task_hash(task)
+depid = self.rqdata.get_task_depid(task)
 if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and 
not (self.cooker.configuration.dry_run or self.rqdata.setscene_enforce):
 if not mc in self.rq.fakeworker:
 try:
@@ -2037,10 +2038,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
 self.rq.state = runQueueFailed
 self.stats.taskFailed()
 return True
-self.rq.fakeworker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, taskhash, False, 
self.cooker.collection.get_file_appends(taskfn), taskdepdata, 
self.rqdata.setscene_enforce)) + b"")
+self.rq.fakeworker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, taskhash, depid, False, 
self.cooker.collection.get_file_appends(taskfn), taskdepdata, 
self.rqdata.setscene_enforce)) + b"")
 self.rq.fakeworker[mc].process.stdin.flush()
 else:
-self.rq.worker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, taskhash, False, 
self.cooker.collection.get_file_appends(taskfn), taskdepdata, 
self.rqdata.setscene_enforce)) + b"")
+self.rq.worker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, taskhash, depid, False, 
self.cooker.collection.get_file_appends(taskfn), taskdepdata, 
self.rqdata.setscene_enforce)) + b"")
 self.rq.worker[mc].process.stdin.flush()
 
 self.build_stamps[task] = bb.build.stampfile(taskname, 
self.rqdata.dataCaches[mc], taskfn, noextra=True)
@@ -2455,13 +2456,14 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
 
 taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
 taskhash = self.rqdata.get_task_hash(task)
+depid = self.rqdata.get_task_depid(task)
 if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and 
not self.cooker.configuration.dry_run:
 if not mc in self.rq.fakeworker:
 self.rq.start_fakeworker(self, mc)
-self.rq.fakeworker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, taskhash, True, 
self.cooker.collection.get_file_appends(taskfn), taskdepdata, 

[OE-core] [RFC v2 07/16] bitbake: tests/persist_data: Add tests

[Joshua Watt]

Adds a test suite for testing the persistent data cache

Signed-off-by: Joshua Watt 
---
 bitbake/bin/bitbake-selftest |   1 +
 bitbake/lib/bb/tests/persist_data.py | 188 +++
 2 files changed, 189 insertions(+)
 create mode 100644 bitbake/lib/bb/tests/persist_data.py

diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest
index 7564de304c8..06a1c9a78dd 100755
--- a/bitbake/bin/bitbake-selftest
+++ b/bitbake/bin/bitbake-selftest
@@ -32,6 +32,7 @@ tests = ["bb.tests.codeparser",
  "bb.tests.event",
  "bb.tests.fetch",
  "bb.tests.parse",
+ "bb.tests.persist_data",
  "bb.tests.utils",
  "layerindexlib.tests.layerindexobj",
  "layerindexlib.tests.restapi",
diff --git a/bitbake/lib/bb/tests/persist_data.py 
b/bitbake/lib/bb/tests/persist_data.py
new file mode 100644
index 000..055f1d9ce47
--- /dev/null
+++ b/bitbake/lib/bb/tests/persist_data.py
@@ -0,0 +1,188 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Test for lib/bb/persist_data/
+#
+# Copyright (C) 2018 Garmin Ltd.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+import unittest
+import bb.data
+import bb.persist_data
+import bb.fork
+import tempfile
+import threading
+
+class PersistDataTest(unittest.TestCase):
+def _create_data(self):
+return bb.persist_data.persist('TEST_PERSIST_DATA', self.d)
+
+def setUp(self):
+self.d = bb.data.init()
+self.tempdir = tempfile.TemporaryDirectory()
+self.d['PERSISTENT_DIR'] = self.tempdir.name
+self.data = self._create_data()
+self.items = {
+'A1': '1',
+'B1': '2',
+'C2': '3'
+}
+self.stress_count = 1
+self.thread_count = 5
+
+for k,v in self.items.items():
+self.data[k] = v
+
+def tearDown(self):
+self.tempdir.cleanup()
+
+def _iter_helper(self, seen, iterator):
+with iter(iterator):
+for v in iterator:
+self.assertTrue(v in seen)
+seen.remove(v)
+self.assertEqual(len(seen), 0, '%s not seen' % seen)
+
+def test_get(self):
+for k, v in self.items.items():
+self.assertEqual(self.data[k], v)
+
+self.assertIsNone(self.data.get('D'))
+with self.assertRaises(KeyError):
+self.data['D']
+
+def test_set(self):
+for k, v in self.items.items():
+self.data[k] += '-foo'
+
+for k, v in self.items.items():
+self.assertEqual(self.data[k], v + '-foo')
+
+def test_delete(self):
+self.data['D'] = '4'
+self.assertEqual(self.data['D'], '4')
+del self.data['D']
+self.assertIsNone(self.data.get('D'))
+with self.assertRaises(KeyError):
+self.data['D']
+
+def test_contains(self):
+for k in self.items:
+self.assertTrue(k in self.data)
+self.assertTrue(self.data.has_key(k))
+self.assertFalse('NotFound' in self.data)
+self.assertFalse(self.data.has_key('NotFound'))
+
+def test_len(self):
+self.assertEqual(len(self.data), len(self.items))
+
+def test_iter(self):
+self._iter_helper(set(self.items.keys()), self.data)
+
+def test_itervalues(self):
+self._iter_helper(set(self.items.values()), self.data.itervalues())
+
+def test_iteritems(self):
+self._iter_helper(set(self.items.items()), self.data.iteritems())
+
+def test_get_by_pattern(self):
+self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1'))
+
+def _stress_read(self, data):
+for i in range(self.stress_count):
+for k in self.items:
+data[k]
+
+def _stress_write(self, data):
+for i in range(self.stress_count):
+for k, v in self.items.items():
+data[k] = v + str(i)
+
+def _validate_stress(self):
+for k, v in self.items.items():
+self.assertEqual(self.data[k], v + str(self.stress_count - 1))
+
+def test_stress(self):
+self._stress_read(self.data)
+self._stress_write(self.data)
+self._validate_stress()
+
+def test_stress_threads(self):
+def read_thread():
+data = 

[OE-core] [RFC v2 11/16] bitbake: runqueue: Track task dependency ID

[Joshua Watt]

Requests the task dependency ID from siggen and tracks it

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/runqueue.py | 25 +
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index b173cc0a951..3ae12d8e69f 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -346,6 +346,7 @@ class RunTaskEntry(object):
 self.depends = set()
 self.revdeps = set()
 self.hash = None
+self.depid = None
 self.task = None
 self.weight = 1
 
@@ -385,6 +386,9 @@ class RunQueueData:
 def get_task_hash(self, tid):
 return self.runtaskentries[tid].hash
 
+def get_task_depid(self, tid):
+return self.runtaskentries[tid].depid
+
 def get_user_idstring(self, tid, task_name_suffix = ""):
 return tid + task_name_suffix
 
@@ -1150,18 +1154,21 @@ class RunQueueData:
 if len(self.runtaskentries[tid].depends - dealtwith) == 0:
 dealtwith.add(tid)
 todeal.remove(tid)
-procdep = []
-for dep in self.runtaskentries[tid].depends:
-procdep.append(fn_from_tid(dep) + "." + 
taskname_from_tid(dep))
-(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
-self.runtaskentries[tid].hash = 
bb.parse.siggen.get_taskhash(taskfn, taskname, procdep, self.dataCaches[mc])
-task = self.runtaskentries[tid].task
+self.prepare_task_hash(tid)
 
 bb.parse.siggen.writeout_file_checksum_cache()
 
 #self.dump_data()
 return len(self.runtaskentries)
 
+def prepare_task_hash(self, tid):
+procdep = []
+for dep in self.runtaskentries[tid].depends:
+procdep.append(fn_from_tid(dep) + "." + taskname_from_tid(dep))
+(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(taskfn, 
taskname, procdep, self.dataCaches[mc])
+self.runtaskentries[tid].depid = bb.parse.siggen.get_depid(fn + "." + 
taskname)
+
 def dump_data(self):
 """
 Dump some debug information on the internal data structures
@@ -2075,7 +2082,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
 deps = self.rqdata.runtaskentries[revdep].depends
 provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
 taskhash = self.rqdata.runtaskentries[revdep].hash
-taskdepdata[revdep] = [pn, taskname, fn, deps, provides, 
taskhash]
+depid = self.rqdata.runtaskentries[revdep].depid
+taskdepdata[revdep] = [pn, taskname, fn, deps, provides, 
taskhash, depid]
 for revdep2 in deps:
 if revdep2 not in taskdepdata:
 additional.append(revdep2)
@@ -2517,7 +2525,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
 deps = getsetscenedeps(revdep)
 provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
 taskhash = self.rqdata.runtaskentries[revdep].hash
-taskdepdata[revdep] = [pn, taskname, fn, deps, provides, 
taskhash]
+depid = self.rqdata.runtaskentries[revdep].depid
+taskdepdata[revdep] = [pn, taskname, fn, deps, provides, 
taskhash, depid]
 for revdep2 in deps:
 if revdep2 not in taskdepdata:
 additional.append(revdep2)
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 09/16] bitbake: siggen: Split out stampfile hash fetch

[Joshua Watt]

The mechanism used to get the hash for a stamp file is split out so that
it can be overridden by derived classes

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/siggen.py | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index e9bb51d736f..7fa5b35337d 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -340,6 +340,13 @@ class SignatureGeneratorBasic(SignatureGenerator):
 class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
 name = "basichash"
 
+def get_stampfile_hash(self, task):
+if task in self.taskhash:
+return self.taskhash[task]
+
+# If task is not in basehash, then error
+return self.basehash[task]
+
 def stampfile(self, stampbase, fn, taskname, extrainfo, clean=False):
 if taskname != "do_setscene" and taskname.endswith("_setscene"):
 k = fn + "." + taskname[:-9]
@@ -347,11 +354,9 @@ class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
 k = fn + "." + taskname
 if clean:
 h = "*"
-elif k in self.taskhash:
-h = self.taskhash[k]
 else:
-# If k is not in basehash, then error
-h = self.basehash[k]
+h = self.get_stampfile_hash(k)
+
 return ("%s.%s.%s.%s" % (stampbase, taskname, h, 
extrainfo)).rstrip('.')
 
 def stampcleanmask(self, stampbase, fn, taskname, extrainfo):
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 10/16] bitbake: siggen: Split out task depend ID

[Joshua Watt]

Abstracts the function to get the dependency ID for a task so it can
return something other that the taskhash

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/siggen.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 7fa5b35337d..4ef175edbbf 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -41,6 +41,9 @@ class SignatureGenerator(object):
 def finalise(self, fn, d, varient):
 return
 
+def get_depid(self, task):
+return self.taskhash[task]
+
 def get_taskhash(self, fn, task, deps, dataCache):
 return "0"
 
@@ -215,7 +218,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
 continue
 if dep not in self.taskhash:
 bb.fatal("%s is not in taskhash, caller isn't calling in 
dependency order?", dep)
-data = data + self.taskhash[dep]
+data = data + self.get_depid(dep)
 self.runtaskdeps[k].append(dep)
 
 if task in dataCache.file_checksums[fn]:
@@ -290,7 +293,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
 data['file_checksum_values'] = [(os.path.basename(f), cs) for f,cs 
in self.file_checksum_values[k]]
 data['runtaskhashes'] = {}
 for dep in data['runtaskdeps']:
-data['runtaskhashes'][dep] = self.taskhash[dep]
+data['runtaskhashes'][dep] = self.get_depid(dep)
 data['taskhash'] = self.taskhash[k]
 
 taint = self.read_taint(fn, task, referencestamp)
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 06/16] bitbake: persist_data: Close databases across fork

[Joshua Watt]

sqlite gets really angry if a database is closed across a fork() call,
and will give all sorts of messages ranging from I/O errors to database
corruption errors. To deal with this, close all database connections
before forking, and reopen them (lazily) on the other side.

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/persist_data.py | 44 +++---
 1 file changed, 41 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index f0d3ce665d9..6bd3924ffb3 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -30,6 +30,8 @@ from bb.compat import total_ordering
 from collections import Mapping
 import sqlite3
 import contextlib
+import bb.fork
+import weakref
 
 sqlversion = sqlite3.sqlite_version_info
 if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
@@ -38,6 +40,28 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and 
sqlversion[1] < 3):
 
 logger = logging.getLogger("BitBake.PersistData")
 
+# Carrying an open database connection across a fork() confuses sqlite and
+# results in fun errors like 'database disk image is malformed'.
+# To remedy this, close all connections before forking, then they will be
+# (lazily) reopen them on the other side. This will cause a lot of problems if
+# there are threads running and trying to access the database at the same time,
+# but if you are mixing threads and fork() you have no one to blame but
+# yourself. If that is discovered to be a problem in the future, some sort of
+# per-table reader-writer lock could be used to block the fork() until all
+# pending transactions complete
+sql_table_weakrefs = []
+def _fork_before_handler():
+for ref in sql_table_weakrefs:
+t = ref()
+if t is not None and t.connection is not None:
+t.connection.close()
+t.connection = None
+
+bb.fork.register_at_fork(before=_fork_before_handler)
+
+def _remove_table_weakref(ref):
+sql_table_weakrefs.remove(ref)
+
 @total_ordering
 class SQLTable(collections.MutableMapping):
 class _Decorators(object):
@@ -48,6 +72,10 @@ class SQLTable(collections.MutableMapping):
 exception occurs.
 """
 def wrap_func(self, *args, **kwargs):
+# Reconnect if necessary
+if self.connection is None:
+self.reconnect()
+
 count = 0
 while True:
 try:
@@ -55,8 +83,7 @@ class SQLTable(collections.MutableMapping):
 except sqlite3.OperationalError as exc:
 if 'is locked' in str(exc) and count < 500:
 count = count + 1
-self.connection.close()
-self.connection = connect(self.cachefile)
+self.reconnect()
 continue
 raise
 return wrap_func
@@ -90,6 +117,11 @@ class SQLTable(collections.MutableMapping):
 
 self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY 
KEY NOT NULL, value TEXT);" % table)
 
+def reconnect(self):
+if self.connection is not None:
+self.connection.close()
+self.connection = connect(self.cachefile)
+
 @_Decorators.retry
 @_Decorators.transaction
 def _execute_single(self, cursor, *query):
@@ -292,4 +324,10 @@ def persist(domain, d):
 
 bb.utils.mkdirhier(cachedir)
 cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3")
-return SQLTable(cachefile, domain)
+t = SQLTable(cachefile, domain)
+
+# Add a weak reference to the table list. The weak reference will not keep
+# the object alive by itself, so it prevents circular reference counts
+sql_table_weakrefs.append(weakref.ref(t, _remove_table_weakref))
+
+return t
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 08/16] bitbake: bitbake-worker: Pass taskhash as runtask parameter

[Joshua Watt]

Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.

Signed-off-by: Joshua Watt 
---
 bitbake/bin/bitbake-worker |  8 
 bitbake/lib/bb/runqueue.py | 15 ++-
 2 files changed, 10 insertions(+), 13 deletions(-)

diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index baa1a84e6dd..41ef6d848ac 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -136,7 +136,7 @@ def sigterm_handler(signum, frame):
 os.killpg(0, signal.SIGTERM)
 sys.exit()
 
-def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, 
appends, taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
+def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, 
taskhash, appends, taskdepdata, extraconfigdata, quieterrors=False, 
dry_run_exec=False):
 # We need to setup the environment BEFORE the fork, since
 # a fork() or exec*() activates PSEUDO...
 
@@ -234,7 +234,7 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, 
task, taskname, append
 ret = 0
 
 the_data = bb_cache.loadDataFull(fn, appends)
-the_data.setVar('BB_TASKHASH', workerdata["runq_hash"][task])
+the_data.setVar('BB_TASKHASH', taskhash)
 
 bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN"), 
taskname.replace("do_", "")))
 
@@ -425,10 +425,10 @@ class BitbakeWorker(object):
 sys.exit(0)
 
 def handle_runtask(self, data):
-fn, task, taskname, quieterrors, appends, taskdepdata, dry_run_exec = 
pickle.loads(data)
+fn, task, taskname, taskhash, quieterrors, appends, taskdepdata, 
dry_run_exec = pickle.loads(data)
 workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
 
-pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, 
self.databuilder, self.workerdata, fn, task, taskname, appends, taskdepdata, 
self.extraconfigdata, quieterrors, dry_run_exec)
+pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, 
self.databuilder, self.workerdata, fn, task, taskname, taskhash, appends, 
taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
 
 self.build_pids[pid] = task
 self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 400709c1601..b173cc0a951 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1224,17 +1224,12 @@ class RunQueue:
 bb.utils.nonblockingfd(worker.stdout)
 workerpipe = runQueuePipe(worker.stdout, None, self.cfgData, self, 
rqexec)
 
-runqhash = {}
-for tid in self.rqdata.runtaskentries:
-runqhash[tid] = self.rqdata.runtaskentries[tid].hash
-
 workerdata = {
 "taskdeps" : self.rqdata.dataCaches[mc].task_deps,
 "fakerootenv" : self.rqdata.dataCaches[mc].fakerootenv,
 "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
 "fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv,
 "sigdata" : bb.parse.siggen.get_taskdata(),
-"runq_hash" : runqhash,
 "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
 "logdefaultverbose" : bb.msg.loggerDefaultVerbose,
 "logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
@@ -2025,6 +2020,7 @@ class RunQueueExecuteTasks(RunQueueExecute):
 taskdepdata = self.build_taskdepdata(task)
 
 taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+taskhash = self.rqdata.get_task_hash(task)
 if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and 
not (self.cooker.configuration.dry_run or self.rqdata.setscene_enforce):
 if not mc in self.rq.fakeworker:
 try:
@@ -2034,10 +2030,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
 self.rq.state = runQueueFailed
 self.stats.taskFailed()
 return True
-self.rq.fakeworker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, False, 
self.cooker.collection.get_file_appends(taskfn), taskdepdata, 
self.rqdata.setscene_enforce)) + b"")
+self.rq.fakeworker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, taskhash, False, 
self.cooker.collection.get_file_appends(taskfn), taskdepdata, 
self.rqdata.setscene_enforce)) + b"")
 self.rq.fakeworker[mc].process.stdin.flush()
 else:
-self.rq.worker[mc].process.stdin.write(b"" + 
pickle.dumps((taskfn, task, taskname, False, 
self.cooker.collection.get_file_appends(taskfn), 

[OE-core] [RFC v2 05/16] bitbake: persist_data: Disable enable_shared_cache

[Joshua Watt]

Turns off the shared cache. It isn't a significant factor in performance
(now that WAL is enabled), and is a really bad idea to have enabled in
processes that fork() (as bitbake it prone to do).

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/persist_data.py | 6 --
 1 file changed, 6 deletions(-)

diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index 9a4e7dd5941..f0d3ce665d9 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -37,12 +37,6 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and 
sqlversion[1] < 3):
 
 
 logger = logging.getLogger("BitBake.PersistData")
-if hasattr(sqlite3, 'enable_shared_cache'):
-try:
-sqlite3.enable_shared_cache(True)
-except sqlite3.OperationalError:
-pass
-
 
 @total_ordering
 class SQLTable(collections.MutableMapping):
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 03/16] bitbake: persist_data: Add key constraints

[Joshua Watt]

Constructs the "key" column in the persistent database as a non-NULL
primary key. This significantly speeds up lookup operations in large
databases.

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/persist_data.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index 1a6319f9498..2bc3e766a93 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -94,7 +94,7 @@ class SQLTable(collections.MutableMapping):
 self.table = table
 self.connection = connect(self.cachefile)
 
-self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT, value 
TEXT);" % table)
+self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY 
KEY NOT NULL, value TEXT);" % table)
 
 @_Decorators.retry
 @_Decorators.transaction
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 04/16] bitbake: persist_data: Enable Write Ahead Log

[Joshua Watt]

Enabling the write ahead log improves database reliability, speeds up
writes (since they mostly happen sequentially), and speeds up readers
(since they are no longer blocked by most write operations). The
persistent database is very read heavy, so the auto-checkpoint size is
reduced from the default (usually 1000) to 100 so that reads remain
fast.

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/persist_data.py | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index 2bc3e766a93..9a4e7dd5941 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -278,7 +278,12 @@ class PersistData(object):
 
 def connect(database):
 connection = sqlite3.connect(database, timeout=5)
-connection.execute("pragma synchronous = off;")
+connection.execute("pragma synchronous = normal;")
+# Enable WAL and keep the autocheckpoint length small (the default is
+# usually 1000). Persistent caches are usually read-mostly, so keeping
+# this short will keep readers running quickly
+connection.execute("pragma journal_mode = WAL;")
+connection.execute("pragma wal_autocheckpoint = 100;")
 connection.text_factory = str
 return connection
 
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 02/16] bitbake: persist_data: Fix leaking cursors causing deadlock

[Joshua Watt]

The original implementation of persistent data executed all SQL
statements via sqlite3.Connection.execute(). Behind the scenes, this
function created a sqlite3 Cursor object, executed the statement, then
returned the cursor. However, the implementation did not account for
this and failed to close the cursor object when it was done. The cursor
would eventually be closed when the garbage collector got around to
destroying it. However, sqlite has a limit on the number of cursors that
can exist at any given time, and once this limit is reached it will
block a query to wait for a cursor to be destroyed. Under heavy database
queries, this can result in Python deadlocking with itself, since the
SQL query will block waiting for a free cursor, but Python can no longer
run garbage collection (as it is blocked) to free one.

This restructures the SQLTable class to use two decorators to aid in
performing actions correctly. The first decorator (@retry) wraps a
member function in the retry logic that automatically restarts the
function in the event that the database is locked.

The second decorator (@transaction) wraps the function so that it occurs
in a database transaction, which will automatically COMMIT the changes
on success and ROLLBACK on failure. This function additionally creates
an explicit cursor, passes it to the wrapped function, and cleans it up
when the function is finished.

Note that it is still possible to leak cursors when iterating. This is
much less frequent, but can still be mitigated by wrapping the iteration
in a `with` statement:

 with db.iteritems() as it:
 for (k, v) in it:
 ...

As a side effect, since most statements are wrapped in a transaction,
setting the isolation_level when the connection is created is no longer
necessary.

Signed-off-by: Joshua Watt 
---
 bitbake/lib/bb/persist_data.py | 188 +++--
 1 file changed, 135 insertions(+), 53 deletions(-)

diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index bef7018614d..1a6319f9498 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -29,6 +29,7 @@ import warnings
 from bb.compat import total_ordering
 from collections import Mapping
 import sqlite3
+import contextlib
 
 sqlversion = sqlite3.sqlite_version_info
 if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
@@ -45,75 +46,154 @@ if hasattr(sqlite3, 'enable_shared_cache'):
 
 @total_ordering
 class SQLTable(collections.MutableMapping):
+class _Decorators(object):
+@staticmethod
+def retry(f):
+"""
+Decorator that restarts a function if a database locked sqlite
+exception occurs.
+"""
+def wrap_func(self, *args, **kwargs):
+count = 0
+while True:
+try:
+return f(self, *args, **kwargs)
+except sqlite3.OperationalError as exc:
+if 'is locked' in str(exc) and count < 500:
+count = count + 1
+self.connection.close()
+self.connection = connect(self.cachefile)
+continue
+raise
+return wrap_func
+
+@staticmethod
+def transaction(f):
+"""
+Decorator that starts a database transaction and creates a database
+cursor for performing queries. If no exception is thrown, the
+database results are commited. If an exception occurs, the database
+is rolled back. In all cases, the cursor is closed after the
+function ends.
+
+Note that the cursor is passed as an extra argument to the function
+after `self` and before any of the normal arguments
+"""
+def wrap_func(self, *args, **kwargs):
+# Context manager will COMMIT the database on success,
+# or ROLLBACK on an exception
+with self.connection:
+# Automatically close the cursor when done
+with contextlib.closing(self.connection.cursor()) as 
cursor:
+return f(self, cursor, *args, **kwargs)
+return wrap_func
+
 """Object representing a table/domain in the database"""
 def __init__(self, cachefile, table):
 self.cachefile = cachefile
 self.table = table
-self.cursor = connect(self.cachefile)
-
-self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);"
-  % table)
-
-def _execute(self, *query):
-"""Execute a query, waiting to acquire a lock if necessary"""
-count = 0
-while True:
-try:
-return self.cursor.execute(*query)
-except sqlite3.OperationalError as exc:
-if 'database is locked' in 

[OE-core] [RFC v2 01/16] bitbake: fork: Add os.fork() wrappers

[Joshua Watt]

Adds a compatibility wrapper around os.fork() that backports the ability
to register fork event handlers (os.register_at_fork()) from Python 3.7

Signed-off-by: Joshua Watt 
---
 bitbake/bin/bitbake-worker |  2 +-
 bitbake/lib/bb/fork.py | 71 ++
 2 files changed, 72 insertions(+), 1 deletion(-)
 create mode 100644 bitbake/lib/bb/fork.py

diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index e925054b7f9..baa1a84e6dd 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -181,7 +181,7 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, 
task, taskname, append
 pipein, pipeout = os.pipe()
 pipein = os.fdopen(pipein, 'rb', 4096)
 pipeout = os.fdopen(pipeout, 'wb', 0)
-pid = os.fork()
+pid = bb.fork.fork()
 except OSError as e:
 logger.critical("fork failed: %d (%s)" % (e.errno, e.strerror))
 sys.exit(1)
diff --git a/bitbake/lib/bb/fork.py b/bitbake/lib/bb/fork.py
new file mode 100644
index 000..5ac5aba1832
--- /dev/null
+++ b/bitbake/lib/bb/fork.py
@@ -0,0 +1,71 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2018 Garmin Ltd.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+"""
+Python wrappers for os.fork() that allow the insertion of callbacks for fork 
events.
+This is designed to exacmimic os.register_at_fork() available in Python 3.7 
with the
+intent that it can be removed when that version becomes standard
+"""
+
+import sys
+import os
+
+before_calls = []
+after_in_parent_calls = []
+after_in_child_calls = []
+
+def _do_calls(l, reverse=False):
+# Make a copy in case the list is modified in the callback
+copy = l[:]
+if reverse:
+copy = reversed(copy)
+
+for f in copy:
+# All exception in calls are ignored
+try:
+f()
+except:
+pass
+
+def fork():
+if sys.hexversion >= 0x030700F0:
+return os.fork()
+
+_do_calls(before_calls, reverse=True)
+
+ret = os.fork()
+if ret == 0:
+_do_calls(after_in_child_calls)
+else:
+_do_calls(after_in_parent_calls)
+return ret
+
+def register_at_fork(*, before=None, after_in_parent=None, 
after_in_child=None):
+if sys.hexversion >= 0x030700F0:
+os.register_at_fork(before=before, after_in_parent=after_in_parent, 
after_in_child=after_in_child)
+return
+
+if before is not None:
+before_calls.append(before)
+
+if after_in_parent is not None:
+after_in_parent_calls.append(after_in_parent)
+
+if after_in_child is not None:
+after_in_child_calls.append(after_in_child)
+
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC v2 00/16] Hash Equivalency Server

[Joshua Watt]

These patches are a first pass at implementing a hash equivalence server
in bitbake & OE.

Apologies for cross-posting this to both the bitbake-devel and
openembedded-devel; this work necessarily intertwines both places, and
it is really necessary to look at both parts to get an idea of what is
going on. For convenience, the bitbake patches are listed first,
followed by the oe-core patches.

The basic premise is that any given task no longer hashes a dependent
task's taskhash to determine it's own taskhash, but instead hashes the
dependent task's "dependency ID" (which doesn't strictly need to be a
hash, but is for consistency. We can have the discussion as to whether
this should be called a "dependency hash" if anyone wants). This allows
multiple taskhashes to map to the same dependency ID, meaning that
trivial changes to a recipe that would change the taskhash don't
necessarily need to change the dependency ID, and thus don't need to
cause downstream tasks to be rebuilt (with caveats, see below).

In the absence of any interaction by the user, the dependency ID for a
task is just that task's taskhash, which effectively maintains the
current behavior. However, if the user enables the "OEEquivHash"
signature generator, they can direct it to look at a hash equivalency
server (of which a reference implementation is provided). The sstate
code will provide the server with an output hash that it calculates, and
the server will record all tasks with the same output hash as
"equivalent" and report the same dependency ID for them when requested.
When initializing tasks, bitbake can ask the server about the dependency
ID for new tasks it has never seen before and potentially skip
rebuilding, or restore the task from an equivalent sstate file. To
facilitate restoring tasks from sstate, sstate objects are now named
based on the tasks dependency ID instead of the taskhash (which, again
has no effect if the server is in use).

This patchset doesn't make any attempt to dynamically update task
dependency IDs after bitbake initializes the tasks, and as such there
are some cases where this isn't accelerating the build as much as it
possibly could. I think it will be possible to add support for this, but
this preliminary support needs to come first.

Some patches have additional NOTEs that indicate places where I wasn't
sure what to do.

You can also see these patches (and my first attempts at dynamic task
re-hashing) on the "jpew/hash-equivalence" branch in poky-contrib.

As always, thanks for your feedback and time

VERSION 2:

At the core, this patch does the same thing as V1 with some very minor
tweaks. The main things that have changed are:
 1) Per request, the Hash Equivalence Server reference implementation is
now based entirely on built in Python modules and requires no
external libraries. It also has a wrapper script to launch it
(bitbake-hashserv) and unittests.
 2) There is a major rework of persist_data in bitbake. I think these
patches could be submitted independently, but I doubt anyone is
clamoring for them. The general gist of them is that there were a
lot of strange edge cases that I found when using persist_data as an
IPC mechanism between the main bitbake process and the
bitbake-worker processes. I went ahead and added extensive unit
tests for this as well.

Joshua Watt (16):
  bitbake: fork: Add os.fork() wrappers
  bitbake: persist_data: Fix leaking cursors causing deadlock
  bitbake: persist_data: Add key constraints
  bitbake: persist_data: Enable Write Ahead Log
  bitbake: persist_data: Disable enable_shared_cache
  bitbake: persist_data: Close databases across fork
  bitbake: tests/persist_data: Add tests
  bitbake: bitbake-worker: Pass taskhash as runtask parameter
  bitbake: siggen: Split out stampfile hash fetch
  bitbake: siggen: Split out task depend ID
  bitbake: runqueue: Track task dependency ID
  bitbake: runqueue: Pass dependency ID to task
  bitbake: runqueue: Pass dependency ID to hash validate
  classes/sstate: Handle depid in hash check
  bitbake: hashserv: Add hash equivalence reference server
  sstate: Implement hash equivalence sstate

 bitbake/bin/bitbake-hashserv |  67 
 bitbake/bin/bitbake-selftest |   3 +
 bitbake/bin/bitbake-worker   |  11 +-
 bitbake/lib/bb/fork.py   |  71 
 bitbake/lib/bb/persist_data.py   | 239 ---
 bitbake/lib/bb/runqueue.py   |  56 ---
 bitbake/lib/bb/siggen.py |  20 ++-
 bitbake/lib/bb/tests/persist_data.py | 188 +
 bitbake/lib/hashserv/__init__.py | 152 +
 bitbake/lib/hashserv/tests.py| 141 
 meta/classes/sstate.bbclass  | 102 +++-
 meta/conf/bitbake.conf   |   4 +-
 meta/lib/oe/sstatesig.py | 166 +++
 13 files changed, 1117 insertions(+), 103 deletions(-)
 create mode 100755 

Re: [OE-core] How to upgrade Qt modules in daisy branch

[Ankur Tyagi]

Thank you Andreas, Martin and Alex for your responses.

I will follow your leads and hopefully will be able to do it.

Regards
Ankur

On Fri 10 Aug, 2018, 4:16 AM Andreas Müller, 
wrote:

> On Thu, Aug 9, 2018 at 2:40 PM, Martin Jansa 
> wrote:
> > For daisy you probably don't have support for PACKAGECONFIG_CONFARGS
> which
> > was added in Yocto 2.1 krogoth.
> >
> I remember doing similar for other reasons. We have images based on
> jethro (ok - daisy is a bit older) with later Qt's. Our builds are
> based on [1] but I created branches up to Qt5.9 (stable release - but
> not updated to latest) [2] for jethro.
>
> It's not daisy - not Qt5.11 - no guarantees - but maybe a start.
>
> [1] https://github.com/schnitzeltony/meta-qt5/tree/jethro+5.7
> [2] https://github.com/schnitzeltony/meta-qt5/tree/jethro+5.9
>
> Andreas
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] multilib_header: recognize BPF as a target

[Daniel Díaz]

When building with `clang -target bpf` using the
multilib_header, a recursion was unavoidable because
bits/wordsize.h would #include itself, still lacking
a definition for __MHWORDSIZE or __WORDSIZE.

Signed-off-by: Daniel Díaz 
---
 scripts/multilib_header_wrapper.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/multilib_header_wrapper.h 
b/scripts/multilib_header_wrapper.h
index f516673..9660225 100644
--- a/scripts/multilib_header_wrapper.h
+++ b/scripts/multilib_header_wrapper.h
@@ -22,7 +22,9 @@
  */
 
 
-#if defined (__arm__)
+#if defined (__bpf__)
+#define __MHWORDSIZE   64
+#elif defined (__arm__)
 #define __MHWORDSIZE   32
 #elif defined (__aarch64__) && defined ( __LP64__)
 #define __MHWORDSIZE   64
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] glibc: Make bits/wordsize.h multilibbed again

[Daniel Díaz]

As reported by ChenQi, leaving bits/wordsize.h out of being
multilibbed introduced a problem in building the SDK for
arm64:
  Error: Transaction check error:
file /usr/include/bits/wordsize.h conflicts between attempted installs of 
lib32-libc6-dev-2.27-r0.armv7vet2hf_vfp and libc6-dev-2.27-r0.aarch64

This effectively reverts commit a74c77d6.

Signed-off-by: Daniel Díaz 
---
 meta/recipes-core/glibc/glibc-package.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc 
b/meta/recipes-core/glibc/glibc-package.inc
index a4f61f8..8ac01ce 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -136,7 +136,7 @@ do_install_append_armeb () {
 }
 
 do_install_armmultilib () {
-   oe_multilib_header bits/endian.h bits/fcntl.h bits/fenv.h 
bits/fp-fast.h bits/hwcap.h bits/ipc.h bits/link.h
+   oe_multilib_header bits/endian.h bits/fcntl.h bits/fenv.h 
bits/fp-fast.h bits/hwcap.h bits/ipc.h bits/link.h bits/wordsize.h
oe_multilib_header bits/local_lim.h bits/mman.h bits/msq.h 
bits/pthreadtypes.h bits/pthreadtypes-arch.h  bits/sem.h  bits/semaphore.h 
bits/setjmp.h
oe_multilib_header bits/shm.h bits/sigstack.h bits/stat.h bits/statfs.h 
bits/typesizes.h
 
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [rocko][PATCH] glibc: Avoid multilibbing on wordsize.h

[Daniel Díaz]

Hello!


On 26 July 2018 at 09:38, Daniel Díaz  wrote:
> Once another header #includes , there is a
> potential recursion going on because the
> multilib_header_wrapper.h #includes  again!
>
> This should not happen because an __arm__ (32-bits) or an
> __aarch64__ (64-bits) environment guarantees that we will
> be getting the correct definition, but when building against
> a different target (like BPF), recursion is what happens.
>
> This can be seen, for instance, when building eBPF programs
> from the kernel with `clang -target bpf', such as the ones
> located in linux/tools/testing/selftests/bpf/.
>
> (From OE-Core rev: a74c77d6168101e88c3a3bce7130f4f52cfab95d)
>
> Signed-off-by: Daniel Díaz 
> Signed-off-by: Aníbal Limón 
> Signed-off-by: Richard Purdie 
> ---
>  meta/recipes-core/glibc/glibc-package.inc | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/meta/recipes-core/glibc/glibc-package.inc 
> b/meta/recipes-core/glibc/glibc-package.inc
> index b6d8074..c360dd9 100644
> --- a/meta/recipes-core/glibc/glibc-package.inc
> +++ b/meta/recipes-core/glibc/glibc-package.inc
> @@ -136,8 +136,7 @@ do_install_append_armeb () {
>  }
>
>  do_install_armmultilib () {
> -
> -   oe_multilib_header bits/endian.h bits/fcntl.h bits/fenv.h 
> bits/fp-fast.h bits/hwcap.h bits/ipc.h bits/link.h bits/wordsize.h
> +   oe_multilib_header bits/endian.h bits/fcntl.h bits/fenv.h 
> bits/fp-fast.h bits/hwcap.h bits/ipc.h bits/link.h
> oe_multilib_header bits/local_lim.h bits/mman.h bits/msq.h 
> bits/pthreadtypes.h bits/pthreadtypes-arch.h  bits/sem.h  bits/semaphore.h 
> bits/setjmp.h
> oe_multilib_header bits/shm.h bits/sigstack.h bits/stat.h 
> bits/statfs.h bits/typesizes.h
>
> --
> 2.7.4


Please disregard this patch. An alternative will be posted shortly.

Greetings!

Daniel Díaz
daniel.d...@linaro.org
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] glibc: Avoid multilibbing on wordsize.h

[Daniel Díaz]

Hello!


On 30 July 2018 at 21:01, ChenQi  wrote:
> This patch causes a regression in multilib SDK for qemuarm64.
>
> Error: Transaction check error:
>   file /usr/include/bits/wordsize.h conflicts between attempted installs of
> lib32-libc6-dev-2.27-r0.armv7vet2hf_vfp and libc6-dev-2.27-r0.aarch64
>
> Please consider sending a follow-up patch to fix this regression.

Thanks for the heads up. Follow-up patches coming along.

Greetings!

Daniel Díaz
daniel.d...@linaro.org
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to upgrade Qt modules in daisy branch

[Andreas Müller]

On Thu, Aug 9, 2018 at 2:40 PM, Martin Jansa  wrote:
> For daisy you probably don't have support for PACKAGECONFIG_CONFARGS which
> was added in Yocto 2.1 krogoth.
>
I remember doing similar for other reasons. We have images based on
jethro (ok - daisy is a bit older) with later Qt's. Our builds are
based on [1] but I created branches up to Qt5.9 (stable release - but
not updated to latest) [2] for jethro.

It's not daisy - not Qt5.11 - no guarantees - but maybe a start.

[1] https://github.com/schnitzeltony/meta-qt5/tree/jethro+5.7
[2] https://github.com/schnitzeltony/meta-qt5/tree/jethro+5.9

Andreas
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] upgrade openssl to 1.1.0h (open embedded yocto project)

[Mackey ME]

 Hi,

I am writing this email with reference to upgrading openssl to 1.1 in
open-embedded packages in yocto project.
I am using older version of open embedded (yocto project version:
2.3872.08518) for one of my projects and would like to upgrade openssl only.

Please help me identify the dependencies and let me know the changes
required for a successful upgrade.

I am building this project for linux arm7 platform.

Thanks,

Manoj Enaganti
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [pyro][PATCH] gcc: Backport upstream r261799

[Raphael Kubo da Costa]

Raphael Kubo da Costa  writes:

> The upstream commit merges several bug fixes into the GCC 6 branch; we are
> specifically interested in the fix for GCC bug 83623, which was causing
> Chromium 68+ to fail to build on ARM and aarch64 with
>
> In file included from 
> ../../third_party/skia/third_party/skcms/src/Transform.c:176:0:
> ../../third_party/skia/third_party/skcms/src/Transform_inl.h: In function 
> 'exec_ops':
> ../../third_party/skia/third_party/skcms/src/Transform_inl.h:459:32: 
> internal compiler error: in convert_move, at expr.c:299
>  v = (U16)( ((v<<8)|(v>>8)) & 0x );
>   ~~^~
>
> Signed-off-by: Raphael Kubo da Costa 

Friendly ping. Is there anything else I'd need to go do get this into
pyro?
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [oe-core][PATCH v3 2/2] runtime selftest: remove the case of hw breakpoint

[Randy MacLeod]

On 08/03/2018 05:48 AM, Hongzhi.Song wrote:

Hw breakpoint is based on hardware.


We need a better commit log explanation.

Was this test added in error originally?

Can you explain briefly whether their are any
qemu configurations would be able to efficiently
test the hardware breakpoint feature using these kernel
modules. For example, if you are running qemux86-64 with
the kvm-intel module in your host's kernel, then the tests
may work and even be efficient. Can you change the test
to detect if the hardware/qemu supports hardware breakpoint
and only run the test if it does?

../Randy



Signed-off-by: Hongzhi.Song 
---
  meta/lib/oeqa/runtime/cases/ksample.py | 16 
  1 file changed, 16 deletions(-)

diff --git a/meta/lib/oeqa/runtime/cases/ksample.py 
b/meta/lib/oeqa/runtime/cases/ksample.py
index 26fbe9d8cb..5e8699a2e2 100644
--- a/meta/lib/oeqa/runtime/cases/ksample.py
+++ b/meta/lib/oeqa/runtime/cases/ksample.py
@@ -161,22 +161,6 @@ class KSampleTest(KSample):
  # rmmod
  self.cmd_and_check("rmmod trace-printk")
  
-# hw breakpoint

-@OETestDepends(['ssh.SSHTest.test_ssh'])
-def test_hw_breakpoint_example(self):
-# check config
-self.check_config("CONFIG_KALLSYMS_ALL")
-# make sure if module exists
-self.check_module_exist("hw_breakpoint/", "data_breakpoint.ko")
-# modprobe
-self.cmd_and_check("modprobe data_breakpoint")
-# lsmod
-self.cmd_and_check("lsmod | grep data_breakpoint | cut -d\' \' -f1", 
"data_breakpoint")
-# check result
-self.cmd_and_check("cat /var/log/messages | grep sample_hbp_handler", 
"sample_hbp_handler")
-# rmmod
-self.cmd_and_check("rmmod data_breakpoint")
-
  @OETestDepends(['ssh.SSHTest.test_ssh'])
  def test_configfs_sample(self):
  # check config




--
# Randy MacLeod
# Wind River Linux
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] oeqa/selftest: check if rm_work is enabled

[Randy MacLeod]

On 08/09/2018 04:54 AM, Richard Purdie wrote:

On Wed, 2018-08-08 at 13:27 -0400, Randy MacLeod wrote:

On 07/28/2018 04:53 AM, Anuj Mittal wrote:

rm_work if enabled leads to some tests failing that rely on
artifacts being present. Check if rm_work.bbclass is included and
show an error and exit if it is.

Fixes [YOCTO #12694]

Signed-off-by: Anuj Mittal 
---
   meta/lib/oeqa/selftest/context.py | 4 
   1 file changed, 4 insertions(+)

diff --git a/meta/lib/oeqa/selftest/context.py
b/meta/lib/oeqa/selftest/context.py
index 16812ba96e..3a70f9e77b 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -187,6 +187,10 @@ class
OESelftestTestContextExecutor(OETestContextExecutor):
   self.tc.logger.error("You have buildhistory enabled
already and this isn't recommended for selftest, please disable it
first.")
   raise OEQAPreRun
   
+if "rm_work.bbclass" in self.tc.td["BBINCLUDED"]:

+self.tc.logger.error("You have rm_work enabled which
isn't recommended while running oe-selftest. Please disable it
before continuing.")
+raise OEQAPreRun
+
   if "PRSERV_HOST" in self.tc.td:
   self.tc.logger.error("Please unset PRSERV_HOST in
order to run oe-selftest")
   raise OEQAPreRun



Enabling rm_work should NOT be an error, and eventually it should not
trigger any warnings. OEQA should only test the results of a build
not how the build was done.


Uh, oh. I had written this email but then realized that
the parts of oeqa that require that rm_work be disabled
are testing the internals of the build so I swear I didn't
send the email! :)




I'm torn on this. My thoughts:

I'd really like the results of oe-selftest to be consistent and
deterministic. If we expect it to work with and without rm_work we
really start having to test both, otherwise you can comparatively
easily find odd failures with one and not the other. I've accidentally
left it enabled, then puzzled over odd test results for an age on too
many occasions :(.

I'm still of the view that rm_work is a horrible hack. Its one I've
bent over backwards to try and keep working over the years[1] since a
lot of people use it, that doesn't mean its any less of a hack though
as when you understand what it does and how it does it, it really is
horrible.
([1]I made 26 out of 47 changes to the class in the past 12 years)


Good to know. It is still useful so I guess it'll stay around until
storage is just incredibly cheap. Aren't we there yet:
   https://jcmit.net/flash2015.htm
If rm_work is such a hack, maybe we should schedule it to be removed.



If you step back and think about it, the use of rm_work precludes tests
from poking into WORKDIR or at the very least makes them have to care
about a lot of very specific task ordering of tasks which may or may
not be present. It doesn't seem unreasonable for selftests to actually
look in WORKDIR to ensure builds are doing the correct things.


I agree!




I reviewed some of the rm_work related defects:

1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12647
The warning was intentional based on Paul's comment.


There were two issues, one rm_work, the other warning piece isn't an
issue.


2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12655
The logs seem to only say that the error goes away when
rm_work is NOT enabled. I'd prefer to understand what it is
that's causing the failure and perhaps add to the packages
recipe so that the output passes the OEQA tests.


The issue is that the test looks for an intermediate file in
WORKDIR/sysroot-destdir. rm_work removes it before the test can find
it.


3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12694
Title was: oe-selftest should warn if rm_work is enabled
but the commit makes it an error.

Any disagreement about making it be a warning?

rm_work saves lots of storage space so I'm reluctant to make
OEQA require that it is NOT used.


We're not requiring OEQA not use it globally, we're requiring oe-
selftest not enable it. The other OEQA components (image, SDK, eSDK
tests) all are unaffected.


Yeah, once I realized that yesterday, I decided not to send
my email (which managed to escape somehow anyway).


As things stand today several tests break with it. I've no doubt we
could 'fix' them in some way to avoid the problems but do we require
any new test author to test with and without rm_work? Do we start
requiring the autobuilder test both combinations?

When the user runs oe-selftest, I really want them to get the same
results as our autobuilder so we're consistent everywhere. We've had
numerous problems in that regard, e.g. buildhistory enabled/not enabled
and I think its reasonable for the code to check for things which are
known to introduce inconsistency? We have made a lot of progress in
being much more consistent.


I see. That makes sense.

We'll continue with some builds using rm_work and the ones that 

[OE-core] [PATCH] man-pages: respect api-documentation

[Martin Jansa]

* let manpages.bbclass to enable manpages PACKAGECONFIG based on
  api-documentation DISTRO_FEATURES
  PACKAGECONFIG_append_class-target = " ${@bb.utils.contains('DISTRO_FEATURES', 
'api-documentation', 'manpages','', d)}"
* it's true that building man-pages without manpages being enabled
  doesn't make much sense, but it's included through couple
  packagegroups:
  meta/recipes-core/packagegroups/packagegroup-self-hosted.bb:man-pages \
  meta/recipes-extended/packagegroups/packagegroup-core-lsb.bb:man-pages \
  or in world even for people who might not be interested
  in man-pages

Signed-off-by: Martin Jansa 
---
 meta/recipes-extended/man-pages/man-pages_4.16.bb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/recipes-extended/man-pages/man-pages_4.16.bb 
b/meta/recipes-extended/man-pages/man-pages_4.16.bb
index 0fa124fe7c..a64587cf0b 100644
--- a/meta/recipes-extended/man-pages/man-pages_4.16.bb
+++ b/meta/recipes-extended/man-pages/man-pages_4.16.bb
@@ -14,8 +14,7 @@ inherit manpages
 
 MAN_PKG = "${PN}"
 
-# help manpages.bbclass deal with RDEPNDS and postinstall scripts
-PACKAGECONFIG = "manpages"
+PACKAGECONFIG ??= ""
 PACKAGECONFIG[manpages] = ""
 
 do_configure[noexec] = "1"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to upgrade Qt modules in daisy branch

[Martin Jansa]

For daisy you probably don't have support for PACKAGECONFIG_CONFARGS which
was added in Yocto 2.1 krogoth.

Read the meta-qt5 commit about PACKAGECONFIG_CONFARGS.

It's relatively easy to fix (I was also using older OE with newer Qt5), but
it's unsupported, so you're on your own and you need to figure out the
details yourself.

Cheers,

On Thu, Aug 9, 2018 at 12:10 PM Ankur Tyagi  wrote:

> Hi,
>
> Our product is based upon 'daisy' branch (I know it's pretty old !) and
> now we have a need to support BLE in it (typical business case).
>
> My inner-self says it's the right time to upgrade everything but from
> business unit point of view, best would be to upgrade "just enough" to get
> BLE working.
>
> So I upgraded Bluez4.101 to Bluez5.46 which went well but when I started
> upgrading QtConnectivity from v5.4.1 to v5.11.0 then things started falling
> apart.
>
> It seems from v5.8 onward, Qt adopted new configuration system which is
> causing configuration and compilation failures when building QtConnectivity.
> Following change in recipe doesn't work.
>
> -PACKAGECONFIG[bluez] = "CONFIG+=OE_BLUEZ_ENABLED,,bluez4"
> +PACKAGECONFIG[bluez] = "-feature-bluez,-no-feature-bluez,bluez5"
>
>
> Is there a way I can upgrade QtConnectivity to v5.11 while staying on
> daisy branch ? Or is it just not possible to do so ?
>
> Any help is appreciated.
>
> Thanks
> Ankur
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to upgrade Qt modules in daisy branch

[Alexander Kanavin]

2018-08-09 12:10 GMT+02:00 Ankur Tyagi :
> Our product is based upon 'daisy' branch (I know it's pretty old !) and now
> we have a need to support BLE in it (typical business case).
>
> My inner-self says it's the right time to upgrade everything but from
> business unit point of view, best would be to upgrade "just enough" to get
> BLE working.
>
> So I upgraded Bluez4.101 to Bluez5.46 which went well but when I started
> upgrading QtConnectivity from v5.4.1 to v5.11.0 then things started falling
> apart.
>
> It seems from v5.8 onward, Qt adopted new configuration system which is
> causing configuration and compilation failures when building QtConnectivity.
> Following change in recipe doesn't work.
>
> -PACKAGECONFIG[bluez] = "CONFIG+=OE_BLUEZ_ENABLED,,bluez4"
> +PACKAGECONFIG[bluez] = "-feature-bluez,-no-feature-bluez,bluez5"
>
>
> Is there a way I can upgrade QtConnectivity to v5.11 while staying on daisy
> branch ? Or is it just not possible to do so ?
>
> Any help is appreciated.

Staying close to upstream should be a continuous, ongoing activity,
and not a massive, disruptive effort when it's already too late :)

As for your specific need, it's basically not possible to tell what
you will need to fix and change. Recent version of Qt may need recent
versions of other things they depend on, and those recent other things
may pull in even more other recent things. They may even need a recent
C++ compiler which supports recent C++ standard, and when you bring in
that compiler, you will find that there's a ton of other things that
compiled before and need fixing now. The whole thing may quickly
unravel and become intractable - essentially you'll be upgrading the
whole stack while still calling it 'daisy'.

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] dropbear.inc: add dependency on virtual/crypt to fix build with glibc-2.28

[Martin Jansa]

configure tests crypt() existence with:

dnl We test for crypt() specially. On Linux (and others?) it resides in libcrypt
dnl but we don't want link all binaries to -lcrypt, just dropbear server.
dnl OS X doesn't need -lcrypt
AC_CHECK_FUNC(crypt, found_crypt_func=here)
AC_CHECK_LIB(crypt, crypt,
[
CRYPTLIB="-lcrypt"
found_crypt_func=here
])
AC_SUBST(CRYPTLIB)
if test "t$found_crypt_func" = there; then
AC_DEFINE(HAVE_CRYPT, 1, [crypt() function])
fi

but that silently fails with glibc-2.28 and a bit later do_compile fails with;
http://errors.yoctoproject.org/Errors/Details/185895/

../dropbear-2018.76/sysoptions.h:237:3: error: #error 
"DROPBEAR_SVR_PASSWORD_AUTH requires `crypt()'."
  #error "DROPBEAR_SVR_PASSWORD_AUTH requires `crypt()'."
   ^

Add dependency on virtual/crypt so that do_configure detects it correctly.

Signed-off-by: Martin Jansa 
---
 meta/recipes-core/dropbear/dropbear.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/dropbear/dropbear.inc 
b/meta/recipes-core/dropbear/dropbear.inc
index 2e2fbc15af..b634e0e53e 100644
--- a/meta/recipes-core/dropbear/dropbear.inc
+++ b/meta/recipes-core/dropbear/dropbear.inc
@@ -7,7 +7,7 @@ SECTION = "console/network"
 LICENSE = "MIT & BSD-3-Clause & BSD-2-Clause & PD"
 LIC_FILES_CHKSUM = "file://LICENSE;md5=a5ec40cafba26fc4396d0b550f824e01"
 
-DEPENDS = "zlib"
+DEPENDS = "zlib virtual/crypt"
 RPROVIDES_${PN} = "ssh sshd"
 
 DEPENDS += "${@bb.utils.contains('DISTRO_FEATURES', 'pam', 'libpam', '', d)}"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCHv2] perf: inherit manpages instead of adding man to RDEPENDS_${PN}-doc

[Martin Jansa]

Signed-off-by: Martin Jansa 
---
 meta/recipes-kernel/perf/perf.bb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index 2c6bbb42b9..ccd2c7d11b 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -39,7 +39,7 @@ do_configure[depends] += "virtual/kernel:do_shared_workdir"
 
 PROVIDES = "virtual/perf"
 
-inherit linux-kernel-base kernel-arch
+inherit linux-kernel-base kernel-arch manpages
 
 # needed for building the tools/perf Python bindings
 inherit ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'pythonnative', '', 
d)}
@@ -247,7 +247,6 @@ PACKAGE_ARCH = "${MACHINE_ARCH}"
 PACKAGES =+ "${PN}-archive ${PN}-tests ${PN}-perl ${PN}-python"
 
 RDEPENDS_${PN} += "elfutils bash"
-RDEPENDS_${PN}-doc += "man"
 RDEPENDS_${PN}-archive =+ "bash"
 RDEPENDS_${PN}-python =+ "bash python python-modules 
${@bb.utils.contains('PACKAGECONFIG', 'audit', 'audit-python', '', d)}"
 RDEPENDS_${PN}-perl =+ "bash perl perl-modules"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] ✗ patchtest: failure for "go(-dep): add INSANE_SKIP for ..." and 2 more

[Patchwork]

== Series Details ==

Series: "go(-dep): add INSANE_SKIP for ..." and 2 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/13444/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch[3/3] perf: inherit manpages instead of adding man to 
RDEPENDS_-doc
 Issue Patch is missing Signed-off-by [test_signed_off_by_presence] 
  Suggested fixSign off the patch (either manually or with "git commit 
--amend -s")



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines: 
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/3] glide: add INSANE_SKIP for textrel

[Martin Jansa]

* I'm not using glide, so I'm not going to fix it proplerly,
  it was just bothering me in world builds
* this is reproducible only with ptest in DISTRO_FEATUREs (for aarch64
  issue) and included security_flags.inc, more specifically with the PIE
  flags, so alternative work around is:
  SECURITY_CFLAGS_pn-glide = "${SECURITY_NOPIE_CFLAGS}"

Signed-off-by: Martin Jansa 
---
 meta/recipes-devtools/glide/glide_0.13.1.bb | 20 
 1 file changed, 20 insertions(+)

diff --git a/meta/recipes-devtools/glide/glide_0.13.1.bb 
b/meta/recipes-devtools/glide/glide_0.13.1.bb
index 1a5b145cf3..ba076d2bb7 100644
--- a/meta/recipes-devtools/glide/glide_0.13.1.bb
+++ b/meta/recipes-devtools/glide/glide_0.13.1.bb
@@ -13,3 +13,23 @@ RDEPENDS_${PN}-dev += "bash"
 RDEPENDS_${PN}-ptest += "bash"
 
 BBCLASSEXTEND = "native nativesdk"
+
+# for x86 ends with textrel in ${PN}
+# http://errors.yoctoproject.org/Errors/Details/185631/
+# ERROR: QA Issue: ELF binary 
'/work/i586-oe-linux/glide/0.13.1-r0/packages-split/glide/usr/bin/glide' has 
relocations in .text [textrel]
+INSANE_SKIP_${PN} += "textrel"
+
+# for aarch64 ends with textrel in ${PN}-ptest
+# http://errors.yoctoproject.org/Errors/Details/185633/
+# ERROR: QA Issue: ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/glide.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/dependency/dependency.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/repo/repo.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/mirrors/mirrors.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/cfg/cfg.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/godep/strip/strip.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/path/path.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/tree/tree.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/util/util.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/action/action.test'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/glide/0.13.1-r0/packages-split/glide-ptest/usr/lib/glide/ptest/github.com/Masterminds/glide/cache/cache.test'
 has relocations in .text [textrel]
+INSANE_SKIP_${PN}-ptest += "textrel"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/3] go(-dep): add INSANE_SKIP for textrel

[Martin Jansa]

* I'm not using go or go-dep, so I'm not going to fix it proplerly,
  it was just bothering me in world builds
* this is reproducible only with ptest in DISTRO_FEATUREs (for aarch64
  issue) and included security_flags.inc, more specifically with the PIE
  flags, so alternative work around is:
  SECURITY_CFLAGS_pn-go = "${SECURITY_NOPIE_CFLAGS}"
  SECURITY_CFLAGS_pn-go-dep = "${SECURITY_NOPIE_CFLAGS}"

Signed-off-by: Martin Jansa 
---
 meta/recipes-devtools/go/go-dep_0.4.1.bb | 10 ++
 meta/recipes-devtools/go/go_1.9.bb   |  7 +++
 2 files changed, 17 insertions(+)

diff --git a/meta/recipes-devtools/go/go-dep_0.4.1.bb 
b/meta/recipes-devtools/go/go-dep_0.4.1.bb
index d82e637205..10af1d660c 100644
--- a/meta/recipes-devtools/go/go-dep_0.4.1.bb
+++ b/meta/recipes-devtools/go/go-dep_0.4.1.bb
@@ -18,3 +18,13 @@ GO_INSTALL = "${GO_IMPORT}/cmd/dep"
 RDEPENDS_${PN}-dev += "bash"
 
 BBCLASSEXTEND = "native nativesdk"
+
+# for x86 ends with textrel in ${PN}
+# http://errors.yoctoproject.org/Errors/Details/185630/
+# ERROR: QA Issue: ELF binary 
'/work/i586-oe-linux/go-dep/0.4.1-r0/packages-split/go-dep/usr/bin/dep' has 
relocations in .text [textrel]
+INSANE_SKIP_${PN} += "textrel"
+
+# for aarch64 ends with textrel in ${PN}-ptest
+# http://errors.yoctoproject.org/Errors/Details/185632/
+# ERROR: QA Issue: ELF binary 
'/work/aarch64-oe-linux/go-dep/0.4.1-r0/packages-split/go-dep-ptest/usr/lib/go-dep/ptest/github.com/golang/dep/cmd/dep/dep.test'
 has relocations in .text [textrel]  
+INSANE_SKIP_${PN}-ptest += "textrel"
diff --git a/meta/recipes-devtools/go/go_1.9.bb 
b/meta/recipes-devtools/go/go_1.9.bb
index ec5a314e7d..7da7897cf5 100644
--- a/meta/recipes-devtools/go/go_1.9.bb
+++ b/meta/recipes-devtools/go/go_1.9.bb
@@ -15,3 +15,10 @@ do_compile() {
./make.bash
cd ${B}
 }
+
+# for aarch64 ends with textrel in ${PN}
+# http://errors.yoctoproject.org/Errors/Details/185634/
+# ERROR: QA Issue: ELF binary 
'/work/aarch64-oe-linux/go/1.9.7-r0/packages-split/go/usr/lib/go/bin/go' has 
relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/go/1.9.7-r0/packages-split/go/usr/lib/go/pkg/tool/linux_arm64/trace'
 has relocations in .text
+# ELF binary 
'/work/aarch64-oe-linux/go/1.9.7-r0/packages-split/go/usr/lib/go/pkg/tool/linux_arm64/pprof'
 has relocations in .text [textrel]
+INSANE_SKIP_${PN} += "textrel"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/3] perf: inherit manpages instead of adding man to RDEPENDS_-doc

[Martin Jansa]

---
 meta/recipes-kernel/perf/perf.bb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index 2c6bbb42b9..ccd2c7d11b 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -39,7 +39,7 @@ do_configure[depends] += "virtual/kernel:do_shared_workdir"
 
 PROVIDES = "virtual/perf"
 
-inherit linux-kernel-base kernel-arch
+inherit linux-kernel-base kernel-arch manpages
 
 # needed for building the tools/perf Python bindings
 inherit ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'pythonnative', '', 
d)}
@@ -247,7 +247,6 @@ PACKAGE_ARCH = "${MACHINE_ARCH}"
 PACKAGES =+ "${PN}-archive ${PN}-tests ${PN}-perl ${PN}-python"
 
 RDEPENDS_${PN} += "elfutils bash"
-RDEPENDS_${PN}-doc += "man"
 RDEPENDS_${PN}-archive =+ "bash"
 RDEPENDS_${PN}-python =+ "bash python python-modules 
${@bb.utils.contains('PACKAGECONFIG', 'audit', 'audit-python', '', d)}"
 RDEPENDS_${PN}-perl =+ "bash perl perl-modules"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] How to upgrade Qt modules in daisy branch

[Ankur Tyagi]

Hi,

Our product is based upon 'daisy' branch (I know it's pretty old !) and now
we have a need to support BLE in it (typical business case).

My inner-self says it's the right time to upgrade everything but from
business unit point of view, best would be to upgrade "just enough" to get
BLE working.

So I upgraded Bluez4.101 to Bluez5.46 which went well but when I started
upgrading QtConnectivity from v5.4.1 to v5.11.0 then things started falling
apart.

It seems from v5.8 onward, Qt adopted new configuration system which is
causing configuration and compilation failures when building QtConnectivity.
Following change in recipe doesn't work.

-PACKAGECONFIG[bluez] = "CONFIG+=OE_BLUEZ_ENABLED,,bluez4"
+PACKAGECONFIG[bluez] = "-feature-bluez,-no-feature-bluez,bluez5"


Is there a way I can upgrade QtConnectivity to v5.11 while staying on daisy
branch ? Or is it just not possible to do so ?

Any help is appreciated.

Thanks
Ankur
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/4] sstate: allow specifying indirect dependencies to exclude from sysroot

[Richard Purdie]

On Thu, 2018-08-09 at 10:00 +0100, André Draszik wrote:
> I should maybe explain my ultimate goal here:
> 
> meta-java is completely self-hosted, which is nice. But to be able to
> compile OpenJDK v8, various other Java compilers need to be compiled
> first.
> This of course means that all those different Java
> compilers/VMs/runtimes
> and tools needed to compile them end up in each Java recipe's
> sysroot.
> 
> While in theory this is fine, and all of them can be installed in
> parallel,
> as everything resides below subdirectories in /usr/lib/jvm/... and
> usr/bin/java normally is a symlink to the default chosen via update-
> alternatives, this doesn't work in yocto for -native recipes.
> 
> In particular this is annoying, because 'usr/bin/java' in the recipe
> sysroot
> is a *very* old version of java, with no possibility of control.
> 
> This has caused me a bit of headache, and if it was possible to avoid
> all
> the recursive OpenJDK-8 build time dependencies then:
> * usr/bin/java could disappear (for -native)
> * tools inherited from the OpenJDK-8 build can be avoided during
> other
>   recipe builds
> 
> In other words, as a follow-up to this series here I have a local
> meta-java
> change to meta-java's layer.conf:
> 
> SSTATE_EXCLUDEDEPS_SYSROOT += "\
> openjdk-8-native->(ant-native|attr-native|coreutils-
> native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-
> native|unzip-native) \
> "
> 
> This really is a similar situation to *-initial treatment in
> sstate.bbclass,
> but for Java / OpenJDK.
> 
> 
> So rather than having to add knowledge about Java / OpenJDK into
> sstate.bbclass, I need a way to inject that instead.
> 
> 
> Given the above, would you be happy to consider a series where the
> infrastructure is put in place, but without the flex-native and
> bison-native 
> changes? I.e. patches 1 through 3, but not patch 4 of this series?
> (With
> your previous review comments addressed, of course).
> 
> 
> > Basically binutils-cross is bust with errors like:
> > 
> > > x86_64-poky-linux-ar: error while loading shared libraries:
> > > libfl.so.2:
> > > cannot open shared object file: No such file or directory
> > > make[5]: *** [libaudio_resampler_sse.la] Error 127
> > 
> > There are a ton of other failures, I've not waded through them but
> > I'm
> > guessing there are similar issues.
> 
> I am not sure why I didn't see these in my builds before submitting,
> but now
> I do, too.
> 
> It looks like the cross binutils-native (or more specifically cross
> -ar)
> dynamically links against libflex-native. So the last patch in this
> series
> is of course responsible for that.
> 
> Comparing to my debian, ar isn't dynamically linked against libflex,
> I
> wonder why yocto does it, or why debian doesn't.
> 
> In any case my builds succeed now by dropping the flex-native from
> patch 4
> of this series. Alternative would be to switch to static linking for
> libflex. But see above.

The above is good info thanks. I'm not against the series, I originally
thought you wanted to optimise out flex-native and the test results
showed it didn't work so I was waiting on your response to that as the
series didn't build as it stood. I'd mentioned there were problems but
couldn't remember the specifics, you pushed me to "prove" it, I gave
you the errors.

The flex-native change doesn't work due to the binutils dependency and
may not work in general for anything linking to libfl. It would be
interesting to investigate why we have that linkage and if we can avoid
it. (Note that bison-native may work as it doesn't use a library iirc
so we could split that patch into two?).

I continue to be worried about the regular expression usage. I believe
modern python does some caching behind the scenes which helps but tight
loops with regexps in have caused us pain in the past, particularly
recompiling the expression each time and I worry about new API which
ties us to this. Is it possible to use glob?

I'm not against the patch series, we just need to drop the flex-native
change (maybe adding a special case for binutils if we can figure out
the linking) and figure out if we can use a simpler mechanism than
regexps or at least stop me worrying about performance. I'd need to
spend more time than I have right now to figure out how to properly
benchmark that function but as I said we have been bitten before :(.

Cheers,

Richard






-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/4] sstate: allow specifying indirect dependencies to exclude from sysroot

[André Draszik]

On Sun, 2018-06-17 at 14:13 +0100, Richard Purdie wrote:
> On Tue, 2018-05-22 at 13:25 +0100, André Draszik wrote:
> > From: André Draszik 
> > 
> > Currently, a dependency on any -native recipe will pull in
> > all dependencies of that -native recipe in the recipe
> > sysroot. This behaviour might not always be wanted, e.g.
> > when that -native recipe depends on build-tools that are
> > not relevant for the current recipe.
> > 
> > This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable,
> > which will be evaluated for such recursive dependencies to
> > be excluded. The idea is similar to
> >http://lists.openembedded.org/pipermail/openembedded-core/2018-Jan
> > uary/146324.html
> > except that the list is not hard-coded anymore.
> > 
> > SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular
> > expressions of recipe and dependency to ignore, e.g. in
> > the above flex-native / bison-native use-case, one would
> > specify
> > 
> > SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native"
> > 
> > in layer.conf.
> > 
> > The existing special handling of "-initial" as well as
> > "base-passwd" and "shadow-sysroot" could also be
> > streamlined:
> > 
> > SSTATE_EXCLUDEDEPS_SYSROOT += "\
> > .*->.*-initial.* \
> > .*(base-passwd|shadow-sysroot)->.* \
> > "
> > 
> > Another anticipated user is meta-java, where certain newer
> > JDKs can only be bootstrapped (built) using older JDKs,
> > but it doesn't make much sense to copy all those older
> > JDKs and their own build tools (ant, etc.) into the
> > sysroot of recipes wanting to be built using the newer JDK
> > (only), e.g.:
> > 
> > SSTATE_EXCLUDEDEPS_SYSROOT += "\
> > openjdk-8-native->(ant-native|attr-native|coreutils-
> > native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-
> > native|unzip-native) \
> > "
> > 
> > Signed-off-by: André Draszik 
> > ---
> >  meta/classes/sstate.bbclass | 13 +
> >  1 file changed, 13 insertions(+)
> > 
> 
> I have some reservations about this series as I've previously mentioned
> but I had a try at testing it on the autobuilder:
> 
> https://autobuilder.yocto.io/builders/build-appliance/builds/1059/steps/Bu
> ildImages_1/logs/stdio

Thank you Richard.

I should maybe explain my ultimate goal here:

meta-java is completely self-hosted, which is nice. But to be able to
compile OpenJDK v8, various other Java compilers need to be compiled first.
This of course means that all those different Java compilers/VMs/runtimes
and tools needed to compile them end up in each Java recipe's sysroot.

While in theory this is fine, and all of them can be installed in parallel,
as everything resides below subdirectories in /usr/lib/jvm/... and
usr/bin/java normally is a symlink to the default chosen via update-
alternatives, this doesn't work in yocto for -native recipes.

In particular this is annoying, because 'usr/bin/java' in the recipe sysroot
is a *very* old version of java, with no possibility of control.

This has caused me a bit of headache, and if it was possible to avoid all
the recursive OpenJDK-8 build time dependencies then:
* usr/bin/java could disappear (for -native)
* tools inherited from the OpenJDK-8 build can be avoided during other
  recipe builds

In other words, as a follow-up to this series here I have a local meta-java
change to meta-java's layer.conf:

SSTATE_EXCLUDEDEPS_SYSROOT += "\

openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native)
 \
"

This really is a similar situation to *-initial treatment in sstate.bbclass,
but for Java / OpenJDK.


So rather than having to add knowledge about Java / OpenJDK into
sstate.bbclass, I need a way to inject that instead.


Given the above, would you be happy to consider a series where the
infrastructure is put in place, but without the flex-native and bison-native 
changes? I.e. patches 1 through 3, but not patch 4 of this series? (With
your previous review comments addressed, of course).


> Basically binutils-cross is bust with errors like:
> 
> > x86_64-poky-linux-ar: error while loading shared libraries: libfl.so.2:
> > cannot open shared object file: No such file or directory
> > make[5]: *** [libaudio_resampler_sse.la] Error 127
> 
> There are a ton of other failures, I've not waded through them but I'm
> guessing there are similar issues.

I am not sure why I didn't see these in my builds before submitting, but now
I do, too.

It looks like the cross binutils-native (or more specifically cross -ar)
dynamically links against libflex-native. So the last patch in this series
is of course responsible for that.

Comparing to my debian, ar isn't dynamically linked against libflex, I
wonder why yocto does it, or why debian doesn't.

In any case my builds succeed now by dropping the flex-native from patch 4
of this series. Alternative would be to switch to static linking for
libflex. But see above.


Cheers,
Andre'

Re: [OE-core] [PATCH] oeqa/selftest: check if rm_work is enabled

[Richard Purdie]

On Wed, 2018-08-08 at 13:27 -0400, Randy MacLeod wrote:
> On 07/28/2018 04:53 AM, Anuj Mittal wrote:
> > rm_work if enabled leads to some tests failing that rely on
> > artifacts being present. Check if rm_work.bbclass is included and
> > show an error and exit if it is.
> > 
> > Fixes [YOCTO #12694]
> > 
> > Signed-off-by: Anuj Mittal 
> > ---
> >   meta/lib/oeqa/selftest/context.py | 4 
> >   1 file changed, 4 insertions(+)
> > 
> > diff --git a/meta/lib/oeqa/selftest/context.py
> > b/meta/lib/oeqa/selftest/context.py
> > index 16812ba96e..3a70f9e77b 100644
> > --- a/meta/lib/oeqa/selftest/context.py
> > +++ b/meta/lib/oeqa/selftest/context.py
> > @@ -187,6 +187,10 @@ class
> > OESelftestTestContextExecutor(OETestContextExecutor):
> >   self.tc.logger.error("You have buildhistory enabled
> > already and this isn't recommended for selftest, please disable it
> > first.")
> >   raise OEQAPreRun
> >   
> > +if "rm_work.bbclass" in self.tc.td["BBINCLUDED"]:
> > +self.tc.logger.error("You have rm_work enabled which
> > isn't recommended while running oe-selftest. Please disable it
> > before continuing.")
> > +raise OEQAPreRun
> > +
> >   if "PRSERV_HOST" in self.tc.td:
> >   self.tc.logger.error("Please unset PRSERV_HOST in
> > order to run oe-selftest")
> >   raise OEQAPreRun
> > 
> 
> Enabling rm_work should NOT be an error, and eventually it should not
> trigger any warnings. OEQA should only test the results of a build
> not how the build was done.

I'm torn on this. My thoughts:

I'd really like the results of oe-selftest to be consistent and
deterministic. If we expect it to work with and without rm_work we
really start having to test both, otherwise you can comparatively
easily find odd failures with one and not the other. I've accidentally
left it enabled, then puzzled over odd test results for an age on too
many occasions :(.

I'm still of the view that rm_work is a horrible hack. Its one I've
bent over backwards to try and keep working over the years[1] since a
lot of people use it, that doesn't mean its any less of a hack though
as when you understand what it does and how it does it, it really is
horrible. 
([1]I made 26 out of 47 changes to the class in the past 12 years)

If you step back and think about it, the use of rm_work precludes tests
from poking into WORKDIR or at the very least makes them have to care
about a lot of very specific task ordering of tasks which may or may
not be present. It doesn't seem unreasonable for selftests to actually
look in WORKDIR to ensure builds are doing the correct things.

> I reviewed some of the rm_work related defects:
> 
> 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12647
>The warning was intentional based on Paul's comment.

There were two issues, one rm_work, the other warning piece isn't an
issue.

> 2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12655
>The logs seem to only say that the error goes away when
>rm_work is NOT enabled. I'd prefer to understand what it is
>that's causing the failure and perhaps add to the packages
>recipe so that the output passes the OEQA tests.

The issue is that the test looks for an intermediate file in
WORKDIR/sysroot-destdir. rm_work removes it before the test can find
it.

> 3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12694
>Title was: oe-selftest should warn if rm_work is enabled
>but the commit makes it an error.
> 
> Any disagreement about making it be a warning?
> 
> rm_work saves lots of storage space so I'm reluctant to make
> OEQA require that it is NOT used.

We're not requiring OEQA not use it globally, we're requiring oe-
selftest not enable it. The other OEQA components (image, SDK, eSDK
tests) all are unaffected.

As things stand today several tests break with it. I've no doubt we
could 'fix' them in some way to avoid the problems but do we require
any new test author to test with and without rm_work? Do we start
requiring the autobuilder test both combinations?

When the user runs oe-selftest, I really want them to get the same
results as our autobuilder so we're consistent everywhere. We've had
numerous problems in that regard, e.g. buildhistory enabled/not enabled
and I think its reasonable for the code to check for things which are
known to introduce inconsistency? We have made a lot of progress in
being much more consistent.

Cheers,

Richard




-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] libxml2: Fix CVE-2018-14404

[Andrej Valek]

Fix nullptr deref with XPath logic ops

If the XPath stack is corrupted, for example by a misbehaving extension
function, the "and" and "or" XPath operators could dereference NULL
pointers. Check that the XPath stack isn't empty and optimize the
logic operators slightly.

CVE: CVE-2018-14404
Signed-off-by: Andrej Valek 
---
 .../libxml/libxml2/fix-CVE-2018-14404.patch| 45 ++
 meta/recipes-core/libxml/libxml2_2.9.8.bb  |  1 +
 2 files changed, 46 insertions(+)
 create mode 100644 meta/recipes-core/libxml/libxml2/fix-CVE-2018-14404.patch

diff --git a/meta/recipes-core/libxml/libxml2/fix-CVE-2018-14404.patch 
b/meta/recipes-core/libxml/libxml2/fix-CVE-2018-14404.patch
new file mode 100644
index 00..21668e25ad
--- /dev/null
+++ b/meta/recipes-core/libxml/libxml2/fix-CVE-2018-14404.patch
@@ -0,0 +1,45 @@
+libxml2-2.9.8: Fix CVE-2018-14404
+
+[No upstream tracking] -- https://gitlab.gnome.org/GNOME/libxml2/issues/5
+ -- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901817
+ -- https://bugzilla.redhat.com/show_bug.cgi?id=1595985
+
+xpath: Fix nullptr deref with XPath logic ops
+
+If the XPath stack is corrupted, for example by a misbehaving extension
+function, the "and" and "or" XPath operators could dereference NULL
+pointers. Check that the XPath stack isn't empty and optimize the
+logic operators slightly.
+
+Upstream-Status: Backport 
[https://gitlab.gnome.org/GNOME/libxml2/commit/a436374994c47b12d5de1b8b1d191a098fa23594]
+CVE: CVE-2018-14404
+Signed-off-by: Andrej Valek 
+
+diff --git a/xpath.c b/xpath.c
+index f440696..75cac5c 100644
+--- a/xpath.c
 b/xpath.c
+@@ -13297,9 +13297,8 @@ xmlXPathCompOpEval(xmlXPathParserContextPtr ctxt, 
xmlXPathStepOpPtr op)
+   return(0);
+   }
+ xmlXPathBooleanFunction(ctxt, 1);
+-arg1 = valuePop(ctxt);
+-arg1->boolval &= arg2->boolval;
+-valuePush(ctxt, arg1);
++  if (ctxt->value != NULL)
++  ctxt->value->boolval &= arg2->boolval;
+   xmlXPathReleaseObject(ctxt->context, arg2);
+ return (total);
+ case XPATH_OP_OR:
+@@ -13323,9 +13322,8 @@ xmlXPathCompOpEval(xmlXPathParserContextPtr ctxt, 
xmlXPathStepOpPtr op)
+   return(0);
+   }
+ xmlXPathBooleanFunction(ctxt, 1);
+-arg1 = valuePop(ctxt);
+-arg1->boolval |= arg2->boolval;
+-valuePush(ctxt, arg1);
++  if (ctxt->value != NULL)
++  ctxt->value->boolval |= arg2->boolval;
+   xmlXPathReleaseObject(ctxt->context, arg2);
+ return (total);
+ case XPATH_OP_EQUAL:
diff --git a/meta/recipes-core/libxml/libxml2_2.9.8.bb 
b/meta/recipes-core/libxml/libxml2_2.9.8.bb
index 27a648e97a..4ebd2ef383 100644
--- a/meta/recipes-core/libxml/libxml2_2.9.8.bb
+++ b/meta/recipes-core/libxml/libxml2_2.9.8.bb
@@ -21,6 +21,7 @@ SRC_URI = 
"http://www.xmlsoft.org/sources/libxml2-${PV}.tar.gz;name=libtar \

file://0001-Make-ptest-run-the-python-tests-if-python-is-enabled.patch \
file://fix-execution-of-ptests.patch \
file://fix-CVE-2017-8872.patch \
+   file://fix-CVE-2018-14404.patch \
"
 
 SRC_URI[libtar.md5sum] = "b786e353e2aa1b872d70d5d1ca0c740d"
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] cmake: fix compiling some C++ projects with Yocto SDK and GCC

[Urs Fässler]

Setting CMAKE_SYSROOT in the toolchain file allows CMake to correctly
remove user-provided system include directories pointing to
/usr/include. The mentioned projects failed with "stdlib.h:
No such file or directory #include_next ".

Signed-off-by: Urs Fässler 
Signed-off-by: Raphael Freudiger 
---
 meta/recipes-devtools/cmake/cmake/OEToolchainConfig.cmake | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/cmake/cmake/OEToolchainConfig.cmake 
b/meta/recipes-devtools/cmake/cmake/OEToolchainConfig.cmake
index d52c29e5a1..8a0fb4cb12 100644
--- a/meta/recipes-devtools/cmake/cmake/OEToolchainConfig.cmake
+++ b/meta/recipes-devtools/cmake/cmake/OEToolchainConfig.cmake
@@ -3,6 +3,7 @@ set( CMAKE_C_FLAGS $ENV{CFLAGS} CACHE STRING "" FORCE )
 set( CMAKE_CXX_FLAGS $ENV{CXXFLAGS}  CACHE STRING "" FORCE )
 set( CMAKE_ASM_FLAGS ${CMAKE_C_FLAGS} CACHE STRING "" FORCE )
 set( CMAKE_LDFLAGS_FLAGS ${CMAKE_CXX_FLAGS} CACHE STRING "" FORCE )
+set( CMAKE_SYSROOT $ENV{OECORE_TARGET_SYSROOT} )
 
 set( CMAKE_FIND_ROOT_PATH $ENV{OECORE_TARGET_SYSROOT} 
$ENV{OECORE_NATIVE_SYSROOT} )
 set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER )
-- 
2.18.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core