Re: [OE-core] RFC: Upgrading rt-tests requires numactl

[Randy Witt]

On 9/23/20 11:00 PM, Khem Raj wrote:

On Wed, Sep 23, 2020 at 2:40 PM Ross Burton  wrote:


On Wed, 23 Sep 2020 at 18:23, Randy Witt  wrote:

Another issue is that ARM is removed from COMPATIBLE_HOST in
http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-support/numactl/numactl_git.bb?h=master#n27.
This may be for legacy reasons, but libnuma can be installed on most
distributions running on an ARM host. Regardless, it would need to be changed to
allow for rt-tests on ARM.


Yeah, that's probably very old.

$ lscpu
Architecture:aarch64
CPU op-mode(s):  64-bit
Byte Order:  Little Endian
CPU(s):  256
On-line CPU(s) list: 0-255
Thread(s) per core:  4
Core(s) per socket:  32
Socket(s):   2
NUMA node(s):2

The recipe has done that since it was added, so I'd call this
historical and just delete it.


question is not aarch64 but arm (32bit) and I think answer is still no for arm.



I compiled numactl on arm, and v1.8 of rt-tests. While the ptests for numactl 
fail, due to there being no actual numa support, the libraries and headers still 
work.


cyclictest from rt-tests seems to still set affinity appropriately and work. 
Should we make the ptests aware of the expected results when numa isn't 
supported? For example:


 root@qemuarm:/# /usr/lib/numactl/ptest/test/distance
 no numa support in kernel

I also tried v1.9 of rt-tests, but it fails due to the following 
https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/oslat/oslat.c?h=v1.9#n72. 
I'm going to email the maintainer about that, since there is no way to disable 
any of the tests, outside of manually specifying each one. And it will break any 
architecture that isn't x86_64, i386, or PPC64.



Ross






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142879): 
https://lists.openembedded.org/g/openembedded-core/message/142879
Mute This Topic: https://lists.openembedded.org/mt/77040068/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] RFC: Upgrading rt-tests requires numactl

[Randy Witt]

While looking into upgrading rt-tests to the latest version, on IRC, John 
Kacur(the maintainer of rt-tests), recommended "that they use 
unstable/devel/latest as the maintained stable branch".


I updated the recipes in 
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rewitt/rt-tests-upgrade=551b25918d06c903fff22d76d5683b548c4f2e6f. 
However, I know this won't work as is, due to the issues below.


The latest version of rt-tests requires the numa libraries for compilation via 
this patch https://marc.info/?l=linux-rt-users=158335896221530=2. rt-tests 
exists in oe-core, but numactl which provides libnuma/numa.h exists in 
meta-openembedded.


Another issue is that ARM is removed from COMPATIBLE_HOST in 
http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-support/numactl/numactl_git.bb?h=master#n27. 
This may be for legacy reasons, but libnuma can be installed on most 
distributions running on an ARM host. Regardless, it would need to be changed to 
allow for rt-tests on ARM.


I'm therefore trying to determine how to proceed to upgrade rt-tests. These are 
the options I thought of, which are most likely not exhaustive.


1. Bring numactl into oe-core
2. move rt-tests out of oe-core
3. TMy to patch out the numa requirement and carry it forever(doesn't appear to 
be a small task)

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#142741): 
https://lists.openembedded.org/g/openembedded-core/message/142741
Mute This Topic: https://lists.openembedded.org/mt/77040068/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Linux v5.8 modules, exec format error

[Randy Witt]

On 8/29/20 11:41 AM, Bruce Ashfield wrote:

On Fri, Aug 28, 2020 at 10:28 PM Bruce Ashfield via
lists.openembedded.org
 wrote:


On Fri, Aug 28, 2020 at 7:15 PM Jack Mitchell  wrote:


Quick update, I just did an armv7 build with exactly the same codebase
and everything worked fine. Do you have an aarch64 build could test and
confirm working?


qemuarm64 was working fine here with -rc1. I've started a new build,
but it'll be several hours before I know more (so sometime saturday).



I saw this on linux-modules today, 
https://lore.kernel.org/linux-modules/20200829100334.gk1362...@hirez.programming.kicks-ass.net/T/#t 
which references
https://lore.kernel.org/lkml/20200808101222.51030...@coco.lan/ saying it is a 
bug in binutils.


I haven't looked at it anymore other than seeing this email and that issue are 
both exec format errors on arm. I have to leave and can't investigate anymore, 
but figured this might be useful. If not feel free to ignore, and sorry for the 
noise.


still working here:

qemuarm64 login: root
root@qemuarm64:~# uname -a
Linux qemuarm64 5.9.0-rc2-yoctodev-standard #1 SMP PREEMPT Sat Aug 29
14:26:30 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
root@qemuarm64:~# lsmod
Module  Size  Used by
sch_fq_codel   20480  1
openvswitch   155648  0
nsh16384  1 openvswitch
nf_conncount   20480  1 openvswitch
nf_nat 40960  1 openvswitch
nf_conntrack  110592  3 nf_nat,openvswitch,nf_conncount
nf_defrag_ipv6 20480  2 nf_conntrack,openvswitch
nf_defrag_ipv4 16384  1 nf_conntrack
root@qemuarm64:~#

Bruce


Cheers,

Bruce



Regards,
Jack.

On 28/08/2020 22:35, Jack Mitchell wrote:

Hi Bruce,

All built in-tree, the same recipe builds an armv7h kernel so I'll try a
build for that and see if it's something aarch64 specific. All the
modules are failing to load so it's not something specific to g_ether.
Please see kernel recipe below for reference.

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"

inherit kernel

S = "${WORKDIR}/git"

SRCREV = "redacted"
KBRANCH = "v5.9-rc2"

LINUX_VERSION ?= "${KBRANCH}-g${SRCREV}"
PV = "${LINUX_VERSION}"

SRC_URI = " \

git://g...@github.com/redacted/linux.git;name=kernel;branch=${KBRANCH};protocol=ssh
\
"

do_configure_prepend() {
 if [ -n "${KBUILD_DEFCONFIG}" ] && [ -f
"${S}/arch/${ARCH}/configs/${KBUILD_DEFCONFIG}" ]; then
 oe_runmake_call -C ${S} CC="${KERNEL_CC}"
LD="${KERNEL_LD}" O=${B} ${KBUILD_DEFCONFIG}
 fi
}

Cheers,
Jack.

On 28/08/2020 21:55, Bruce Ashfield wrote:

On Fri, Aug 28, 2020 at 4:20 PM Jack Mitchell  wrote:


Having just upgraded my mainline kernel recipe to a v5.8/v5.9-rc2 kernel
from v5.5.8 I've found that modules have somehow broken. I've flicked
between the two and confirmed that the old kernel build works, and the
5.8/5.9 build doesn't. I haven't changed anything bar the git commit
hash. It's a very simple kernel recipe basically just inheriting the
kernel bbclass and setting SRCREV. Running on current tip of master.

I assume it's something symver related but wanted to ask if anybody
knows anything before I dig too deep.


I can say that it is working for me on 5.8 and 5.9-rcX on the reference kernels.

qemux86-64 login: root
root@qemux86-64:~# uname -a
Linux qemux86-64 5.8.4-yocto-standard #1 SMP PREEMPT Wed Aug 26
16:07:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@qemux86-64:~# lsmod
Module  Size  Used by
parport_pc 24576
parport28672  1 parport_pc
ata_piix   36864  0
floppy 77824  0
sch_fq_codel   20480  1

my 5.9-rc is rebuilding right now, so I can double check it over the weekend.

Not super useful, but there shouldn't be anything fundamentally
broken, since we've been following along with the latest as usual.

Is your g_ether built in-tree, or out of tree ?

Bruce




Cheers,
Jack.

root@rk3399:~# uname  -a
Linux rk3399 5.9.0-rc2 #1 SMP PREEMPT Fri Aug 28 18:47:44 UTC 2020
aarch64 GNU/Linux

root@rk3399:~# modprobe g_ether
modprobe: ERROR: could not insert 'g_ether': Exec format error

root@rk3399:~# modinfo
/lib/modules/5.9.0-rc2/kernel/drivers/usb/gadget/legacy/g_ether.ko
filename:
/lib/modules/5.9.0-rc2/kernel/drivers/usb/gadget/legacy/g_ether.ko
license:GPL
author: David Brownell, Benedikt Spanger
description:RNDIS/Ethernet Gadget
depends:libcomposite,u_ether,usb_f_rndis
intree: Y
name:   g_ether
vermagic:   5.9.0-rc2 SMP preempt mod_unload aarch64
parm:   idVendor:USB Vendor ID (ushort)
parm:   idProduct:USB Product ID (ushort)
parm:   bcdDevice:USB Device version (BCD) (ushort)
parm:   iSerialNumber:SerialNumber string (charp)
parm:   iManufacturer:USB Manufacturer string (charp)
parm:   iProduct:USB Product string (charp)
parm:   qmult:queue length multiplier at high/super 

Re: [OE-core] [PATCH v2] wic/bootimg-efi: IMAGE_EFI_BOOT_FILES added to separate bootimg-efi and bootimg-partition

[Randy Witt]

On 8/26/20 1:13 AM, Jamaluddin, Khairul Rohaizzat wrote:

From: Khairul Rohaizzat Jamaluddin 

Due to recent changes in bootimg-efi to include IMAGE_BOOT_FILES,
when both bootimg-partition and bootimg-efi occur in a single .wks
and IMAGE_BOOT_FILES are defined, files listed in IMAGE_BOOT_FILES
will be duplicated in both partition.
Since IMAGE_BOOT_FILES are crucial for bootimg-partition, but
optional for bootimg-efi, hence allowing bootimg-efi to have the option
to ignore it.

Added a new variable, IMAGE_EFI_BOOT_FILES, to handle this
issue. Its basic usage is the same as IMAGE_BOOT_FILES.
Usage example:
 IMAGE_EFI_BOOT_FILES = "u-boot.img uImage;kernel"
 IMAGE_EFI_BOOT_FILES = "u-boot.${UBOOT_SUFFIX} ${KERNEL_IMAGETYPE}"

This commit is also squashed with the updated testcase to cover for this change.

[YOCTO #14011]

Signed-off-by: Khairul Rohaizzat Jamaluddin 

---
  meta/classes/image_types_wic.bbclass  |  2 +-
  meta/lib/oeqa/selftest/cases/wic.py   | 10 +-
  scripts/lib/wic/plugins/source/bootimg-efi.py |  8 
  3 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass 
b/meta/classes/image_types_wic.bbclass
index 7b1db50..def44bb 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -1,7 +1,7 @@
  # The WICVARS variable is used to define list of bitbake variables used in 
wic code
  # variables from this list is written to .env file
  WICVARS ?= "\
-   BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME 
IMAGE_BOOT_FILES \
+   BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME 
IMAGE_EFI_BOOT_FILES IMAGE_BOOT_FILES \
 IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE 
ISODIR RECIPE_SYSROOT_NATIVE \
 ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS \
 KERNEL_IMAGETYPE MACHINE INITRAMFS_IMAGE INITRAMFS_IMAGE_BUNDLE 
INITRAMFS_LINK_NAME APPEND"
diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index e6b23c6..1149ae0 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -235,6 +235,14 @@ class Wic(WicTestCase):
  runCmd(cmd)
  self.assertEqual(1, len(glob(self.resultdir + 
"systemd-bootdisk-*direct")))
  
+def test_efi_bootpart(self):

+"""Test creation of efi-bootpart image"""
+cmd = "wic create efi-bootpart -e core-image-minimal -o %s" % 
self.resultdir
+kimgtype = get_bb_var('KERNEL_IMAGETYPE', 'core-image-minimal')
+self.write_config('IMAGE_EFI_BOOT_FILES = "%s"\n' % kimgtype)
+runCmd(cmd)
+self.assertEqual(1, len(glob(self.resultdir + "efi-bootpart-*direct")))
+


The test fails because "efi-bootpart" is not a wks file like 
"sdimage-bootpart"(which this appears to be copied from). There is an existing 
wks file ./scripts/lib/wic/canned-wks/mkefidisk.wks which uses the "bootimg-efi" 
plugin and there is already a test, test_mkefidisk(), for that wks file.


This new test, test_efi_bootpart(), doesn't actually verify the files in 
IMAGE_EFI_BOOT_FILES exist in the partition created, it only checks that the wic 
command ran successfully. So there isn't really any utility in this test over 
what already exists in test_mkefidisk().


Also, the file being added in IMAGE_EFI_BOOT_FILES in this test ends up being 
"bzImage", which is already added by default when using bootimage-efi.py.


For this test to actually test the IMAGE_EFI_BOOT_FILES functionality, it would 
need to do something similar to test_include_path() which actually verifies that 
files exist in the partition created. In this case it would need to verify that 
the files in IMAGE_EFI_BOOT_FILES exist in the partition. If we assume "wic ls" 
works, it could probably be used instead of debugfs.



  def test_sdimage_bootpart(self):
  """Test creation of sdimage-bootpart image"""
  cmd = "wic create sdimage-bootpart -e core-image-minimal -o %s" % 
self.resultdir
@@ -689,7 +697,7 @@ class Wic2(WicTestCase):
  wicvars = wicvars.difference(('DEPLOY_DIR_IMAGE', 'IMAGE_BOOT_FILES',
'INITRD', 'INITRD_LIVE', 
'ISODIR','INITRAMFS_IMAGE',
'INITRAMFS_IMAGE_BUNDLE', 
'INITRAMFS_LINK_NAME',
-  'APPEND'))
+  'APPEND', 'IMAGE_EFI_BOOT_FILES'))
  with open(path) as envfile:
  content = dict(line.split("=", 1) for line in envfile)
  # test if variables used by wic present in the .env file
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py 
b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 14c1723..cdc7254 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -212,8 +212,8 @@ class 

[OE-core] New changes to bootimg-efi.py break some wic installer image methods

[Randy Witt]

Hi,

The changes in 
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=801f07fc00d43d107cb6ae671dbbb57ee91cc6be 
prevent creating an installer image using wic described here: 
http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel/tree/README#n201.


This is because the IMAGE_BOOT_FILES intended for bootimg-partition, now 
override the efi files added by efi-bootimg.


The simplest solution would be to use a different variable for the bootimg-efi 
plugin such as IMAGE_EFI_BOOT_FILES. IMAGE_BOOT_FILES and IMAGE_EFI_BOOT_FILES 
could always be set to the same values if desired.


I'm not sure this is the "best" solution, but it would allow for both methods to 
coexist with a relatively small change.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#141781): 
https://lists.openembedded.org/g/openembedded-core/message/141781
Mute This Topic: https://lists.openembedded.org/mt/76391516/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] openssh: Atomically generate host keys

[Randy Witt]

On 05/23/2017 08:29 AM, Joshua Watt wrote:

On Tue, May 23, 2017 at 9:37 AM, Burton, Ross  wrote:


On 7 May 2017 at 02:33, Joshua Watt  wrote:


+if [ ! -f "$NAME" ]; then
+echo "  generating ssh $TYPE key..."
+ssh-keygen -q -f "${NAME}.tmp" -N '' -t $TYPE
+
+# Sync to ensure data is written to temp file before renaming
+sync
+
+# Move (Atomically rename) files
+# Rename the .pub file first, since the check that triggers a
+# key generation is based on the private file.
+mv -f "${NAME}.tmp.pub" "${NAME}.pub"
+sync
+
+mv -f "${NAME}.tmp" "${NAME}"
+sync
+fi



All of these syncs seem quite enthusiastic, are they really needed?  Writing
the file to a temporary name and then mving it to the real name should
result in either no file or a complete file in the event of power loss,
surely?


My understanding (and observation) of most journal file systems is
that only metadata (i.e. directory entries and such) are journaled in
typical usage. The first sync is necessary in this case to ensure that
the actual file data gets put on the disk before we rename the files,
otherwise in the event of interruption journaled rename might get
replayed but have garbage data. The second one is more of a "force
operation order" sync to make sure the public file is written before
the private one, as a reordering would cause problems. The last sync
is the most optional, but I've seen it take minutes for disk to sync
contents if no one calls "sync", so it is very possible that all our
work of regenerating keys would be for naught if power is interrupted
in the meantime.

I think some of these syncs can be removed. Namely, the first and
third one. The second one needs to be there, but can server double
duty of syncing data to disk and enforcing the order between the
public and private rename. It does mean we could get a state where the
public key exists but is garbage, but this should be OK because the
private key won't exist and it would be regenerated.

The third sync can be removed and I can put one final sync at the end
after all keys are generated to ensure we won't go through all the
effort of regenerating the (last) key again in the event of
interruption shortly after, unless you would prefer I didn't.



The typical convention for this is,

1. Update file data.
2. sync file
3. sync containing directory
4. mv file to new location
5. sync directory containing new file (although I've seen this step left out 
before)

https://lwn.net/Articles/457672/ is a good example which is linked from 
https://lwn.net/Articles/457667/


But one of the important parts vs your code, is also that the example is only 
calling sync on the files/directory needed, vs calling "sync" with no arguments 
which is going to cause all data in all filesystem caches to be flushed.






diff --git a/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
b/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
index 148e6ad..af56404 100644
--- a/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
+++ b/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
@@ -1,22 +1,14 @@
  [Unit]
  Description=OpenSSH Key Generation
  RequiresMountsFor=/var /run
-ConditionPathExists=!/var/run/ssh/ssh_host_rsa_key
-ConditionPathExists=!/var/run/ssh/ssh_host_dsa_key
-ConditionPathExists=!/var/run/ssh/ssh_host_ecdsa_key
-ConditionPathExists=!/var/run/ssh/ssh_host_ed25519_key
-ConditionPathExists=!/etc/ssh/ssh_host_rsa_key
-ConditionPathExists=!/etc/ssh/ssh_host_dsa_key
-ConditionPathExists=!/etc/ssh/ssh_host_ecdsa_key
-ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key



Can you not continue to use ConditionPathExists to only run this unit if it
needs to run?  You can prepend the argument with | to make them logical OR
instead of logical AND, if I'm reading this documentation correctly.


I will try that. I wasn't aware that was an option since systemd conf
files are somewhat new to me.



Ross


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH resend] qemuboot.conf: make cpus match built artifacts

[Randy Witt]

On 05/22/2017 10:29 AM, Martin Kelly wrote:

(friendly ping)

On 05/02/2017 12:20 PM, Martin Kelly wrote:

Currently, the qemu CPUs for are specified as generic, but the built
artifacts are not. For example, we build x86-64 artifacts targeting
core2duo but run them in qemu with generic qemu/kvm CPUs. This causes
some packages that take advantage of the host architecture to crash
because they try to use CPU features not advertised by qemu. As an
example, Qt uses ssse3. When artifacts linked against Qt and built
targeting core2duo attempt to run on a generic qemu/kvm CPU, we get
the following crash:

Incompatible processor. This Qt build requires the following features:
 ssse3

We could fix this by making packages like Qt not take advantage of CPU
features. However, we will probably keep facing similar issues over
time, so it's better to resolve them in a more enduring way.


If the MACHINE is a generic qemu, it seems more correct to build without the 
extensions. For instance, what happens when core2duo doesn't have all the 
necessary instructions that some package decided to use?


I like the idea of being able to exercise the code, but I only see this fix as 
pushing the maintenance until the problem appears again later.




Fix this by making the qemu -cpu arguments match the built artifacts.

Signed-off-by: Martin Kelly 
---

I sent this to p...@yoctoproject.org but it should have gone to OE-core,
so I'm resending it now to restart the discussion on the right mailing
list. There were some comments about it in a previous mail thread on the
poky mailing list:

https://lists.yoctoproject.org/pipermail/poky/2017-April/010956.html

 meta/conf/machine/include/qemuboot-x86.inc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/conf/machine/include/qemuboot-x86.inc 
b/meta/conf/machine/include/qemuboot-x86.inc

index 06ac983..acd03a1 100644
--- a/meta/conf/machine/include/qemuboot-x86.inc
+++ b/meta/conf/machine/include/qemuboot-x86.inc
@@ -1,12 +1,12 @@
 # For runqemu
 IMAGE_CLASSES += "qemuboot"
 QB_SYSTEM_NAME_x86 = "qemu-system-i386"
-QB_CPU_x86 = "-cpu qemu32"
-QB_CPU_KVM_x86 = "-cpu kvm32"
+QB_CPU_x86 = "-cpu pentium2"
+QB_CPU_KVM_x86 = "-cpu pentium2"

 QB_SYSTEM_NAME_x86-64 = "qemu-system-x86_64"
 QB_CPU_x86-64 = "-cpu core2duo"
-QB_CPU_KVM_x86-64 = "-cpu kvm64"
+QB_CPU_KVM_x86-64 = "-cpu core2duo"

 QB_AUDIO_DRV = "alsa"
 QB_AUDIO_OPT = "-soundhw ac97,es1370"



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2] libxslt: Add PACKAGECONFIG support for python

[Randy Witt]

Hi Patel,

On 03/22/2017 06:05 PM, Vedang Patel wrote:

Add support to enable python bindings for libxslt using PACKAGECONFIG.
This will include --with-python argument while compiling the package.
The support is disabled by default.

Signed-off-by: Vedang Patel 
---
 meta/recipes-support/libxslt/libxslt_1.1.29.bb | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-support/libxslt/libxslt_1.1.29.bb 
b/meta/recipes-support/libxslt/libxslt_1.1.29.bb
index be747e608d9d..1e00306207c5 100644
--- a/meta/recipes-support/libxslt/libxslt_1.1.29.bb
+++ b/meta/recipes-support/libxslt/libxslt_1.1.29.bb
@@ -22,7 +22,7 @@ S = "${WORKDIR}/libxslt-${PV}"

 BINCONFIG = "${bindir}/xslt-config"

-inherit autotools pkgconfig binconfig-disabled lib_package
+inherit autotools pkgconfig binconfig-disabled lib_package 
distutils-common-base

 # We don't DEPEND on binutils for ansidecl.h so ensure we don't use the header
 do_configure_prepend () {
@@ -33,7 +33,9 @@ do_configure_prepend () {
touch ${S}/doc/xsltproc.1
 }

-EXTRA_OECONF = "--without-python --without-debug --without-mem-debug 
--without-crypto"
+EXTRA_OECONF = "--without-debug --without-mem-debug --without-crypto"
+PACKAGECONFIG[libxslt-python] = "--with-python=${PYTHON_BASE_VERSION}, 
--without-python"
+

You kept "libxslt-python", and I think you meant to change it to "python".


 # older versions of this recipe had ${PN}-utils
 RPROVIDES_${PN}-bin += "${PN}-utils"
 RCONFLICTS_${PN}-bin += "${PN}-utils"



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v4 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test

[Randy Witt]

This test checks to make sure only the files expected exist in a
container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to
all images without the user specifying it.

But this test should help if a developer in the future ever silently
adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can
make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 .../container-image/container-image-testpkg.bb |  8 +++
 .../container-image/container-test-image.bb|  8 +++
 meta/lib/oeqa/selftest/containerimage.py   | 83 ++
 3 files changed, 99 insertions(+)
 create mode 100644 
meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 
meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py

diff --git 
a/meta-selftest/recipes-test/container-image/container-image-testpkg.bb 
b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
new file mode 100644
index 000..f8dd229
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+install -d ${D}${bindir}
+touch ${D}${bindir}/theapp
+}
diff --git a/meta-selftest/recipes-test/container-image/container-test-image.bb 
b/meta-selftest/recipes-test/container-image/container-test-image.bb
new file mode 100644
index 000..d5f939c
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-test-image.bb
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
diff --git a/meta/lib/oeqa/selftest/containerimage.py 
b/meta/lib/oeqa/selftest/containerimage.py
new file mode 100644
index 000..def481f
--- /dev/null
+++ b/meta/lib/oeqa/selftest/containerimage.py
@@ -0,0 +1,83 @@
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import bitbake, get_bb_vars, runCmd
+
+# This test builds an image with using the "container" IMAGE_FSTYPE, and
+# ensures that then files in the image are only the ones expected.
+#
+# The only package added to the image is container_image_testpkg, which
+# contains one file. However, due to some other things not cleaning up during
+# rootfs creation, there is some cruft. Ideally bugs will be filed and the
+# cruft removed, but for now we whitelist some known set.
+#
+# Also for performance reasons we're only checking the cruft when using ipk.
+# When using deb, and rpm it is a bit different and we could test all
+# of them, but this test is more to catch if other packages get added by
+# default other than what is in ROOTFS_BOOTSTRAP_INSTALL.
+#
+class ContainerImageTests(oeSelfTest):
+
+# Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that
+# the conversion type bar gets added as a dep as well
+def test_expected_files(self):
+
+def get_each_path_part(path):
+if path:
+part = [ '.' + path + '/' ]
+result = get_each_path_part(path.rsplit('/', 1)[0])
+if result:
+return part + result
+else:
+return part
+else:
+return None
+
+self.write_config("""PREFERRED_PROVIDER_virtual/kernel = "linux-dummy"
+IMAGE_FSTYPES = "container"
+PACKAGE_CLASSES = "package_ipk"
+IMAGE_FEATURES = ""
+""")
+
+bbvars = get_bb_vars(['bindir', 'sysconfdir', 'localstatedir',
+  'DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'],
+  target='container-test-image')
+expected_files = [
+'./',
+'.{bindir}/theapp',
+'.{sysconfdir}/default/',
+'.{sysconfdir}/default/postinst',
+'.{sysconfdir}/ld.so.cache',
+'.{sysconfdir}/timestamp',
+'.{sysconfdir}/version',
+'./run/',
+'.{localstatedir}/cache/',
+'.{localstatedir}/cache/ldconfig/',
+'.{localstatedir}/cache/ldconfig/aux-cache',
+'.{localstatedir}/cache/opkg/',
+'.{localstatedir}/lib/',
+'.{localstatedir}/lib/opkg/'
+]
+
+expected_files = [ x.format(bindir=bbvars['bindir'],
+sysconfdir=bbvars['sysconfdir'],
+localstatedir=bbvars['localstatedir'])
+for x in expected_files ]
+
+# Since tar lists all di

[OE-core] [PATCH v3 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test

[Randy Witt]

This test checks to make sure only the files expected exist in a
container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to
all images without the user specifying it.

But this test should help if a developer in the future ever silently
adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can
make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 .../container-image/container-image-testpkg.bb |  8 +++
 .../container-image/container-test-image.bb|  8 +++
 meta/lib/oeqa/selftest/containerimage.py   | 82 ++
 3 files changed, 98 insertions(+)
 create mode 100644 
meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 
meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py

diff --git 
a/meta-selftest/recipes-test/container-image/container-image-testpkg.bb 
b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
new file mode 100644
index 000..f8dd229
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+install -d ${D}${bindir}
+touch ${D}${bindir}/theapp
+}
diff --git a/meta-selftest/recipes-test/container-image/container-test-image.bb 
b/meta-selftest/recipes-test/container-image/container-test-image.bb
new file mode 100644
index 000..d5f939c
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-test-image.bb
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
diff --git a/meta/lib/oeqa/selftest/containerimage.py 
b/meta/lib/oeqa/selftest/containerimage.py
new file mode 100644
index 000..109f61b
--- /dev/null
+++ b/meta/lib/oeqa/selftest/containerimage.py
@@ -0,0 +1,82 @@
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import bitbake, get_bb_vars, runCmd
+
+# This test builds an image with using the "container" IMAGE_FSTYPE, and
+# ensures that then files in the image are only the ones expected.
+#
+# The only package added to the image is container_image_testpkg, which
+# contains one file. However, due to some other things not cleaning up during
+# rootfs creation, there is some cruft. Ideally bugs will be filed and the
+# cruft removed, but for now we whitelist some known set.
+#
+# Also for performance reasons we're only checking the cruft when using ipk.
+# When using deb, and rpm it is a bit different and we could test all
+# of them, but this test is more to catch if other packages get added by
+# default other than what is in ROOTFS_BOOTSTRAP_INSTALL.
+#
+class ContainerImageTests(oeSelfTest):
+
+# Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that
+# the conversion type bar gets added as a dep as well
+def test_expected_files(self):
+
+def get_each_path_part(path):
+if path:
+part = [ '.' + path + '/' ]
+result = get_each_path_part(path.rsplit('/', 1)[0])
+if result:
+return part + result
+else:
+return part
+else:
+return None
+
+self.write_config("""PREFERRED_PROVIDER_virtual/kernel = "linux-dummy"
+IMAGE_FSTYPES = "container"
+PACKAGE_CLASSES = "package_ipk"
+IMAGE_FEATURES = ""
+""")
+
+bbvars = get_bb_vars(['bindir', 'sysconfdir', 'localstatedir',
+  'DEPLOY_DIR_IMAGE'])
+expected_files = [
+'./',
+'.{bindir}/theapp',
+'.{sysconfdir}/default/',
+'.{sysconfdir}/default/postinst',
+'.{sysconfdir}/ld.so.cache',
+'.{sysconfdir}/timestamp',
+'.{sysconfdir}/version',
+'./run/',
+'.{localstatedir}/cache/',
+'.{localstatedir}/cache/ldconfig/',
+'.{localstatedir}/cache/ldconfig/aux-cache',
+'.{localstatedir}/cache/opkg/',
+'.{localstatedir}/lib/',
+'.{localstatedir}/lib/opkg/'
+]
+
+expected_files = [ x.format(bindir=bbvars['bindir'],
+sysconfdir=bbvars['sysconfdir'],
+localstatedir=bbvars['localstatedir'])
+for x in expected_files ]
+
+# Since tar lists all directories individually, make sure each element
+# from bindi

[OE-core] [PATCH v2 1/3] image-container.bbclass: Add the "container" IMAGE_FSTYPES

[Randy Witt]

The "container" fstype does very little other than pick tar.bz2 as the
actual image type and disable installation of ROOTFS_BOOTSTRAP_INSTALL.

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/image-container.bbclass | 3 +++
 meta/classes/image.bbclass   | 3 +++
 meta/classes/image_types.bbclass | 1 +
 3 files changed, 7 insertions(+)
 create mode 100644 meta/classes/image-container.bbclass

diff --git a/meta/classes/image-container.bbclass 
b/meta/classes/image-container.bbclass
new file mode 100644
index 000..a10ee32
--- /dev/null
+++ b/meta/classes/image-container.bbclass
@@ -0,0 +1,3 @@
+ROOTFS_BOOTSTRAP_INSTALL = ""
+IMAGE_TYPES_MASKED += "container"
+IMAGE_TYPEDEP_container = "tar.bz2"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 04fd5f9..b2c2a28 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -144,6 +144,9 @@ inherit ${IMAGE_TYPE_live}
 IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", 
"qcow2", "hdddirect"], "image-vm", "", d)}'
 inherit ${IMAGE_TYPE_vm}
 
+IMAGE_TYPE_container = '${@bb.utils.contains("IMAGE_FSTYPES", "container", 
"image-container", "", d)}'
+inherit ${IMAGE_TYPE_container}
+
 def build_uboot(d):
 if 'u-boot' in (d.getVar('IMAGE_FSTYPES') or ''):
 return "image_types_uboot"
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 7748dee..727a87e 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -328,6 +328,7 @@ IMAGE_TYPES = " \
 hdddirect \
 elf \
 wic wic.gz wic.bz2 wic.lzma \
+container \
 "
 
 # Compression is a special case of conversion. The old variable
-- 
2.9.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 2/3] image-container.bbclass: Error if not using linux-dummy

[Randy Witt]

In the typical case of a container, the user doesn't particularly care
about the kernel version used. Since using linux-dummy prevents building
a kernel in the most common case, alert the user if they choose a
container image, but don't also specify linux-dummy.

In the cases the user actually does want to build against particular
kernel headers, etc, they can override by setting
IMAGE_CONTAINER_NO_DUMMY = "1".

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/image-container.bbclass | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/meta/classes/image-container.bbclass 
b/meta/classes/image-container.bbclass
index a10ee32..f002858 100644
--- a/meta/classes/image-container.bbclass
+++ b/meta/classes/image-container.bbclass
@@ -1,3 +1,21 @@
 ROOTFS_BOOTSTRAP_INSTALL = ""
 IMAGE_TYPES_MASKED += "container"
 IMAGE_TYPEDEP_container = "tar.bz2"
+
+python __anonymous() {
+if "container" in d.getVar("IMAGE_FSTYPES") and \
+   d.getVar("IMAGE_CONTAINER_NO_DUMMY") != "1" and \
+   "linux-dummy" not in d.getVar("PREFERRED_PROVIDER_virtual/kernel"):
+msg = '"container" is in IMAGE_FSTYPES, but ' \
+  'PREFERRED_PROVIDER_virtual/kernel is not "linux-dummy". ' \
+  'Unless a particular kernel is needed, using linux-dummy will ' \
+  'prevent a kernel from being built, which can reduce ' \
+  'build times. If you don\'t want to use "linux-dummy", set ' \
+  '"IMAGE_CONTAINER_NO_DUMMY" to "1".'
+
+# Raising skip recipe was Paul's clever idea. It causes the error to
+# only be shown for the recipes actually requested to build, rather
+# than bb.fatal which would appear for all recipes inheriting the
+# class.
+raise bb.parse.SkipRecipe(msg)
+}
-- 
2.9.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 0/3] Add initial container IMAGE_FSTYPES support

[Randy Witt]

Updated with Ross' requests.

The following changes since commit a304b94eae99ade6d99800eeaf588c48d01506f6:

  oeqa/selftest/devtool: rewrite modify testcase (2017-01-19 22:47:22 +)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/contwip
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/contwip

Randy Witt (3):
  image-container.bbclass: Add the "container" IMAGE_FSTYPES
  image-container.bbclass: Error if not using linux-dummy
  selftest/containerimage.py: Add container IMAGE_FSTYPES test

 .../container-image/container-image-testpkg.bb |  8 +++
 .../container-image/container-test-image.bb|  8 +++
 meta/classes/image-container.bbclass   | 21 ++
 meta/classes/image.bbclass |  3 +
 meta/classes/image_types.bbclass   |  1 +
 meta/lib/oeqa/selftest/containerimage.py   | 82 ++
 6 files changed, 123 insertions(+)
 create mode 100644 
meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 
meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/classes/image-container.bbclass
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py

-- 
2.9.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test

[Randy Witt]

This test checks to make sure only the files expected exist in a
container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to
all images without the user specifying it.

But this test should help if a developer in the future ever silently
adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can
make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 .../container-image/container-image-testpkg.bb |  8 +++
 .../container-image/container-test-image.bb|  8 +++
 meta/lib/oeqa/selftest/containerimage.py   | 82 ++
 3 files changed, 98 insertions(+)
 create mode 100644 
meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 
meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py

diff --git 
a/meta-selftest/recipes-test/container-image/container-image-testpkg.bb 
b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
new file mode 100644
index 000..f8dd229
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+install -d ${D}${bindir}
+touch ${D}${bindir}/theapp
+}
diff --git a/meta-selftest/recipes-test/container-image/container-test-image.bb 
b/meta-selftest/recipes-test/container-image/container-test-image.bb
new file mode 100644
index 000..d5f939c
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-test-image.bb
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
diff --git a/meta/lib/oeqa/selftest/containerimage.py 
b/meta/lib/oeqa/selftest/containerimage.py
new file mode 100644
index 000..6a85222
--- /dev/null
+++ b/meta/lib/oeqa/selftest/containerimage.py
@@ -0,0 +1,82 @@
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import bitbake, get_bb_vars, runCmd
+
+# This test builds an image with using the "container" IMAGE_FSTYPE, and
+# ensures that then files in the image are only the ones expected.
+#
+# The only package added to the image is container_image_testpkg, which
+# contains one file. However, due to some other things not cleaning up during
+# rootfs creation, there is some cruft. Ideally bugs will be filed and the
+# cruft removed, but for now we whitelist some known set.
+#
+# Also for performance reasons we're only checking the cruft when using ipk.
+# When using deb, and rpm it is a bit different and we could test all
+# of them, but this test is more to catch if other packages get added by
+# default other than what is in ROOTFS_BOOTSTRAP_INSTALL.
+#
+class ContainerImageTests(oeSelfTest):
+
+# Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that
+# the conversion type bar gets added as a dep as well
+def test_expected_files(self):
+
+def get_each_path_part(path):
+if path:
+part = [ '.' + path + '/' ]
+result = get_each_path_part(path.rsplit('/', 1)[0])
+if result:
+return part + result
+else:
+return part
+else:
+return None
+
+bbvars = get_bb_vars(['bindir', 'sysconfdir', 'localstatedir',
+  'DEPLOY_DIR_IMAGE'])
+expected_files = [
+'./',
+'.{bindir}/theapp',
+'.{sysconfdir}/default/',
+'.{sysconfdir}/default/postinst',
+'.{sysconfdir}/ld.so.cache',
+'.{sysconfdir}/timestamp',
+'.{sysconfdir}/version',
+'./run/',
+'.{localstatedir}/cache/',
+'.{localstatedir}/cache/ldconfig/',
+'.{localstatedir}/cache/ldconfig/aux-cache',
+'.{localstatedir}/cache/opkg/',
+'.{localstatedir}/lib/',
+'.{localstatedir}/lib/opkg/'
+]
+
+expected_files = [ x.format(bindir=bbvars['bindir'],
+sysconfdir=bbvars['sysconfdir'],
+localstatedir=bbvars['localstatedir'])
+for x in expected_files ]
+
+# Since tar lists all directories individually, make sure each element
+# from bindir, sysconfdir, etc is added
+expected_files += get_each_path_part(bbvars['bindir'])
+expected_files += get_each_path_part(bbvars['sysconfdir'])
+expected_files += get_each_path_part(bbvars['localstatedir'])
+
+expected_files = so

Re: [OE-core] [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test

[Randy Witt]

On 01/19/2017 04:01 AM, Burton, Ross wrote:

On 18 January 2017 at 18:31, Randy Witt <randy.e.w...@linux.intel.com>
wrote:


+expected_files = [
+'./',
+'./bin/',
+'./bin/theapp',
+'./etc/',
+'./etc/default/',
+'./etc/default/postinst',
+'./etc/ld.so.cache',
+'./etc/timestamp',
+'./etc/version',
+'./run/',
+'./var/',
+'./var/cache/',
+'./var/cache/ldconfig/',
+'./var/cache/ldconfig/aux-cache',
+'./var/cache/opkg/',
+'./var/lib/',
+'./var/lib/opkg/'
+]

These should use ${bindir} etc so they're resilient to distros changing

the paths (please use get_bb_vars() to only invoke bitbake once).

+image = os.path.join(self.builddir, "tmp/deploy/images/qemux86/

container-test-image-qemux86.tar.bz2")



Don't hardcode this path, use ${DEPLOY_DIR_IMAGE} (fetch it with the paths
above for speed) which will get you to tmp/deploy/images/[machine].  Again,
these paths can change (poky sets TMPDIR to tmp, nodistro uses
tmp-${TCLIBC}).

Ross



I just sent a v2, but screwed up the subject. I'll resend the whole series again 
even though only 3/3 changed, to hopefully make it easier on you.


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] selftest/containerimage.py: Add container IMAGE_FSTYPES test

[Randy Witt]

This test checks to make sure only the files expected exist in a
container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to
all images without the user specifying it.

But this test should help if a developer in the future ever silently
adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can
make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 .../container-image/container-image-testpkg.bb |  8 +++
 .../container-image/container-test-image.bb|  8 +++
 meta/lib/oeqa/selftest/containerimage.py   | 82 ++
 3 files changed, 98 insertions(+)
 create mode 100644 
meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 
meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py

diff --git 
a/meta-selftest/recipes-test/container-image/container-image-testpkg.bb 
b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
new file mode 100644
index 000..f8dd229
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+install -d ${D}${bindir}
+touch ${D}${bindir}/theapp
+}
diff --git a/meta-selftest/recipes-test/container-image/container-test-image.bb 
b/meta-selftest/recipes-test/container-image/container-test-image.bb
new file mode 100644
index 000..d5f939c
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-test-image.bb
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
diff --git a/meta/lib/oeqa/selftest/containerimage.py 
b/meta/lib/oeqa/selftest/containerimage.py
new file mode 100644
index 000..6a85222
--- /dev/null
+++ b/meta/lib/oeqa/selftest/containerimage.py
@@ -0,0 +1,82 @@
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import bitbake, get_bb_vars, runCmd
+
+# This test builds an image with using the "container" IMAGE_FSTYPE, and
+# ensures that then files in the image are only the ones expected.
+#
+# The only package added to the image is container_image_testpkg, which
+# contains one file. However, due to some other things not cleaning up during
+# rootfs creation, there is some cruft. Ideally bugs will be filed and the
+# cruft removed, but for now we whitelist some known set.
+#
+# Also for performance reasons we're only checking the cruft when using ipk.
+# When using deb, and rpm it is a bit different and we could test all
+# of them, but this test is more to catch if other packages get added by
+# default other than what is in ROOTFS_BOOTSTRAP_INSTALL.
+#
+class ContainerImageTests(oeSelfTest):
+
+# Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that
+# the conversion type bar gets added as a dep as well
+def test_expected_files(self):
+
+def get_each_path_part(path):
+if path:
+part = [ '.' + path + '/' ]
+result = get_each_path_part(path.rsplit('/', 1)[0])
+if result:
+return part + result
+else:
+return part
+else:
+return None
+
+bbvars = get_bb_vars(['bindir', 'sysconfdir', 'localstatedir',
+  'DEPLOY_DIR_IMAGE'])
+expected_files = [
+'./',
+'.{bindir}/theapp',
+'.{sysconfdir}/default/',
+'.{sysconfdir}/default/postinst',
+'.{sysconfdir}/ld.so.cache',
+'.{sysconfdir}/timestamp',
+'.{sysconfdir}/version',
+'./run/',
+'.{localstatedir}/cache/',
+'.{localstatedir}/cache/ldconfig/',
+'.{localstatedir}/cache/ldconfig/aux-cache',
+'.{localstatedir}/cache/opkg/',
+'.{localstatedir}/lib/',
+'.{localstatedir}/lib/opkg/'
+]
+
+expected_files = [ x.format(bindir=bbvars['bindir'],
+sysconfdir=bbvars['sysconfdir'],
+localstatedir=bbvars['localstatedir'])
+for x in expected_files ]
+
+# Since tar lists all directories individually, make sure each element
+# from bindir, sysconfdir, etc is added
+expected_files += get_each_path_part(bbvars['bindir'])
+expected_files += get_each_path_part(bbvars['sysconfdir'])
+expected_files += get_each_path_part(bbvars['localstatedir'])
+
+expected_files = so

[OE-core] [PATCH 1/3] image-container.bbclass: Add the "container" IMAGE_FSTYPES

[Randy Witt]

The "container" fstype does very little other than pick tar.bz2 as the
actual image type and disable installation of ROOTFS_BOOTSTRAP_INSTALL.

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/image-container.bbclass | 3 +++
 meta/classes/image.bbclass   | 3 +++
 meta/classes/image_types.bbclass | 1 +
 3 files changed, 7 insertions(+)
 create mode 100644 meta/classes/image-container.bbclass

diff --git a/meta/classes/image-container.bbclass 
b/meta/classes/image-container.bbclass
new file mode 100644
index 000..a10ee32
--- /dev/null
+++ b/meta/classes/image-container.bbclass
@@ -0,0 +1,3 @@
+ROOTFS_BOOTSTRAP_INSTALL = ""
+IMAGE_TYPES_MASKED += "container"
+IMAGE_TYPEDEP_container = "tar.bz2"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 04fd5f9..b2c2a28 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -144,6 +144,9 @@ inherit ${IMAGE_TYPE_live}
 IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", 
"qcow2", "hdddirect"], "image-vm", "", d)}'
 inherit ${IMAGE_TYPE_vm}
 
+IMAGE_TYPE_container = '${@bb.utils.contains("IMAGE_FSTYPES", "container", 
"image-container", "", d)}'
+inherit ${IMAGE_TYPE_container}
+
 def build_uboot(d):
 if 'u-boot' in (d.getVar('IMAGE_FSTYPES') or ''):
 return "image_types_uboot"
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 7748dee..727a87e 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -328,6 +328,7 @@ IMAGE_TYPES = " \
 hdddirect \
 elf \
 wic wic.gz wic.bz2 wic.lzma \
+container \
 "
 
 # Compression is a special case of conversion. The old variable
-- 
2.9.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/3] Add initial container IMAGE_FSTYPES support

[Randy Witt]

The following changes since commit 63f899a950daf1018999455bafa7a2be8b22f164:

  bitbake: toaster: bin/toaster whitelist TOASTER_DIR (2017-01-17 13:18:47 
+)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/contwip
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/contwip

Randy Witt (3):
  image-container.bbclass: Add the "container" IMAGE_FSTYPES
  image-container.bbclass: Error if not using linux-dummy
  selftest/containerimage.py: Add container IMAGE_FSTYPES test

 .../container-image/container-image-testpkg.bb |  8 +++
 .../container-image/container-test-image.bb|  8 +++
 meta/classes/image-container.bbclass   | 21 
 meta/classes/image.bbclass |  3 ++
 meta/classes/image_types.bbclass   |  1 +
 meta/lib/oeqa/selftest/containerimage.py   | 57 ++
 6 files changed, 98 insertions(+)
 create mode 100644 
meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 
meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/classes/image-container.bbclass
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py

-- 
2.9.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test

[Randy Witt]

This test checks to make sure only the files expected exist in a
container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to
all images without the user specifying it.

But this test should help if a developer in the future ever silently
adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can
make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 .../container-image/container-image-testpkg.bb |  8 +++
 .../container-image/container-test-image.bb|  8 +++
 meta/lib/oeqa/selftest/containerimage.py   | 57 ++
 3 files changed, 73 insertions(+)
 create mode 100644 
meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 
meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py

diff --git 
a/meta-selftest/recipes-test/container-image/container-image-testpkg.bb 
b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
new file mode 100644
index 000..42014dd
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+   install -d ${D}${base_bindir}
+touch ${D}${base_bindir}/theapp
+}
diff --git a/meta-selftest/recipes-test/container-image/container-test-image.bb 
b/meta-selftest/recipes-test/container-image/container-test-image.bb
new file mode 100644
index 000..d5f939c
--- /dev/null
+++ b/meta-selftest/recipes-test/container-image/container-test-image.bb
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
diff --git a/meta/lib/oeqa/selftest/containerimage.py 
b/meta/lib/oeqa/selftest/containerimage.py
new file mode 100644
index 000..54e792a
--- /dev/null
+++ b/meta/lib/oeqa/selftest/containerimage.py
@@ -0,0 +1,57 @@
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import bitbake, runCmd
+
+# This test builds an image with using the "container" IMAGE_FSTYPE, and
+# ensures that then files in the image are only the ones expected.
+#
+# The only package added to the image is container_image_testpkg, which
+# contains one file. However, due to some other things not cleaning up during
+# rootfs creation, there is some kruft. Ideally bugs will be filed and the
+# kruft removed, but for now we whitelist some known set.
+#
+# Also for performance reason we're only checking the kruft when using ipk.
+# When using deb, and rpm it is a bit different and we could test all
+# of them, but this test is more to catch if other packages get added by
+# default other than what is in ROOTFS_BOOTSTRAP_INSTALL.
+#
+class ContainerImageTests(oeSelfTest):
+
+# Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that
+# the conversion type bar gets added as a dep as well
+def test_expected_files(self):
+
+expected_files = [
+'./',
+'./bin/',
+'./bin/theapp',
+'./etc/',
+'./etc/default/',
+'./etc/default/postinst',
+'./etc/ld.so.cache',
+'./etc/timestamp',
+'./etc/version',
+'./run/',
+'./var/',
+'./var/cache/',
+'./var/cache/ldconfig/',
+'./var/cache/ldconfig/aux-cache',
+'./var/cache/opkg/',
+'./var/lib/',
+'./var/lib/opkg/'
+]
+
+self.write_config('PREFERRED_PROVIDER_virtual/kernel = "linux-dummy"')
+self.append_config('MACHINE = "qemux86"')
+self.append_config('IMAGE_FSTYPES = "container"')
+self.append_config('PACKAGE_CLASSES = "package_ipk"')
+
+# Build the image of course
+bitbake('container-test-image')
+
+image = os.path.join(self.builddir, 
"tmp/deploy/images/qemux86/container-test-image-qemux86.tar.bz2")
+
+# Ensure the files in the image are what we expect
+result = runCmd("tar tf {} | sort".format(image), shell=True)
+self.assertEqual(result.output.split('\n'), expected_files)
-- 
2.9.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/3] image-container.bbclass: Error if not using linux-dummy

[Randy Witt]

In the typical case of a container, the user doesn't particularly care
about the kernel version used. Since using linux-dummy prevents building
a kernel in the most common case, alert the user if they choose a
container image, but don't also specify linux-dummy.

In the cases the user actually does want to build against particular
kernel headers, etc, they can override by setting
IMAGE_CONTAINER_NO_DUMMY = "1".

[YOCTO #9502]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/image-container.bbclass | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/meta/classes/image-container.bbclass 
b/meta/classes/image-container.bbclass
index a10ee32..f002858 100644
--- a/meta/classes/image-container.bbclass
+++ b/meta/classes/image-container.bbclass
@@ -1,3 +1,21 @@
 ROOTFS_BOOTSTRAP_INSTALL = ""
 IMAGE_TYPES_MASKED += "container"
 IMAGE_TYPEDEP_container = "tar.bz2"
+
+python __anonymous() {
+if "container" in d.getVar("IMAGE_FSTYPES") and \
+   d.getVar("IMAGE_CONTAINER_NO_DUMMY") != "1" and \
+   "linux-dummy" not in d.getVar("PREFERRED_PROVIDER_virtual/kernel"):
+msg = '"container" is in IMAGE_FSTYPES, but ' \
+  'PREFERRED_PROVIDER_virtual/kernel is not "linux-dummy". ' \
+  'Unless a particular kernel is needed, using linux-dummy will ' \
+  'prevent a kernel from being built, which can reduce ' \
+  'build times. If you don\'t want to use "linux-dummy", set ' \
+  '"IMAGE_CONTAINER_NO_DUMMY" to "1".'
+
+# Raising skip recipe was Paul's clever idea. It causes the error to
+# only be shown for the recipes actually requested to build, rather
+# than bb.fatal which would appear for all recipes inheriting the
+# class.
+raise bb.parse.SkipRecipe(msg)
+}
-- 
2.9.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Randy Witt]

On Mon, Jan 9, 2017 at 2:25 AM, Patrick Ohly  wrote:

>
> >
> > In principle I agree. Okay, let's separate the two.
>
> I hit one snag when starting to do that: the implementation of download
> removal depends on do_rm_work from rm_work.bbclass for hooking into the
> build of a recipe at the right place.
>
> I can't think of an easy way of avoiding that without duplicating the
> complex logic from rm_work.bbclass, which itself is currently about to
> change, so I'd prefer to keep rm_work_and_downloads.bbclass as proposed.
>

Sounds good to me. If it turns out someone would like them separate in the
future, we can schedule the time to do it then.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Randy Witt]

On Fri, Jan 6, 2017 at 11:22 AM, Patrick Ohly <patrick.o...@intel.com>
wrote:

> On Fri, 2017-01-06 at 10:31 -0800, Randy Witt wrote:
>
> >
> > I'd rather see this be a new class independent from rm_work. People
> > can always in inherit both rm_work and rm_download.
>
> I'm not sure whether it's worth supporting that mode: what would be the
> motivation for removing downloads, but not the work directory?
>
>
There are times that the work directories help with the debugging of build
failures. The logs aren't always enough. So a person might want to do
something like remove the downloads, but preserve the actual work,
especially in the case of failures.

I'll admit it is a fabricated scenario, but in general I'm against one knob
requiring the turning of another knob.


> The motivation for rm_work_and_downloads.bbclass is the same as for
> rm_work.bbclass: reducing the required disk space *during* the build, to
> get builds to succeed which otherwise would run out of disk space.
>
>
Disk space isn't the only motivation for rm_work. It also allows for
deletion of large amounts of inodes before they've been flushed to disk. So
rather than taking minutes worth of time to clean up post build, and
thrashing media that other tasks may be using, it is almost instantaneous
because everything is still in the page cache.

Also the build itself can be faster for the same reason, there is no
waiting for data to be flushed out to disk( because it was deleted while
still in the page cache) and thus impacting reads in other parts of the
build.


> A "rm -rf downloads" at the end of the build doesn't help achieve that
> goal.
>
>
That's true, I hadn't yet processed it completely. Ignore that comment. :)

As an overall comment though, I really like the idea of rm_work not being
deferred for so long. It always seemed strange to me to see the thundering
herd of rm_works after certain do_builds would finish.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Randy Witt]

Hi Patrick,

On Fri, Jan 6, 2017 at 1:55 AM, Patrick Ohly  wrote:

> rm_work.bbclass never deletes downloaded files, even if they are not
> going to be needed again during the
> build. rm_work_and_downloads.bbclass is more aggressive in minimizing
> the used disk space during a build, but has other disadvantages:
> - sources required by different recipes need to be fetched once per
>   recipe, not once per build
> - incremental builds do not work reliably because sources get
>   removed without ensuring that sources gets fetched again
>
> That makes rm_work_and_downloads.bbclass useful for one-time builds in
> a constrained environment (like a CI system), but not for general use.
>
>
I'd rather see this be a new class independent from rm_work. People can
always in inherit both rm_work and rm_download.

That being said, since this doesn't operate at the fetch level, i.e. only
remove things that were fetched as part of the current invocation of
bitbake, I don't see why the user couldn't just "rm -rf downloads"
themselves as part of the ci teardown.


> Signed-off-by: Patrick Ohly 
> ---
>  meta/classes/rm_work_and_downloads.bbclass | 33
> ++
>  1 file changed, 33 insertions(+)
>  create mode 100644 meta/classes/rm_work_and_downloads.bbclass
>
> diff --git a/meta/classes/rm_work_and_downloads.bbclass
> b/meta/classes/rm_work_and_downloads.bbclass
> new file mode 100644
> index 000..7c00bea
> --- /dev/null
> +++ b/meta/classes/rm_work_and_downloads.bbclass
> @@ -0,0 +1,33 @@
> +# Author:   Patrick Ohly 
> +# Copyright:Copyright (C) 2015 Intel Corporation
> +#
> +# This file is licensed under the MIT license, see COPYING.MIT in
> +# this source distribution for the terms.
> +
> +# This class is used like rm_work:
> +# INHERIT += "rm_work_and_downloads"
> +#
> +# In addition to removing local build directories of a recipe, it also
> +# removes the downloaded source. This is achieved by making the DL_DIR
> +# recipe-specific. While reducing disk usage, it increases network usage
> (for
> +# example, compiling the same source for target and host implies
> downloading
> +# the source twice).
> +#
> +# Because the "do_fetch" task does not get re-run after removing the
> downloaded
> +# sources, this class is also not suitable for incremental builds.
> +#
> +# Where it works well is in well-connected build environments with limited
> +# disk space (like TravisCI).
> +
> +inherit rm_work
> +
> +# This would ensure that the existing do_rm_work() removes the downloads,
> +# but does not work because some recipes have a circular dependency
> between
> +# WORKDIR and DL_DIR (via ${SRCPV}?).
> +# DL_DIR = "${WORKDIR}/downloads"
> +
> +# Instead go up one level and remove ourself.
> +DL_DIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/downloads"
> +do_rm_work_append () {
> +rm -rf ${DL_DIR}
> +}
> --
> 2.1.4
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] image_types.bbclass: IMAGE_TYPEDEP_ now adds deps for conversion types

[Randy Witt]

Previously if IMAGE_TYPEDEP_* contained a conversion type of the form,
"foo.bar", the dependency on CONVERSION_DEPENDS_bar would not get added
to the task depends for do_rootfs.

[YOCTO #10883]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/image_types.bbclass | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 8f04849..c2ad141 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -17,17 +17,25 @@ def imagetypes_getdepends(d):
 d += ":do_populate_sysroot"
 deps.add(d)
 
+# Take a type in the form of foo.bar.car and split it into the items
+# needed for the image deps "foo", and the conversion deps ["bar", "car"]
+def split_types(typestring):
+types = typestring.split(".")
+return types[0], types[1:]
+
 fstypes = set((d.getVar('IMAGE_FSTYPES') or "").split())
 fstypes |= set((d.getVar('IMAGE_FSTYPES_DEBUGFS') or "").split())
 
 deps = set()
 for typestring in fstypes:
-types = typestring.split(".")
-basetype, resttypes = types[0], types[1:]
-
+basetype, resttypes = split_types(typestring)
 adddep(d.getVar('IMAGE_DEPENDS_%s' % basetype) , deps)
+
 for typedepends in (d.getVar("IMAGE_TYPEDEP_%s" % basetype) or 
"").split():
+base, rest = split_types(typedepends)
+resttypes += rest
 adddep(d.getVar('IMAGE_DEPENDS_%s' % typedepends) , deps)
+
 for ctype in resttypes:
 adddep(d.getVar("CONVERSION_DEPENDS_%s" % ctype), deps)
 adddep(d.getVar("COMPRESS_DEPENDS_%s" % ctype), deps)
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] image_typedep.py: Add a test that ensures conversion type deps get added

[Randy Witt]

Add a test that ensures if IMAGE_TYPEDEP_* contains a conversion type,
that the corresponding CONVERSION_DEPENDS_ for that type gets added to
the dependency tree for do_rootfs.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/lib/oeqa/selftest/image_typedep.py | 51 +
 1 file changed, 51 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/image_typedep.py

diff --git a/meta/lib/oeqa/selftest/image_typedep.py 
b/meta/lib/oeqa/selftest/image_typedep.py
new file mode 100644
index 000..256142d25
--- /dev/null
+++ b/meta/lib/oeqa/selftest/image_typedep.py
@@ -0,0 +1,51 @@
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import bitbake
+
+class ImageTypeDepTests(oeSelfTest):
+
+# Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that
+# the conversion type bar gets added as a dep as well
+def test_conversion_typedep_added(self):
+
+self.write_recipeinc('emptytest', """
+# Try to empty out the default dependency list
+PACKAGE_INSTALL = ""
+DISTRO_EXTRA_RDEPENDS=""
+
+LICENSE = "MIT"
+IMAGE_FSTYPES = "testfstype"
+
+IMAGE_TYPES_MASKED += "testfstype"
+IMAGE_TYPEDEP_testfstype = "tar.bz2"
+
+inherit image
+
+""")
+# First get the dependency that should exist for bz2, it will look
+# like CONVERSION_DEPENDS_bz2="somedep"
+result = bitbake('-e emptytest')
+
+for line in result.output.split('\n'):
+if line.startswith('CONVERSION_DEPENDS_bz2'):
+dep = line.split('=')[1].strip('"')
+break
+
+# Now get the dependency task list and check for the expected task
+# dependency
+bitbake('-g emptytest')
+
+taskdependsfile = os.path.join(self.builddir, 'task-depends.dot')
+dep =  dep + ".do_populate_sysroot"
+depfound = False
+expectedline = '"emptytest.do_rootfs" -> "{}"'.format(dep)
+
+with open(taskdependsfile, "r") as f:
+for line in f:
+if line.strip() == expectedline:
+depfound = True
+break
+
+if not depfound:
+raise AssertionError("\"{}\" not found".format(expectedline))
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/2] Fix depends for conversion types in IMAGE_TYPEDEP_*

[Randy Witt]

Fix conversion type dependencies missing when set in IMAGE_TYPEDEP_*. This
also adds a test.

The following changes since commit 9f6a1043f68580ed9604e750fd0f993f933bb66e:

  bitbake: prserv/serv: Tweak stdout manipulation to be stream safe (2017-01-05 
13:54:07 +)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/image_typedep
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/image_typedep

Randy Witt (2):
  image_typedep.py: Add a test that ensures conversion type deps get
added
  image_types.bbclass: IMAGE_TYPEDEP_ now adds deps for conversion types

 meta/classes/image_types.bbclass| 14 +++--
 meta/lib/oeqa/selftest/image_typedep.py | 51 +
 2 files changed, 62 insertions(+), 3 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/image_typedep.py

-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH V2 5/6] runqemu: fixes for slirp, network device and hostfwd

[Randy Witt]

 def setup_slirp(self):
 """Setup user networking"""

 if self.fstype == 'nfs':
 self.setup_nfs()
 self.kernel_cmdline_script += ' ip=dhcp'
-self.set('NETWORK_CMD', self.get('QB_SLIRP_OPT'))
+# Port mapping
+hostfwd = ",hostfwd=tcp::-:22,hostfwd=tcp::2323-:23"
+qb_slirp_opt_default = "-netdev user,id=net0%s" % hostfwd
+qb_slirp_opt = self.get('QB_SLIRP_OPT') or qb_slirp_opt_default
+# Figure out the port
+ports = re.findall('hostfwd=[^-]*:([0-9]+)-[^,-]*', qb_slirp_opt)
+ports = [int(i) for i in ports]
+mac = 2
+# Find a free port to avoid conflicts
+for p in ports[:]:
+p_new = p
+while not check_free_port('localhost', p_new):
+p_new += 1
+mac += 1
+while p_new in ports:
+p_new += 1
+mac += 1
+if p != p_new:
+ports.append(p_new)
+qb_slirp_opt = re.sub(':%s-' % p, ':%s-' % p_new, qb_slirp_opt)
+logger.info("Port forward changed: %s -> %s" % (p, p_new))
Regardless if the port is changed or not, so that things like tests have an 
easier time of figuring out the port mappings, would it be good add a flag that 
prints out the entire list of ports used, or always do it? i.e. "Port forwarding 
:22 2333:23 It's not necessary of course, you can  always look at the 
command line and then check to see if anything is remapped. But that's a bit 
more work.



+mac = "%s%02x" % (self.mac_slirp, mac)
+self.set('NETWORK_CMD', '%s %s' % 
(self.network_device.replace('@MAC@', mac), qb_slirp_opt))


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] qemu*.conf: Add slirp support for all qemu machines

[Randy Witt]

"-device virtio-net-device,netdev=net0" and
"-netdev user,id=net0,hostfwd=tcp::-:22"

Then there is no way for user to override the second part, and we can't
call the first part as a "QB_SLIRP_OPT" since it is only part of them,
so I won't break them into 2, and if the user defines QB_SLIRP_OPT
in there bsp conf file, then it will be replaced totally.



I'm aware that this will only work for one instance, but currently it doesn't 
work at all and it blocks a high priority bug. The only way to make it 100% 
work, would be to have qemu do the binding of the port to a dynamic one and then 
the user queries it after. Otherwise there is always a window where another 
process could bind to the port.


I don't think the bsp should be in the business of choosing which ports to 
forward. And if it is, then it should be a separate option in the qemuboot.conf, 
with just the "-netdev user" part because otherwise the user will always have to 
modify qemuboot.conf just to forward different ports.


It makes more sense to let the user do something like "runqemu slirp=22,65,..." 
and then return back the actual port used in the forwarding.


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] qemu*.conf: Add slirp support for all qemu machines

[Randy Witt]

This patch also moves the "-net user" part out of the config and
into runqemu. This is done because the "-net user" portion isn't
machine specific, and it contains what ports to forward.

[ YOCTO #7887 ]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/conf/machine/include/qemuboot-mips.inc | 1 +
 meta/conf/machine/include/qemuboot-x86.inc  | 2 +-
 meta/conf/machine/qemuarm.conf  | 1 +
 meta/conf/machine/qemuarm64.conf| 2 +-
 meta/conf/machine/qemuppc.conf  | 1 +
 scripts/runqemu | 7 ++-
 6 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/meta/conf/machine/include/qemuboot-mips.inc 
b/meta/conf/machine/include/qemuboot-mips.inc
index 0c60cf2..c85dc86 100644
--- a/meta/conf/machine/include/qemuboot-mips.inc
+++ b/meta/conf/machine/include/qemuboot-mips.inc
@@ -6,3 +6,4 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyS0 console=tty"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-vga cirrus -show-cursor -usb -usbdevice tablet -device 
virtio-rng-pci"
 QB_SYSTEM_NAME = "qemu-system-${TUNE_ARCH}"
+QB_SLIRP_OPT = "-device virtio-net-pci,netdev=net0"
diff --git a/meta/conf/machine/include/qemuboot-x86.inc 
b/meta/conf/machine/include/qemuboot-x86.inc
index 0870294..cc31dab 100644
--- a/meta/conf/machine/include/qemuboot-x86.inc
+++ b/meta/conf/machine/include/qemuboot-x86.inc
@@ -13,4 +13,4 @@ QB_AUDIO_OPT = "-soundhw ac97,es1370"
 QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 
oprofile.timer=1 uvesafb.task_timeout=-1"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet -device 
virtio-rng-pci"
-QB_SLIRP_OPT = "-net nic,model=e1000 -net user,hostfwd=tcp::-:22"
+QB_SLIRP_OPT = "-device virtio-net-pci,netdev=net0"
diff --git a/meta/conf/machine/qemuarm.conf b/meta/conf/machine/qemuarm.conf
index 17402ef..974657b 100644
--- a/meta/conf/machine/qemuarm.conf
+++ b/meta/conf/machine/qemuarm.conf
@@ -18,3 +18,4 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,115200 
console=tty"
 QB_OPT_APPEND = "-show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
 PREFERRED_VERSION_linux-yocto ??= "4.8%"
 QB_DTB = "${@base_version_less_or_equal('PREFERRED_VERSION_linux-yocto', 
'4.7', '', 'zImage-versatile-pb.dtb', d)}"
+QB_SLIRP_OPT = "-device virtio-net-pci,netdev=net0"
diff --git a/meta/conf/machine/qemuarm64.conf b/meta/conf/machine/qemuarm64.conf
index df2010c..ebd360e 100644
--- a/meta/conf/machine/qemuarm64.conf
+++ b/meta/conf/machine/qemuarm64.conf
@@ -18,7 +18,7 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,38400"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-show-cursor -device virtio-rng-pci -monitor null"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device 
virtio-net-device,netdev=net0,mac=@MAC@"
-QB_SLIRP_OPT = "-netdev user,id=net0 -device virtio-net-device,netdev=net0"
+QB_SLIRP_OPT = "-device virtio-net-device,netdev=net0"
 QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=raw -device 
virtio-blk-device,drive=disk0"
 QB_SERIAL_OPT = "-device virtio-serial-device -chardev null,id=virtcon -device 
virtconsole,chardev=virtcon"
 QB_TCPSERIAL_OPT = " -device virtio-serial-device -chardev 
socket,id=virtcon,port=@PORT@,host=127.0.0.1 -device 
virtconsole,chardev=virtcon"
diff --git a/meta/conf/machine/qemuppc.conf b/meta/conf/machine/qemuppc.conf
index 8703c20..4711751 100644
--- a/meta/conf/machine/qemuppc.conf
+++ b/meta/conf/machine/qemuppc.conf
@@ -19,3 +19,4 @@ QB_KERNEL_CMDLINE_APPEND = "console=tty console=ttyS0"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device 
virtio-net-pci,netdev=net0,mac=@MAC@"
+QB_SLIRP_OPT = "-device virtio-net-pci,netdev=net0"
diff --git a/scripts/runqemu b/scripts/runqemu
index af25423..faab522 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -730,7 +730,12 @@ class BaseConfig(object):
 if self.fstype == 'nfs':
 self.setup_nfs()
 self.kernel_cmdline_script += ' ip=dhcp'
-self.set('NETWORK_CMD', self.get('QB_SLIRP_OPT'))
+network_cmd = self.get('QB_SLIRP_OPT')
+
+# The network device created in qemuboot.conf is expected to be
+# id=net0
+network_cmd += ' -netdev user,id=net0,hostfwd=tcp::-:22'
+self.set('NETWORK_CMD', network_cmd)
 
 def setup_tap(self):
 """Setup tap"""
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] mkefidisk.sh: add deprecation warning to the output

[Randy Witt]

We should also document, the wic steps in wiki pages e.g. 
http://wiki.minnowboard.org/Yocto_Project

It's already documented in README.hardware: 
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/README.hardware

And in Yocto manual:
http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware


Thanks, now can you also nudge the minnowboard.org to do the same ?



Pinging you John. :)



--
Regards,
Ed



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [RFC][PATCH] report-error.bbclass: Add support for autosending

[Randy Witt]

On 09/08/2016 02:26 AM, Elizabeth 'pidge' Flanagan wrote:

This commit pulls and extends the functionality of .oe-error-report into
local.conf. It maintains the functionality of .oe-error-report.

It also enables report-error to automatically send error reports to a
specified error report server. This patch enables infrastructure ppl to
set development teams up so that we can gather error metrics automatically.

This relies on a new set of variables:

REPORTERROR[autosend]="yes|no|ask"
REPORTERROR[user]=username
REPORTERROR[email]=email
REPORTERROR[server]=error-report-web instance

For autosend yes and no are pretty self explainatory. ask does not pass
the -y flag to send-error-report, thus enabling the end user to review
the report prior to submission.

Signed-off-by: Elizabeth 'pidge' Flanagan 
---
 meta/classes/report-error.bbclass | 41 +--
 1 file changed, 39 insertions(+), 2 deletions(-)

diff --git a/meta/classes/report-error.bbclass 
b/meta/classes/report-error.bbclass
index 82b5bcd..909adf0 100644
--- a/meta/classes/report-error.bbclass
+++ b/meta/classes/report-error.bbclass
@@ -85,10 +85,47 @@ python errorreport_handler () {
 bb.utils.unlockfile(lock)
 failures = jsondata['failures']
 if(len(failures) > 0):
+
 filename = "error_report_" + e.data.getVar("BUILDNAME", 
True)+".txt"
 datafile = errorreport_savedata(e, jsondata, filename)
-bb.note("The errors for this build are stored in %s\nYou can send 
the errors to a reports server by running:\n  send-error-report %s [-s server]" % 
(datafile, datafile))
-bb.note("The contents of these logs will be posted in public if you 
use the above command with the default server. Please ensure you remove any identifying 
or proprietary information when prompted before sending.")
+
+cmd_s = "send-error-report %s" % datafile
+response_s = "The errors for this build are stored in %s\n" % 
datafile
+sendit = False
+
+autosend = e.data.getVarFlag("REPORTERROR","autosend")
+user = e.data.getVarFlag("REPORTERROR","user")
+email = e.data.getVarFlag("REPORTERROR","email")
+server = e.data.getVarFlag("REPORTERROR","server")
+
+if user is not None:
+cmd_s += " -n %s" % user
+if email is not None:
+cmd_s += " -e %s" % email
+if server is not None:
+cmd_s += " -s %s" % server
+
+if autosend is not None:
+if autosend == "yes":
+response_s += "The errors for this build are set to 
automatically be sent.\n"
+cmd_s += " -y"
+sendit = True
+elif autosend == "ask":
+response_s += "The errors for this build are set to 
automatically be sent.\n"
+sendit = True
+elif autosend is None:
+response_s += "You can send the errors to a reports server 
by running:\n  %s\n" % cmd_s
+elif e.data.getVarFlags('REPORTERROR') is None:


The last elif will never be executed.


+response_s += "You can send the errors to a reports server 
by running:\n  %s [-s server]\n" % cmd_s
+response_s += "The contents of these logs will be 
posted in public if you use the above command with the default server.\n \
+ Please ensure you remove any identifying or 
proprietary information when prompted before sending.\n"
+
+bb.note(response_s)
+
+if sendit == True:
+import shlex, subprocess
+args = shlex.split(cmd_s)
+subprocess.call(args)
 }

 addhandler errorreport_handler



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] sstate.bbclass: Don't create symlinks, download to the correct location

[Randy Witt]

Previously the sstate was all downloaded to the same directory and then
symlinks were added in the directories that pointed to the siginfo and
sstate in the parent directory.

This change makes it so that now the files are just downloaded to the
correct location without the need for symlinks.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/sstate.bbclass | 12 +++-
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 22ce1ee..6bf94b0 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -623,10 +623,10 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 
 # Try a fetch from the sstate mirror, if it fails just return and
 # we will build the package
-uris = ['file://{0}'.format(sstatefetch),
-'file://{0}.siginfo'.format(sstatefetch)]
+uris = ['file://{0};downloadfilename={0}'.format(sstatefetch),
+
'file://{0}.siginfo;downloadfilename={0}.siginfo'.format(sstatefetch)]
 if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG", True), False):
-uris += ['file://{0}.sig'.format(sstatefetch)]
+uris += ['file://{0}.sig;downloadfilename={0}.sig'.format(sstatefetch)]
 
 for srcuri in uris:
 localdata.setVar('SRC_URI', srcuri)
@@ -634,12 +634,6 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
 fetcher.download()
 
-# Need to optimise this, if using file:// urls, the fetcher just 
changes the local path
-# For now work around by symlinking
-localpath = bb.data.expand(fetcher.localpath(srcuri), localdata)
-if localpath != sstatepkg and os.path.exists(localpath) and not 
os.path.exists(sstatepkg):
-os.symlink(localpath, sstatepkg)
-
 except bb.fetch2.BBFetchException:
 break
 
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/1] sstate fetch symlink removal

[Randy Witt]

This change removes the clutter of symlinks when sstate comes from a mirror
rather than being created on the host.

The following changes since commit c7e614c438706fb3ed7520b4990ebb3973366942:

  useradd: Fix infinite build loop (2016-05-23 10:33:45 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/sstate-symlink
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/sstate-symlink

Randy Witt (1):
  sstate.bbclass: Don't create symlinks, download to the correct
location

 meta/classes/sstate.bbclass | 12 +++-
 1 file changed, 3 insertions(+), 9 deletions(-)

-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/1] sstate.bbclass: Don't create symlinks, download to the correct location

[Randy Witt]

Previously the sstate was all downloaded to the same directory and then
symlinks were added in the directories that pointed to the siginfo and
sstate in the parent directory.

This change makes it so that now the files are just downloaded to the
correct location without the need for symlinks.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/sstate.bbclass | 12 +++-
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 22ce1ee..6d5411f 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -623,10 +623,10 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 
 # Try a fetch from the sstate mirror, if it fails just return and
 # we will build the package
-uris = ['file://{0}'.format(sstatefetch),
-'file://{0}.siginfo'.format(sstatefetch)]
+uris = ['file://{0};downloadfilename={1}'.format(sstatefetch, sstatefetch),
+
'file://{0}.siginfo;downloadfilename={1}.siginfo'.format(sstatefetch, 
sstatefetch)]
 if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG", True), False):
-uris += ['file://{0}.sig'.format(sstatefetch)]
+uris += ['file://{0}.sig;downloadfilename={1}.sig'.format(sstatefetch, 
sstatefetch)]
 
 for srcuri in uris:
 localdata.setVar('SRC_URI', srcuri)
@@ -634,12 +634,6 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
 fetcher.download()
 
-# Need to optimise this, if using file:// urls, the fetcher just 
changes the local path
-# For now work around by symlinking
-localpath = bb.data.expand(fetcher.localpath(srcuri), localdata)
-if localpath != sstatepkg and os.path.exists(localpath) and not 
os.path.exists(sstatepkg):
-os.symlink(localpath, sstatepkg)
-
 except bb.fetch2.BBFetchException:
 break
 
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Make runqemu knows nothing about machine

[Randy Witt]

On 04/29/2016 03:09 AM, Robert Yang wrote:



On 04/29/2016 05:45 PM, Richard Purdie wrote:

On Tue, 2016-04-26 at 11:42 +0800, Robert Yang wrote:

Hello,

The qemu-native can boot a lot of machines, but oe-core's runqemu can
only
boot a few of them which are hardcoded into runqemu. I'd like to
change
it little to make it drop the hardcode and can boot more machines.
Here
are some basic thoughts, please feel free to give your comments.

runqemu is a helpful script which can help us boot images easily, but
it
has a lot of hard code for machine + args. I'd like to remove these
from
runqemu, and make it as a frame. The logical is that, who knows
clearly
about whether qemu can boot the machine and how to boot it (args),
the
answer is the machine/bsp developer, so we can:

* Add a var like QEMU_SUPPORTED = "yes/no" in the bsp conf file
(default to no)
* Add a var like QEMU_BOOT_ARGS = "foo" if there are special args.
* Let do_rootfs or do_image_foo write data such as QEMU_BOOT_ARGS to
DEPLOY_DIR_IMAGE/runqemu/ or tmp/deploy/images/runqemu/, we can
treat
the "runqemu/" dir as a database, and anything we need there, for
example,
efi/pcbios, root args, and so on. We won't miss anything since all
the
images which can be boot by runqemu are built by oe-core.
* Then we can easily add supported machine to runqemu from the bsp
itself
without change runqemu.

I will start working on it if there is no objections, and make sure
it won't
break any current supported machines.


I guess we have some decisions to make. Firstly, is that script better
in shell or python at this point? Originally, shell made sense but I'm
starting to wonder if it would be clearer in python.

If we left it as shell, I'd envisaged it going through something like
BBPATH and sourcing files from each of the layers to build up its own
list of supported machines. The trouble is we can't easily get BBPATH
in the shell environment since we can't assume bitbake and we ideally
want it to work outside of our build environment.


Thanks for the reply, my thought is that move part of the implementation
into a class called runqemu.bbclass (or other names), and leave the runqemu
as a shell script, when QEMU_BOOT_SUPPORTED = "1", runqemu.bbclass will
create a:
tmp/deploy/images//runqemu

And the content in runqemu can be: (for example, arm)
QEMU="/path/to/sysroots/x86_64-linux/usr/bin/qemu-system-arm -m 128 -machine
versatilepb  "
TUNE_ARCH="arm"
QEMU_DEFAULT_KERNEL="zImage-qemuarm.bin"
QEMU_DEFAULT_FSTYPE="ext4"
QEMU_MEM="-m 128"
[maybe more]


Would it be better to just have a meta-qemu-config-[whateverarch]-native that 
publishes the config file? Then it could also publish the config in different 
forms such as  json/yaml/ini for users who don't want to use runqemu, but still 
need to know the *required* args. They might want to ignore the rest.



All of these variables can be defined in bsp layer or conf files. I will
add some default values there according to TUNE_ARCH, so that the bsp may
not need do anything to boot from qemu when QEMU_BOOT_SUPPORTED = "1".

// Robert



With the development of devtool, I'm wondering if runqemu should become
another such "command", written in python and working in any of the
environments we'd expect devtool to work (eSDK, main build). If we did
that, we'd have the layers and could easily access BBPATH and search
for additional modules to add support for additional machines.

Thoughts?

Cheers,

Richard





--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] devtool: Fix build-sdk when pn doesn't match filename

[Randy Witt]

If an image with the filename foo.bb could be built using the name "bar"
instead, then build-sdk would fail to create the derivative sdk.

This was because the code assumed that the file name matched the target,
which is not necessarily the case.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 scripts/devtool|  3 +++
 scripts/lib/devtool/build_image.py | 16 
 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index 4780390..f090427 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -86,6 +86,9 @@ class ConfigHandler(object):
 with open(self.config_file, 'w') as f:
 self.config_obj.write(f)
 
+def set(self, section, option, value):
+self.config_obj.set(section, option, value)
+
 class Context:
 def __init__(self, **kwargs):
 self.__dict__.update(kwargs)
diff --git a/scripts/lib/devtool/build_image.py 
b/scripts/lib/devtool/build_image.py
index e51d766..0f8ca93 100644
--- a/scripts/lib/devtool/build_image.py
+++ b/scripts/lib/devtool/build_image.py
@@ -72,12 +72,12 @@ def build_image(args, config, basepath, workspace):
 return result
 
 def build_image_task(config, basepath, workspace, image, add_packages=None, 
task=None, extra_append=None):
-appendfile = os.path.join(config.workspace_path, 'appends',
-  '%s.bbappend' % image)
-
 # remove .bbappend to make sure setup_tinfoil doesn't
 # break because of it
-if os.path.isfile(appendfile):
+target_basename = config.get('SDK', 'target_basename', '')
+if target_basename:
+appendfile = os.path.join(config.workspace_path, 'appends',
+  '%s.bbappend' % target_basename)
 os.unlink(appendfile)
 
 tinfoil = setup_tinfoil(basepath=basepath)
@@ -88,6 +88,14 @@ def build_image_task(config, basepath, workspace, image, 
add_packages=None, task
 if not bb.data.inherits_class('image', rd):
 raise TargetNotImageError()
 
+# Get the actual filename used and strip the .bb and full path
+target_basename = rd.getVar('FILE', True)
+target_basename = os.path.basename(target_basename).rsplit('.', 1)[0]
+config.set('SDK', 'target_basename', target_basename)
+
+appendfile = os.path.join(config.workspace_path, 'appends',
+  '%s.bbappend' % target_basename)
+
 outputdir = None
 try:
 if workspace or add_packages:
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] populate_sdk_ext: Change lockedsigs task mismatch to a warning

[Randy Witt]

It has been determined that it is highly likely that users might get
signatures that don't match in an extensible sdk. This doesn't
necessarily happen with oe-core, so we can set the mismatch to an error
during testing if we like.

However, for the case where users are creating their own sdks, we don't
need an error halting their progress. locked-sigs will still function as
it should.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 87518d1..935d4e5 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -221,7 +221,7 @@ python copy_buildsystem () {
 
 # Error if the sigs in the locked-signature file don't match
 # the sig computed from the metadata.
-f.write('SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "error"\n\n')
+f.write('SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n\n')
 
 # Hide the config information from bitbake output (since it's 
fixed within the SDK)
 f.write('BUILDCFG_HEADER = ""\n')
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/2] extensible sdk fixes

[Randy Witt]

Fixes for the building a derivative sdk from the extensible sdk.

The following changes since commit f7b520878babbaa7527151f22c031ae160512753:

  poky: Switch to post release name/version (2016-04-29 07:58:46 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/esdk-fixes
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/esdk-fixes

Randy Witt (2):
  populate_sdk_ext: Change lockedsigs task mismatch to a warning
  devtool: Fix build-sdk when pn doesn't match filename

 meta/classes/populate_sdk_ext.bbclass |  2 +-
 scripts/devtool   |  3 +++
 scripts/lib/devtool/build_image.py| 16 
 3 files changed, 16 insertions(+), 5 deletions(-)

-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/5] populate_sdk_ext.bbclass: Enable locked sigs errors

[Randy Witt]

With the extensible sdk we want there to be an error if a task tries to
run without signatures that match locked-sigs.inc. This patch enables
that error.

[YOCTO #9195]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 4ad190b..d1977a7 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -209,8 +209,15 @@ python copy_buildsystem () {
 # Bypass the default connectivity check if any
 f.write('CONNECTIVITY_CHECK_URIS = ""\n\n')
 
-# Ensure locked sstate cache objects are re-used without error
-f.write('SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "none"\n\n')
+# This warning will come out if reverse dependencies for a task
+# don't have sstate as well as the task itself. We already know
+# this will be the case for the extensible sdk, so turn off the
+# warning.
+f.write('SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"\n\n')
+
+# Error if the sigs in the locked-signature file don't match
+# the sig computed from the metadata.
+f.write('SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "error"\n\n')
 
 # Hide the config information from bitbake output (since it's 
fixed within the SDK)
 f.write('BUILDCFG_HEADER = ""\n')
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 4/5] sstatesig.py: Add a method to "unlock" recipes

[Randy Witt]

In order to support workflows using devtool where a user might want to
modify tasks that exist in locked-sigs.inc, there must be a way to unlock
recipes.

This patch adds that support by allowing the user to add recipes to
SIGGEN_UNLOCKED_RECIPES. Recipes that exist in that variable will have
all their tasks unlocked, as well as any tasks that depend on that
recipe.

For example if foo->bar->baz, if you unlock baz, it will also unlock bar
so that foo can be rebuilt without explicitly specifying bar as being
unlocked.

[YOCTO #9195]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/lib/oe/sstatesig.py | 24 +++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 009adea..01dce66 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -94,6 +94,9 @@ class 
SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
 self.lockedhashfn = {}
 self.machine = data.getVar("MACHINE", True)
 self.mismatch_msgs = []
+self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES", True) or
+"").split()
+self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
 pass
 
 def tasks_resolved(self, virtmap, virtpnmap, dataCache):
@@ -136,7 +139,26 @@ class 
SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
 recipename = dataCache.pkg_fn[fn]
 self.lockedpnmap[fn] = recipename
 self.lockedhashfn[fn] = dataCache.hashfn[fn]
-if recipename in self.lockedsigs:
+
+unlocked = False
+if recipename in self.unlockedrecipes:
+unlocked = True
+else:
+def recipename_from_dep(dep):
+# The dep entry will look something like
+# /path/path/recipename.bb.task, virtual:native:/p/foo.bb.task,
+# ...
+fn = dep.rsplit('.', 1)[0]
+return dataCache.pkg_fn[fn]
+
+# If any unlocked recipe is in the direct dependencies then the
+# current recipe should be unlocked as well.
+depnames = [ recipename_from_dep(x) for x in deps ]
+if any(x in y for y in depnames for x in self.unlockedrecipes):
+self.unlockedrecipes[recipename] = ''
+unlocked = True
+
+if not unlocked and recipename in self.lockedsigs:
 if task in self.lockedsigs[recipename]:
 k = fn + "." + task
 h_locked = self.lockedsigs[recipename][task][0]
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 5/5] devtool: Create unlocked-sigs.inc containing items in the workspace

[Randy Witt]

When a recipe is added to the workspace, the signatures for the tasks
will change. This means that bitbake must be told to allow the
signatures to be different if they are in locked-sigs.inc.

This is done by creating an unlocked-sigs.inc file which contains all
the recipes in the workspace each time devtool reads the workspace.

So not only will necessary things get added, previously added items will
be removed by virtue of them no longer being in the workspace.

This also makes sure that the extensible sdk picks up unlocked-sigs.inc
as part of the configuration.

[YOCTO #9195]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  5 +
 scripts/devtool   | 20 
 2 files changed, 25 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index d1977a7..87518d1 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -140,6 +140,10 @@ python copy_buildsystem () {
 with open(os.path.join(baseoutpath, 'conf', 'devtool.conf'), 'w') as f:
 config.write(f)
 
+unlockedsigs =  os.path.join(baseoutpath, 'conf', 'unlocked-sigs.inc')
+with open(unlockedsigs, 'w') as f:
+pass
+
 # Create a layer for new recipes / appends
 bbpath = d.getVar('BBPATH', True)
 bb.process.run(['devtool', '--bbpath', bbpath, '--basepath', baseoutpath, 
'create-workspace', '--create-only', os.path.join(baseoutpath, 'workspace')])
@@ -238,6 +242,7 @@ python copy_buildsystem () {
 f.write(line.strip() + '\n')
 
 f.write('require conf/locked-sigs.inc\n')
+f.write('require conf/unlocked-sigs.inc\n')
 
 if os.path.exists(builddir + '/conf/auto.conf'):
 if derivative:
diff --git a/scripts/devtool b/scripts/devtool
index e1198b1..4780390 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -125,6 +125,25 @@ def read_workspace():
  'recipefile': recipefile}
 logger.debug('Found recipe %s' % workspace[pn])
 
+def create_unlockedsigs():
+""" This function will make unlocked-sigs.inc match the recipes in the
+workspace. This runs on every run of devtool, but it lets us ensure
+the unlocked items are in sync with the workspace. """
+
+confdir = os.path.join(basepath, 'conf')
+unlockedsigs = os.path.join(confdir, 'unlocked-sigs.inc')
+bb.utils.mkdirhier(confdir)
+with open(os.path.join(confdir, 'unlocked-sigs.inc'), 'w') as f:
+f.write("# DO NOT MODIFY! YOUR CHANGES WILL BE LOST.\n" +
+"# This layer was created by the OpenEmbedded devtool" +
+" utility in order to\n" +
+"# contain recipes that are unlocked.\n")
+
+f.write('SIGGEN_UNLOCKED_RECIPES += "\\\n')
+for pn in workspace:
+f.write('' + pn)
+f.write('"')
+
 def create_workspace(args, config, basepath, workspace):
 if args.layerpath:
 workspacedir = os.path.abspath(args.layerpath)
@@ -299,6 +318,7 @@ def main():
 
 if not getattr(args, 'no_workspace', False):
 read_workspace()
+create_unlockedsigs()
 
 try:
 ret = args.func(args, config, basepath, workspace)
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/5] sstatesig.py: Improve the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK message

[Randy Witt]

The previous message when signatures didn't match between the metadata
and the locked signatures file, the message output was a bit confusing.

Now the message should be of the form:

The zlib-native:do_install sig is computed to be
53531910a2a7848432da89def942a91a, but the sig is locked to
d25ba9035f7ccb308e51bbe1066e8d27 in SIGGEN_LOCKEDSIGS_t-x86-64

which will hopefully be more useful in understanding the problem.

[YOCTO #9195]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/lib/oe/sstatesig.py | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index b2319ff..009adea 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -65,12 +65,13 @@ def sstate_lockedsigs(d):
 sigs = {}
 types = (d.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split()
 for t in types:
-lockedsigs = (d.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split()
+siggen_lockedsigs_var = "SIGGEN_LOCKEDSIGS_%s" % t
+lockedsigs = (d.getVar(siggen_lockedsigs_var, True) or "").split()
 for ls in lockedsigs:
 pn, task, h = ls.split(":", 2)
 if pn not in sigs:
 sigs[pn] = {}
-sigs[pn][task] = h
+sigs[pn][task] = [h, siggen_lockedsigs_var]
 return sigs
 
 class SignatureGeneratorOEBasic(bb.siggen.SignatureGeneratorBasic):
@@ -138,14 +139,15 @@ class 
SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
 if recipename in self.lockedsigs:
 if task in self.lockedsigs[recipename]:
 k = fn + "." + task
-h_locked = self.lockedsigs[recipename][task]
+h_locked = self.lockedsigs[recipename][task][0]
+var = self.lockedsigs[recipename][task][1]
 self.lockedhashes[k] = h_locked
 self.taskhash[k] = h_locked
 #bb.warn("Using %s %s %s" % (recipename, task, h))
 
 if h != h_locked:
-self.mismatch_msgs.append('The %s:%s sig (%s) changed, use 
locked sig %s to instead'
-  % (recipename, task, h, h_locked))
+self.mismatch_msgs.append('The %s:%s sig is computed to be 
%s, but the sig is locked to %s in %s'
+  % (recipename, task, h, h_locked, 
var))
 
 return h_locked
 #bb.warn("%s %s %s" % (recipename, task, h))
-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/5] sstatesig.py: Split single locked sigs check into multiple checks

[Randy Witt]

Add the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK and
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK variables to replace
SIGGEN_LOCKEDSIGS_CHECK_LEVEL.

SIGGEN_LOCKEDSIGS_TASKSIG_CHECK will no control whether there is a
warning or error if a task's hash in the locked signature file doesn't match
the computed hash from the current metadata.

SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK will control whther there is a
warning or error if a task that supports sstate is in the locked
signature file, but no sstate exists for the task.

Previously you could only have warning/errors for both controlled by
SIGGEN_LOCKEDSIGS_CHECK_LEVEL. This was an issue in the extensible sdk,
because we know sstate won't exist for certain items in the reverse
dependencies list for tasks. However, we still want to error if task
signatures don't match.

[YOCTO #9195]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/sstate.bbclass   |  8 +++-
 meta/lib/oe/sstatesig.py  | 27 +--
 meta/lib/oeqa/selftest/signing.py |  4 ++--
 3 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 3234e79..8c62327 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -53,7 +53,13 @@ SSTATEPOSTINSTFUNCS = ""
 EXTRA_STAGING_FIXMES ?= ""
 SSTATECLEANFUNCS = ""
 
-SIGGEN_LOCKEDSIGS_CHECK_LEVEL ?= 'error'
+# Check whether sstate exists for tasks that support sstate and are in the
+# locked signatures file.
+SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK ?= 'error'
+
+# Check whether the task's computed hash matches the task's hash in the
+# locked signatures file.
+SIGGEN_LOCKEDSIGS_TASKSIG_CHECK ?= "error"
 
 # The GnuPG key ID and passphrase to use to sign sstate archives (or unset to
 # not sign)
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 5828a9d..b2319ff 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -189,20 +189,35 @@ class 
SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
 f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " 
".join(types.keys(
 
 def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d):
-checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True)
+warn_msgs = []
+error_msgs = []
+sstate_missing_msgs = []
+
 for task in range(len(sq_fn)):
 if task not in ret:
 for pn in self.lockedsigs:
 if sq_hash[task] in self.lockedsigs[pn].itervalues():
 if sq_task[task] == 'do_shared_workdir':
 continue
-self.mismatch_msgs.append("Locked sig is set for %s:%s 
(%s) yet not in sstate cache?"
+sstate_missing_msgs.append("Locked sig is set for 
%s:%s (%s) yet not in sstate cache?"
% (pn, sq_task[task], 
sq_hash[task]))
 
-if self.mismatch_msgs and checklevel == 'warn':
-bb.warn("\n".join(self.mismatch_msgs))
-elif self.mismatch_msgs and checklevel == 'error':
-bb.fatal("\n".join(self.mismatch_msgs))
+checklevel = d.getVar("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK", True)
+if checklevel == 'warn':
+warn_msgs += self.mismatch_msgs
+elif checklevel == 'error':
+error_msgs += self.mismatch_msgs
+
+checklevel = d.getVar("SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK", True)
+if checklevel == 'warn':
+warn_msgs += sstate_missing_msgs
+elif checklevel == 'error':
+error_msgs += sstate_missing_msgs
+
+if warn_msgs:
+bb.warn("\n".join(warn_msgs))
+if error_msgs:
+bb.fatal("\n".join(error_msgs))
 
 
 # Insert these classes into siggen's namespace so it can see and select them
diff --git a/meta/lib/oeqa/selftest/signing.py 
b/meta/lib/oeqa/selftest/signing.py
index d2b3f00..1babca0 100644
--- a/meta/lib/oeqa/selftest/signing.py
+++ b/meta/lib/oeqa/selftest/signing.py
@@ -160,7 +160,7 @@ class LockedSignatures(oeSelfTest):
 bitbake('-S none %s' % test_recipe)
 
 feature = 'require %s\n' % locked_sigs_file
-feature += 'SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"\n'
+feature += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n'
 self.write_config(feature)
 
 # Build a locked recipe
@@ -180,7 +180,7 @@ class LockedSignatures(oeSelfTest):
 ret = bitbake(test_recipe)
 
 # Verify you get the warning and that the real task *isn't* run (i.e. 
the locked signature has worked)
-patt = r'WARNING: The %s:do_package sig \S+ changed, use locked sig 
\S+ to instead' % test_recipe
+patt

[OE-core] [PATCH 0/5] extensible sdk changes

[Randy Witt]

This set of changes is to make sure the locked signatures are enforced as
part of the extensible sdk usage.

Therefore it also allows for the unlocking of recipes that are part of the
devtool workspace.

The following changes since commit 14e2b90893f57570c2002317b5afb0e61d15e9ca:

  toasterconf.json: Add DL_DIR and SSTATE_DIR to poky toasterconf (2016-04-07 
14:58:10 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib sigs
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=sigs

Randy Witt (5):
  sstatesig.py: Split single locked sigs check into multiple checks
  sstatesig.py: Improve the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK message
  populate_sdk_ext.bbclass: Enable locked sigs errors
  sstatesig.py: Add a method to "unlock" recipes
  devtool: Create unlocked-sigs.inc containing items in the workspace

 meta/classes/populate_sdk_ext.bbclass | 16 +++--
 meta/classes/sstate.bbclass   |  8 -
 meta/lib/oe/sstatesig.py  | 63 ---
 meta/lib/oeqa/selftest/signing.py |  4 +--
 scripts/devtool   | 20 +++
 5 files changed, 94 insertions(+), 17 deletions(-)

-- 
2.5.5

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] toaster: add DL_DIR and SSTATE_DIR to oe toasterconf

[Randy Witt]

On 04/06/2016 10:00 AM, Michael Wood wrote:

From: Ed Bartosh 


Shouldn't the message from the cover letter actually be the commit message here, 
rather than an empty message?




Signed-off-by: Ed Bartosh 
Signed-off-by: Michael Wood 
---
  meta/conf/toasterconf.json | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/conf/toasterconf.json b/meta/conf/toasterconf.json
index 1987793..bbd40e9 100644
--- a/meta/conf/toasterconf.json
+++ b/meta/conf/toasterconf.json
@@ -2,9 +2,11 @@
  "config": {
  "MACHINE"  : "qemux86",
  "DISTRO"   : "poky",
+"DL_DIR"   : "${TOPDIR}/../downloads",
  "IMAGE_FSTYPES": "ext3 jffs2 tar.bz2",
  "IMAGE_INSTALL_append": "",
-"PACKAGE_CLASSES": "package_rpm"
+"PACKAGE_CLASSES": "package_rpm",
+"SSTATE_DIR"   : "${TOPDIR}/../sstate-cache"
  },
  "layersources": [
  {



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] scripts:/oe-selftest: Use timestamp instead of test names in coverage data file

[Randy Witt]

On 03/23/2016 05:32 PM, Humberto Ibarra wrote:

This fixes the problem by changing the name to the coverage data file,
using the timestamp as an identifier.


Fixes what problem?



The name for the coverage data file is constructed based on the tests
ran; this has created a couple of issues so far, affecting coverage report.

If --run-tests-by option is given, the data file name won't have any
identifier, causing following runs to overwrite themselves. On the
other hand, if too many tests are given, the file name exceeds linux
limits and fails to store the coverage data all together.

[Yocto #9253]

Signed-off-by: Humberto Ibarra 
---
  scripts/oe-selftest | 13 +
  1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index d18348d..9b61bfa 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -364,16 +364,21 @@ def list_tags():

  print 'Tags:\t%s' % ', '.join(str(x) for x in tags)

-def coverage_setup(run_tests, run_all_tests, coverage_source, 
coverage_include, coverage_omit):
+def coverage_setup(coverage_source, coverage_include, coverage_omit):
  """ Set up the coverage measurement for the testcases to be run """
+import datetime
+import subprocess
  builddir = os.environ.get("BUILDDIR")
+pokydir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+curcommit= subprocess.check_output(["git", "--git-dir", os.path.join(pokydir, ".git"), 
"rev-parse", "HEAD"])
  coveragerc = "%s/.coveragerc" % builddir
  data_file = "%s/.coverage." % builddir
-data_file += ((run_tests and '.'.join(run_tests)) or
-(run_all_tests and "all_tests") or "")
+data_file += datetime.datetime.now().strftime('%Y%m%dT%H%M%S')
  if os.path.isfile(data_file):
  os.remove(data_file)
  with open(coveragerc, 'w') as cps:
+cps.write("# Generated with command '%s'\n" % " ".join(sys.argv))
+cps.write("# HEAD commit %s\n" % curcommit.strip())
  cps.write("[run]\n")
  cps.write("data_file = %s\n" % data_file)
  cps.write("branch = True\n")
@@ -586,7 +591,7 @@ def buildResultClass(args):
  # value indicates where the coverage configuration file 
resides
  # More info on https://pypi.python.org/pypi/coverage
  if not os.environ.get('COVERAGE_PROCESS_START'):
-os.environ['COVERAGE_PROCESS_START'] = 
coverage_setup(args.run_tests, args.run_all_tests, args.coverage_source, 
args.coverage_include, args.coverage_omit)
+os.environ['COVERAGE_PROCESS_START'] = 
coverage_setup(args.coverage_source, args.coverage_include, args.coverage_omit)

  # Use default site.USER_SITE and write corresponding config 
file
  site.ENABLE_USER_SITE = True



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] mtools: Make sure mcopy works when using uninative

[Randy Witt]

uninative changes the location of the IBM850 codepage used by mcopy. So
make sure to set GCONV_PATH to the correct location when using
uninative.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/recipes-devtools/mtools/mtools_4.0.18.bb | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/mtools/mtools_4.0.18.bb 
b/meta/recipes-devtools/mtools/mtools_4.0.18.bb
index 479fd32..691539b 100644
--- a/meta/recipes-devtools/mtools/mtools_4.0.18.bb
+++ b/meta/recipes-devtools/mtools/mtools_4.0.18.bb
@@ -50,6 +50,12 @@ do_install_prepend () {
 }
 
 do_install_append_class-native () {
-create_wrapper ${D}${bindir}/mcopy \
+if  [ "${UNINATIVE_LIBDIR}x" != "x" ]; then
+GCONV_PATH=${UNINATIVE_LIBDIR}/gconv
+else
 GCONV_PATH=${libdir}/gconv
+fi
+
+create_wrapper ${D}${bindir}/mcopy \
+GCONV_PATH=$GCONV_PATH
 }
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/2] mtools uninative fix

[Randy Witt]

These patchset is against master next to fix previous uninative patches. They
most likely will not apply cleanly to master.

The following changes since commit dd359830ad267f9763e9c35493b2846fd2269234:

  poky: Enable uninative (2016-03-04 17:15:56 +)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib 
b82b129692a4eb3c216901cd57d0fc51b0715a9a
  
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=b82b129692a4eb3c216901cd57d0fc51b0715a9a

Randy Witt (2):
  uninative.bbclass: Set UNINATIVE_LIBDIR if uninative is enabled
  mtools: Make sure mcopy works when using uninative

 meta/classes/uninative.bbclass| 5 -
 meta/recipes-devtools/mtools/mtools_4.0.18.bb | 8 +++-
 2 files changed, 11 insertions(+), 2 deletions(-)

-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] uninative.bbclass: Set UNINATIVE_LIBDIR if uninative is enabled

[Randy Witt]

Set a variable so other metadata can easily tell where uninative is
located.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/uninative.bbclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 7f242de..99dee20 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -80,9 +80,12 @@ python uninative_event_enable() {
 loader = d.getVar("UNINATIVE_LOADER", True)
 if os.path.exists(loader):
 bb.debug(2, "Enabling uninative")
+
+uninative_dir = "${STAGING_DIR}-uninative/${BUILD_ARCH}-linux"
+d.setVar("UNINATIVE_LIBDIR", "%s${libdir_native}" % uninative_dir)
 d.setVar("NATIVELSBSTRING", "universal")
 d.appendVar("SSTATEPOSTUNPACKFUNCS", " uninative_changeinterp")
-d.prependVar("PATH", 
"${STAGING_DIR}-uninative/${BUILD_ARCH}-linux${bindir_native}:")
+d.prependVar("PATH", "%s${bindir_native}:" % uninative_dir)
 }
 
 python uninative_changeinterp () {
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_ext: Make populate_sdk_ext nostamp

[Randy Witt]

Since the metadata for multiple layers will be put into the sdk, anytime
those layers change the sdk needs to be rebuilt. Ideally, this would use
checksums to only rebuild when necessary.

However, since this feature needs to be in the release, opt for a less
intrusive change by setting the task to nostamp. Unnecessary rebuilds
may occur, but it's better than a user wondering why their changes
didn't get added.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 77de969..e93be29 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -414,8 +414,9 @@ do_populate_sdk_ext[rdepends] += "${@' '.join([x + 
':do_build' for x in d.getVar
 do_populate_sdk_ext[vardeps] += "copy_buildsystem \
  sdk_ext_postinst"
 
-do_populate_sdk_ext[file-checksums] += 
"${COREBASE}/meta/files/toolchain-shar-relocate.sh:True \
-
${COREBASE}/meta/files/toolchain-shar-extract.sh:True \
-
${COREBASE}/scripts/gen-lockedsig-cache:True"
+# Since any change in the metadata of any layer should cause a rebuild of the
+# sdk(since the layers are put in the sdk) set the task to nostamp so it
+# always runs.
+do_populate_sdk_ext[nostamp] = "1"
 
 addtask populate_sdk_ext after do_sdk_depends
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_ext: Only write LCONF_VERSION to bblayers if it is set

[Randy Witt]

It is possible that LCONF_VERSION won't be set, such as if meta-poky is
used. Without this change, bblayers.conf would have LCONF_VERSION =
"None" if LCONF_VERSION wasn't set, which would cause a sanity check
failure.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 9e2fc61..674e1af 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -143,7 +143,12 @@ python copy_buildsystem () {
 f.write('# this configuration provides, it is strongly suggested that 
you set\n')
 f.write('# up a proper instance of the full build system and use that 
instead.\n\n')
 
-f.write('LCONF_VERSION = "%s"\n\n' % d.getVar('LCONF_VERSION', False))
+# LCONF_VERSION may not be set, for example when using meta-poky
+# so don't error if it isn't found
+lconf_version = d.getVar('LCONF_VERSION', False)
+if lconf_version is not None:
+f.write('LCONF_VERSION = "%s"\n\n' % lconf_version)
+
 f.write('BBPATH = "$' + '{TOPDIR}"\n')
 f.write('SDKBASEMETAPATH = "$' + '{TOPDIR}"\n')
 f.write('BBLAYERS := " \\\n')
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_ext: Delete the buildtools tar file after installation

[Randy Witt]

When installing the ext sdk, buildtools is extracted and installed as
well. The tar file containing buildtools isn't used after installation
so was wasted space and clutter.

[YOCTO #9172]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 9e2fc61..f927cb9 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -313,6 +313,9 @@ sdk_ext_postinst() {
cd $target_sdk_dir
printf "buildtools\ny" | ./*buildtools-nativesdk-standalone* > 
/dev/null || ( printf 'ERROR: buildtools installation failed\n' ; exit 1 )
 
+   # Delete the buildtools tar file since it won't be used again
+   rm ./*buildtools-nativesdk-standalone*.sh -f
+
# Make sure when the user sets up the environment, they also get
# the buildtools-tarball tools in their path.

env_setup_script="$target_sdk_dir/environment-setup-${REAL_MULTIMACH_TARGET_SYS}"
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_ext: Add images to SDK_INSTALL_TARGETS

[Randy Witt]

When running ext-sdk-prepare.py during sdk installation a check is done to
make sure no tasks would run that aren't provided by the "leaf" recipes
specified in SDK_INSTALL_TARGETS.

However sometimes an image recipe can cause other images to be created
such as an initramfs. So make sure those additional images are
recognized by ext-sdk-prepare.py and don't flag an error.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index a6badb8..9e2fc61 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -40,12 +40,17 @@ def get_sdk_install_targets(d):
 if d.getVar('SDK_EXT_TYPE', True) != 'minimal':
 sdk_install_targets = d.getVar('SDK_TARGETS', True)
 
+depd = d.getVar('BB_TASKDEPDATA', False)
+for v in depd.itervalues():
+if v[1] == 'do_image_complete':
+if v[0] not in sdk_install_targets:
+sdk_install_targets += ' {}'.format(v[0])
+
 if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
 sdk_install_targets += ' meta-world-pkgdata:do_allpackagedata'
 
 return sdk_install_targets
 
-SDK_INSTALL_TARGETS = "${@get_sdk_install_targets(d)}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
 
 # The files from COREBASE that you want preserved in the COREBASE copied
@@ -344,12 +349,15 @@ SDK_POST_INSTALL_COMMAND_task-populate-sdk-ext = 
"${sdk_ext_postinst}"
 
 SDK_POSTPROCESS_COMMAND_prepend_task-populate-sdk-ext = "copy_buildsystem; 
install_tools; "
 
+SDK_INSTALL_TARGETS = ""
 fakeroot python do_populate_sdk_ext() {
 # FIXME hopefully we can remove this restriction at some point, but 
uninative
 # currently forces this upon us
 if d.getVar('SDK_ARCH', True) != d.getVar('BUILD_ARCH', True):
 bb.fatal('The extensible SDK can currently only be built for the same 
architecture as the machine being built on - SDK_ARCH is set to %s (likely via 
setting SDKMACHINE) which is different from the architecture of the build 
machine (%s). Unable to continue.' % (d.getVar('SDK_ARCH', True), 
d.getVar('BUILD_ARCH', True)))
 
+d.setVar('SDK_INSTALL_TARGETS', get_sdk_install_targets(d))
+
 bb.build.exec_func("do_populate_sdk", d)
 }
 
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/4] classes/testsdk: Move the removal of bitbake PATH to eSDK context only

[Randy Witt]

On 02/22/2016 09:05 AM, Aníbal Limón wrote:



On 02/22/2016 10:48 AM, Burton, Ross wrote:

On 22 February 2016 at 16:37, Aníbal Limón 
wrote:


I agree with you to modify avoid_paths_in_environ for return the new
PATH variable is better than only modify it internally but for
simplicity i will maintain the os.environ['PATH'] set/restore instead of
generate the environment line.



Totally agree with Randy here for what it's worth.  The environment-munging
code in avoid_paths... should return the strings instead of manipulating
the current environment so the caller has the choice whether to modify the
current environment or pass a new environment to subprocess.  And in
general I'd say that passing modified environments to subprocess is a
cleaner solution as it means that there's no way cleanup can fail to
happen.  Whilst that's just a try/except now, the code could get copied and
extended and end up with codepaths that don't hit the right cleanup.  By
having an explicit environment passed in, this isn't possible.


Agree, now modified at,

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=alimon/esdk_update_v2=3143bf09130c52cd71e3f2f9795208e17152005d


If you either convert path to a dict or have avoid_paths_in_environ() return a 
dict you can do:


+output = subprocess.check_output(". %s > /dev/null; %s;" % \
+(self.tc.sdkenv, cmd), env=path, shell=True)

It's not quite obvious in the docs that you can pass that in, they refer you to 
the Popen docs.



Cheers,
alimon



Ross





--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 4/4] oeqa/sdkext: Add sdk_update.SDKUpdateTest class.

[Randy Witt]

On 02/22/2016 08:26 AM, Aníbal Limón wrote:



On 02/22/2016 10:18 AM, Randy Witt wrote:

On 02/22/2016 07:03 AM, Aníbal Limón wrote:

From: Aníbal Limón <limon.ani...@gmail.com>

The SDKUpdateTest class test devtool sdk-update mechanism inside
eSDK.

The SDKUpdateTest class search for new sdk if not found uses
the main one then it publish the eSDK into known folder
inside work and it starts a web server for serve the eSDK.

Finally it executes sdk-update over http, the local test is
commented due to bug [1].

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9043

[YOCTO #9089]

Signed-off-by: Aníbal Limón <limon.ani...@gmail.com>
---
   meta/lib/oeqa/sdkext/sdk_update.py | 39
++
   1 file changed, 39 insertions(+)
   create mode 100644 meta/lib/oeqa/sdkext/sdk_update.py

diff --git a/meta/lib/oeqa/sdkext/sdk_update.py
b/meta/lib/oeqa/sdkext/sdk_update.py
new file mode 100644
index 000..16f5b10
--- /dev/null
+++ b/meta/lib/oeqa/sdkext/sdk_update.py
@@ -0,0 +1,39 @@
+import os
+import shutil
+import subprocess
+
+from oeqa.oetest import oeSDKExtTest
+from oeqa.utils.httpserver import HTTPService
+
+class SdkUpdateTest(oeSDKExtTest):
+
+@classmethod
+def setUpClass(self):
+self.publish_dir = os.path.join(self.tc.sdktestdir,
'esdk_publish')
+if os.path.exists(self.publish_dir):
+shutil.rmtree(self.publish_dir)
+os.mkdir(self.publish_dir)
+
+tcname_new = self.tc.d.expand(
+"${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}-new.sh")
+if not os.path.exists(tcname_new):
+tcname_new = self.tc.tcname
+
+cmd = 'oe-publish-sdk %s %s' % (tcname_new, self.publish_dir)
+subprocess.check_output(cmd, shell=True)
+
+self.http_service = HTTPService(self.publish_dir)
+self.http_service.start()


I think Paul and I briefly mentioned it, but SimpleHTTPServer fails as
an sstate mirror if enough fetchers run. We think it was because of a
limit on the number of simultaneous connections. So I would expect this
to fail for instance if all packages were updated.


I tested running an update and didn't fail this implementation of
oeqa.utils open another process with multiprocessing for serve http only.


My suspicion is because nothing actually updated. And even if something updated, 
unless you hit the max connection limit when pulling from sstate_mirror it would 
be fine.




But really we shouldn't care too much about the sstate updating part,
and more about the layer updating etc. Ideally this test would check to
make sure the local sdk now matches the remote.


I could add a test for the output that i saw displays "Already updated"
or what you mean about match?


devtool sdk-update will check the remote conf/sdk-conf-manifest to see if any 
configuration changed. So for instance the creator of the sdk could add new 
configuration variables, or even items to bblayers.conf.


If the sdk-conf-manifest is changed, the updater should pull down the new layer 
by pulling(cloning if that fails) layers/.git on the remote and moving the 
layers to the appropriate local location. It should also update the local 
configuration to match what is on the remote.


So if you get "Already up-to-date" nothing changed and nothing was updated. To 
test the updating, we need to make sure the configuration actually changes, and 
potentially even add another layer, then update, and then make sure the local 
sdk matches what we think it should based on the remote sdk we published.







+self.http_url = "http://127.0.0.1:%d; % self.http_service.port
+
+def test_sdk_update_http(self):
+output = self._run("devtool sdk-update \"%s\"" % self.http_url)
+
+#def test_sdk_update_local(self):
+#output = self._run("devtool sdk-update \"%s\"" %
self.publish_dir)
+
+@classmethod
+def tearDownClass(self):
+self.http_service.stop()
+shutil.rmtree(self.publish_dir)







--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/4] classes/testsdk: Move the removal of bitbake PATH to eSDK context only

[Randy Witt]

On 02/22/2016 08:09 AM, Aníbal Limón wrote:



On 02/22/2016 09:54 AM, Randy Witt wrote:

On 02/22/2016 07:03 AM, Aníbal Limón wrote:

From: Aníbal Limón <limon.ani...@gmail.com>

The removal of bitbake and scripts PATH is only needed by eSDK tests
so move to eSDK context only.

This also it's a support for eSDK update test because it needs to
execute oe-publish-sdk from scripts.

Signed-off-by: Aníbal Limón <limon.ani...@gmail.com>
---
   meta/classes/testsdk.bbclass |  9 -
   meta/lib/oeqa/oetest.py  | 22 +-
   2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 7e245e9..01d37c4 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -96,11 +96,10 @@ def testsdkext_main(d):
   # extensible sdk use network
   export_proxies(d)

-# extensible sdk shows a warning if found bitbake in the path
-# because can cause problems so clean it
-paths_to_avoid = ['bitbake/bin', 'poky/scripts',
-   d.getVar('STAGING_DIR', True),
-   d.getVar('BASE_WORKDIR', True)]
+# extensible sdk can be contaminated if native programs are
+# in PATH, i.e. use perl-native instead of eSDK one.
+paths_to_avoid = [d.getVar('STAGING_DIR', True),
+  d.getVar('BASE_WORKDIR', True)]
   avoid_paths_in_environ(paths_to_avoid)

   pn = d.getVar("PN", True)
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 3e2ea0f..cd1e7e0 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -21,6 +21,7 @@ import logging
   import oeqa.runtime
   import oeqa.sdkext
   from oeqa.utils.decorators import LogResults, gettag, getResults
+from oeqa.utils import avoid_paths_in_environ

   logger = logging.getLogger("BitBake")

@@ -128,7 +129,26 @@ class oeSDKTest(oeTest):
   return subprocess.check_output(". %s > /dev/null; %s;" %
(self.tc.sdkenv, cmd), shell=True)

   class oeSDKExtTest(oeSDKTest):
-pass
+def _run(self, cmd):
+output = None
+
+paths = os.environ['PATH']
+
+# extensible sdk shows a warning if found bitbake in the path
+# because can cause contamination, i.e. use devtool from
+# poky/scripts instead of eSDK one.
+paths_to_avoid = ['bitbake/bin', 'poky/scripts']
+avoid_paths_in_environ(paths_to_avoid)
+
+try:
+output = subprocess.check_output(". %s > /dev/null; %s;" % \
+(self.tc.sdkenv, cmd), shell=True)


Just pass env to the check_output call, then you won't have to worry
about resetting the parent process' env back to the original value.


Could be but then we need the code for generate the env line so for
practically the result is the same.


Modify avoid_paths_in_environ() to instead return the environment rather than 
modify it as a side effect. Then the code reads more easily and prevents future 
bugs where someone uses avoid_paths_in_environ, but forgets to reset the os.environ.







+except:
+os.environ['PATH'] = paths
+raise
+
+os.environ['PATH'] = paths
+return output

   def getmodule(pos=2):
   # stack returns a list of tuples containg frame information







--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 4/4] oeqa/sdkext: Add sdk_update.SDKUpdateTest class.

[Randy Witt]

On 02/22/2016 07:03 AM, Aníbal Limón wrote:

From: Aníbal Limón 

The SDKUpdateTest class test devtool sdk-update mechanism inside
eSDK.

The SDKUpdateTest class search for new sdk if not found uses
the main one then it publish the eSDK into known folder
inside work and it starts a web server for serve the eSDK.

Finally it executes sdk-update over http, the local test is
commented due to bug [1].

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9043

[YOCTO #9089]

Signed-off-by: Aníbal Limón 
---
  meta/lib/oeqa/sdkext/sdk_update.py | 39 ++
  1 file changed, 39 insertions(+)
  create mode 100644 meta/lib/oeqa/sdkext/sdk_update.py

diff --git a/meta/lib/oeqa/sdkext/sdk_update.py 
b/meta/lib/oeqa/sdkext/sdk_update.py
new file mode 100644
index 000..16f5b10
--- /dev/null
+++ b/meta/lib/oeqa/sdkext/sdk_update.py
@@ -0,0 +1,39 @@
+import os
+import shutil
+import subprocess
+
+from oeqa.oetest import oeSDKExtTest
+from oeqa.utils.httpserver import HTTPService
+
+class SdkUpdateTest(oeSDKExtTest):
+
+@classmethod
+def setUpClass(self):
+self.publish_dir = os.path.join(self.tc.sdktestdir, 'esdk_publish')
+if os.path.exists(self.publish_dir):
+shutil.rmtree(self.publish_dir)
+os.mkdir(self.publish_dir)
+
+tcname_new = self.tc.d.expand(
+"${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}-new.sh")
+if not os.path.exists(tcname_new):
+tcname_new = self.tc.tcname
+
+cmd = 'oe-publish-sdk %s %s' % (tcname_new, self.publish_dir)
+subprocess.check_output(cmd, shell=True)
+
+self.http_service = HTTPService(self.publish_dir)
+self.http_service.start()


I think Paul and I briefly mentioned it, but SimpleHTTPServer fails as an sstate 
mirror if enough fetchers run. We think it was because of a limit on the number 
of simultaneous connections. So I would expect this to fail for instance if all 
packages were updated.


But really we shouldn't care too much about the sstate updating part, and more 
about the layer updating etc. Ideally this test would check to make sure the 
local sdk now matches the remote.



+self.http_url = "http://127.0.0.1:%d; % self.http_service.port
+
+def test_sdk_update_http(self):
+output = self._run("devtool sdk-update \"%s\"" % self.http_url)
+
+#def test_sdk_update_local(self):
+#output = self._run("devtool sdk-update \"%s\"" % self.publish_dir)
+
+@classmethod
+def tearDownClass(self):
+self.http_service.stop()
+shutil.rmtree(self.publish_dir)



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/4] classes/testsdk: Move the removal of bitbake PATH to eSDK context only

[Randy Witt]

On 02/22/2016 07:03 AM, Aníbal Limón wrote:

From: Aníbal Limón 

The removal of bitbake and scripts PATH is only needed by eSDK tests
so move to eSDK context only.

This also it's a support for eSDK update test because it needs to
execute oe-publish-sdk from scripts.

Signed-off-by: Aníbal Limón 
---
  meta/classes/testsdk.bbclass |  9 -
  meta/lib/oeqa/oetest.py  | 22 +-
  2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 7e245e9..01d37c4 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -96,11 +96,10 @@ def testsdkext_main(d):
  # extensible sdk use network
  export_proxies(d)

-# extensible sdk shows a warning if found bitbake in the path
-# because can cause problems so clean it
-paths_to_avoid = ['bitbake/bin', 'poky/scripts',
-   d.getVar('STAGING_DIR', True),
-   d.getVar('BASE_WORKDIR', True)]
+# extensible sdk can be contaminated if native programs are
+# in PATH, i.e. use perl-native instead of eSDK one.
+paths_to_avoid = [d.getVar('STAGING_DIR', True),
+  d.getVar('BASE_WORKDIR', True)]
  avoid_paths_in_environ(paths_to_avoid)

  pn = d.getVar("PN", True)
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 3e2ea0f..cd1e7e0 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -21,6 +21,7 @@ import logging
  import oeqa.runtime
  import oeqa.sdkext
  from oeqa.utils.decorators import LogResults, gettag, getResults
+from oeqa.utils import avoid_paths_in_environ

  logger = logging.getLogger("BitBake")

@@ -128,7 +129,26 @@ class oeSDKTest(oeTest):
  return subprocess.check_output(". %s > /dev/null; %s;" % 
(self.tc.sdkenv, cmd), shell=True)

  class oeSDKExtTest(oeSDKTest):
-pass
+def _run(self, cmd):
+output = None
+
+paths = os.environ['PATH']
+
+# extensible sdk shows a warning if found bitbake in the path
+# because can cause contamination, i.e. use devtool from
+# poky/scripts instead of eSDK one.
+paths_to_avoid = ['bitbake/bin', 'poky/scripts']
+avoid_paths_in_environ(paths_to_avoid)
+
+try:
+output = subprocess.check_output(". %s > /dev/null; %s;" % \
+(self.tc.sdkenv, cmd), shell=True)


Just pass env to the check_output call, then you won't have to worry about 
resetting the parent process' env back to the original value.



+except:
+os.environ['PATH'] = paths
+raise
+
+os.environ['PATH'] = paths
+return output

  def getmodule(pos=2):
  # stack returns a list of tuples containg frame information



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] signing-keys: Make signing keys the only publisher of keys

[Randy Witt]

Previously the keys were put into the os-release package. The package
indexing code was also deploying the keys rather than only using the keys.

This change makes signing-keys.bb the only publisher of the keys and also
uses standard tasks that already have sstate.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/sign_package_feed.bbclass |  9 +
 meta/classes/sign_rpm.bbclass  | 11 --
 meta/lib/oe/package_manager.py | 10 -
 meta/recipes-core/meta/signing-keys.bb | 61 +-
 meta/recipes-core/os-release/os-release.bb | 11 --
 5 files changed, 52 insertions(+), 50 deletions(-)

diff --git a/meta/classes/sign_package_feed.bbclass 
b/meta/classes/sign_package_feed.bbclass
index 63ca02f..e1ec82e 100644
--- a/meta/classes/sign_package_feed.bbclass
+++ b/meta/classes/sign_package_feed.bbclass
@@ -27,12 +27,7 @@ python () {
 for var in ('PACKAGE_FEED_GPG_NAME', 'PACKAGE_FEED_GPG_PASSPHRASE_FILE'):
 if not d.getVar(var, True):
 raise_sanity_error("You need to define %s in the config" % var, d)
-
-# Set expected location of the public key
-d.setVar('PACKAGE_FEED_GPG_PUBKEY',
- os.path.join(d.getVar('STAGING_ETCDIR_NATIVE', False),
-   'PACKAGE-FEED-GPG-PUBKEY'))
 }
 
-do_package_index[depends] += "signing-keys:do_export_public_keys"
-do_rootfs[depends] += "signing-keys:do_export_public_keys"
+do_package_index[depends] += "signing-keys:do_deploy"
+do_rootfs[depends] += "signing-keys:do_populate_sysroot"
diff --git a/meta/classes/sign_rpm.bbclass b/meta/classes/sign_rpm.bbclass
index 8b59bac..c21e3f0 100644
--- a/meta/classes/sign_rpm.bbclass
+++ b/meta/classes/sign_rpm.bbclass
@@ -28,8 +28,11 @@ python () {
 raise_sanity_error("You need to define %s in the config" % var, d)
 
 # Set the expected location of the public key
-d.setVar('RPM_GPG_PUBKEY', os.path.join(d.getVar('STAGING_ETCDIR_NATIVE', 
False),
-'RPM-GPG-PUBKEY'))
+d.setVar('RPM_GPG_PUBKEY', os.path.join(d.getVar('STAGING_DIR_TARGET', 
False),
+d.getVar('sysconfdir', False),
+'pki',
+'rpm-gpg',
+'RPM-GPG-KEY-${DISTRO_VERSION}'))
 }
 
 python sign_rpm () {
@@ -44,5 +47,5 @@ python sign_rpm () {
  d.getVar('RPM_GPG_PASSPHRASE_FILE', True))
 }
 
-do_package_index[depends] += "signing-keys:do_export_public_keys"
-do_rootfs[depends] += "signing-keys:do_export_public_keys"
+do_package_index[depends] += "signing-keys:do_deploy"
+do_rootfs[depends] += "signing-keys:do_populate_sysroot"
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b30a4da..5cd43e9 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -144,16 +144,6 @@ class RpmIndexer(Indexer):
 signer.detach_sign(repomd,
self.d.getVar('PACKAGE_FEED_GPG_NAME', 
True),

self.d.getVar('PACKAGE_FEED_GPG_PASSPHRASE_FILE', True))
-# Copy pubkey(s) to repo
-distro_version = self.d.getVar('DISTRO_VERSION', True) or "oe.0"
-if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
-shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
- os.path.join(self.deploy_dir,
-  'RPM-GPG-KEY-%s' % distro_version))
-if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
-shutil.copy2(self.d.getVar('PACKAGE_FEED_GPG_PUBKEY', True),
- os.path.join(self.deploy_dir,
-  'REPODATA-GPG-KEY-%s' % distro_version))
 
 
 class OpkgIndexer(Indexer):
diff --git a/meta/recipes-core/meta/signing-keys.bb 
b/meta/recipes-core/meta/signing-keys.bb
index d7763c6..1d0e834 100644
--- a/meta/recipes-core/meta/signing-keys.bb
+++ b/meta/recipes-core/meta/signing-keys.bb
@@ -3,37 +3,62 @@
 
 DESCRIPTION = "Make public keys of the signing keys available"
 LICENSE = "MIT"
-PACKAGES = ""
-
-do_fetch[noexec] = "1"
-do_unpack[noexec] = "1"
-do_patch[noexec] = "1"
-do_configure[noexec] = "1"
-do_compile[noexec] = "1"
-do_install[noexec] = "1"
-do_package[noexec] = "1"
-do_packagedata[noexec] = "1"
-do_package_write_ipk[noexec] = "1"
-do_package_write_rpm[noexec] = "1"
-do_package_write_deb[noexec] = "1"
-do_populate_sysroot[noexec] = "1"
+LIC_FILES_CHKSUM = 
"file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \
+
file:

[OE-core] [PATCH v2 3/3] populate_sdk_ext.bbclass: Add SDK_RECRDEP_TASKS variable

[Randy Witt]

Currently there isn't a way for the extensible sdk to know all the tasks
that will need sstate for an image. This is because a layer can add it's
on custom tasks that are required for an image to be generated.

The extensible sdk solved this for poky by using recrdeptask and
specifying the tasks known to be required for the image as well as for
building new recipes.

So the SDK_RECRDEP_TASKS variable allows a user to specify additional
tasks that need to be pulled in.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 7c95301..1c1bb47 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -21,6 +21,8 @@ SDK_EXT_task-populate-sdk-ext = "-ext"
 # Options are full or minimal
 SDK_EXT_TYPE ?= "full"
 
+SDK_RECRDEP_TASKS ?= ""
+
 SDK_LOCAL_CONF_WHITELIST ?= ""
 SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
  BB_NUMBER_THREADS \
@@ -359,7 +361,7 @@ addtask sdk_depends
 do_sdk_depends[dirs] = "${WORKDIR}"
 do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)}"
 do_sdk_depends[recrdeptask] = "${@d.getVarFlag('do_populate_sdk', 
'recrdeptask', False)}"
-do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa 
do_populate_sysroot do_deploy"
+do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa 
do_populate_sysroot do_deploy ${SDK_RECRDEP_TASKS}"
 do_sdk_depends[rdepends] = "${@get_sdk_ext_rdepends(d)}"
 
 def get_sdk_ext_rdepends(d):
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/3] populate_sdk_ext.bbclass: Add SDK_RECRDEP_TASKS variable

[Randy Witt]

On 02/18/2016 02:57 PM, Richard Purdie wrote:

On Thu, 2016-02-18 at 09:17 -0800, Randy Witt wrote:

Currently there isn't a way for the extensible sdk to know all the
tasks
that will need sstate for an image. This is because a layer can add
it's
on custom tasks that are required for an image to be generated.

The extensible sdk solved this for poky by using recrdeptask and
specifying the tasks known to be required for the image as well as
for
building new recipes.

So the SDK_RECRDEP_TASKS variable allows a user to specify additional
tasks that need to be pulled in.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
  meta/classes/populate_sdk_ext.bbclass | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass
b/meta/classes/populate_sdk_ext.bbclass
index 7c95301..5f90aea 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -359,7 +359,7 @@ addtask sdk_depends
  do_sdk_depends[dirs] = "${WORKDIR}"
  do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)}"
  do_sdk_depends[recrdeptask] = "${@d.getVarFlag('do_populate_sdk',
'recrdeptask', False)}"
-do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa
do_populate_sysroot do_deploy"
+do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa
do_populate_sysroot do_deploy ${SDK_RECRDEP_TASKS}"
  do_sdk_depends[rdepends] = "${@get_sdk_ext_rdepends(d)}"

  def get_sdk_ext_rdepends(d):


Doesn't this need a default value or it will throw parse failures?


Actually, it didn't generate parse errors. My guess is it just didn't expand it 
and put a bogus task in the list. So I'll send a v2.




Cheers,

Richard



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/3] devtool: Don't recursively look for .devtoolbase in --basepath

[Randy Witt]

If the user specifies --basepath on the commandline, only the directory
specified should be searched for .devtoolbase. Otherwise when --basepath
is a child of the sdk directory, .devtoolbase will always be found and
devtool will only show options meant to be used within an sdk.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 scripts/devtool | 28 
 1 file changed, 16 insertions(+), 12 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index 2d57da0b..23e9b50 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -192,13 +192,6 @@ def main():
 
 # Default basepath
 basepath = os.path.dirname(os.path.abspath(__file__))
-pth = basepath
-while pth != '' and pth != os.sep:
-if os.path.exists(os.path.join(pth, '.devtoolbase')):
-context.fixed_setup = True
-basepath = pth
-break
-pth = os.path.dirname(pth)
 
 parser = argparse_oe.ArgumentParser(description="OpenEmbedded development 
tool",
 add_help=False,
@@ -224,11 +217,22 @@ def main():
 if global_args.basepath:
 # Override
 basepath = global_args.basepath
-elif not context.fixed_setup:
-basepath = os.environ.get('BUILDDIR')
-if not basepath:
-logger.error("This script can only be run after initialising the 
build environment (e.g. by using oe-init-build-env)")
-sys.exit(1)
+if os.path.exists(os.path.join(basepath, '.devtoolbase')):
+context.fixed_setup = True
+else:
+pth = basepath
+while pth != '' and pth != os.sep:
+if os.path.exists(os.path.join(pth, '.devtoolbase')):
+context.fixed_setup = True
+basepath = pth
+break
+pth = os.path.dirname(pth)
+
+if not context.fixed_setup:
+basepath = os.environ.get('BUILDDIR')
+if not basepath:
+logger.error("This script can only be run after initialising 
the build environment (e.g. by using oe-init-build-env)")
+sys.exit(1)
 
 logger.debug('Using basepath %s' % basepath)
 
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/3] populate_sdk_ext: Don't ignore SDK_TARGETS value

[Randy Witt]

This fixes a problem where SDK_INSTALL_TARGETS wouldn't pick up the
value in SDK_TARGETS. It also removes the inline python to make the
code more readable.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 6e36642..7c95301 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -32,7 +32,18 @@ SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
 SDK_TARGETS ?= "${PN}"
-SDK_INSTALL_TARGETS = "${@SDK_TARGETS if d.getVar('SDK_EXT_TYPE', True) != 
'minimal' else ''} ${@'meta-world-pkgdata:do_allpackagedata' if 
d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''}"
+
+def get_sdk_install_targets(d):
+sdk_install_targets = ''
+if d.getVar('SDK_EXT_TYPE', True) != 'minimal':
+sdk_install_targets = d.getVar('SDK_TARGETS', True)
+
+if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
+sdk_install_targets += ' meta-world-pkgdata:do_allpackagedata'
+
+return sdk_install_targets
+
+SDK_INSTALL_TARGETS = "${@get_sdk_install_targets(d)}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
 
 # The files from COREBASE that you want preserved in the COREBASE copied
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/3] populate_sdk_ext.bbclass: Add SDK_RECRDEP_TASKS variable

[Randy Witt]

Currently there isn't a way for the extensible sdk to know all the tasks
that will need sstate for an image. This is because a layer can add it's
on custom tasks that are required for an image to be generated.

The extensible sdk solved this for poky by using recrdeptask and
specifying the tasks known to be required for the image as well as for
building new recipes.

So the SDK_RECRDEP_TASKS variable allows a user to specify additional
tasks that need to be pulled in.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 7c95301..5f90aea 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -359,7 +359,7 @@ addtask sdk_depends
 do_sdk_depends[dirs] = "${WORKDIR}"
 do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)}"
 do_sdk_depends[recrdeptask] = "${@d.getVarFlag('do_populate_sdk', 
'recrdeptask', False)}"
-do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa 
do_populate_sysroot do_deploy"
+do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa 
do_populate_sysroot do_deploy ${SDK_RECRDEP_TASKS}"
 do_sdk_depends[rdepends] = "${@get_sdk_ext_rdepends(d)}"
 
 def get_sdk_ext_rdepends(d):
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/3] extsdk bug fixes

[Randy Witt]

These are bug fixes for bugs found during installation of the ext sdk.
The following changes since commit bc72f64eb4041ef44198e3e0caeb2be9f43eb44e:

  linux-yocto: Update SRCREV for genericx86* for 4.4 (2016-02-11 22:58:38 +)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib 
941d876fe835bc1aa26039d0edec9573d482bca3
  
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=941d876fe835bc1aa26039d0edec9573d482bca3

Randy Witt (3):
  populate_sdk_ext: Don't ignore SDK_TARGETS value
  devtool: Don't recursively look for .devtoolbase in --basepath
  populate_sdk_ext.bbclass: Add SDK_RECRDEP_TASKS variable

 meta/classes/populate_sdk_ext.bbclass | 15 +--
 scripts/devtool   | 28 
 2 files changed, 29 insertions(+), 14 deletions(-)

-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] signing-keys: Make signing keys the only publisher of keys

[Randy Witt]

Previously the keys were put into the os-release package. The package
indexing code was also deploying the keys rather than only using the keys.

This change makes signing-keys.bb the only publisher of the keys and also
uses standard tasks that already have sstate.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/sign_package_feed.bbclass | 11 --
 meta/classes/sign_rpm.bbclass  | 11 --
 meta/lib/oe/package_manager.py | 10 -
 meta/recipes-core/meta/signing-keys.bb | 61 +-
 meta/recipes-core/os-release/os-release.bb | 11 --
 5 files changed, 57 insertions(+), 47 deletions(-)

diff --git a/meta/classes/sign_package_feed.bbclass 
b/meta/classes/sign_package_feed.bbclass
index 63ca02f..d6d1603 100644
--- a/meta/classes/sign_package_feed.bbclass
+++ b/meta/classes/sign_package_feed.bbclass
@@ -30,9 +30,12 @@ python () {
 
 # Set expected location of the public key
 d.setVar('PACKAGE_FEED_GPG_PUBKEY',
- os.path.join(d.getVar('STAGING_ETCDIR_NATIVE', False),
-   'PACKAGE-FEED-GPG-PUBKEY'))
+ os.path.join(d.getVar('STAGING_DIR_TARGET', False),
+  d.getVar('sysconfdir', False),
+  'pki',
+  'packagefeed-gpg',
+  'PACKAGEFEED-GPG-KEY-${DISTRO_VERSION}'))
 }
 
-do_package_index[depends] += "signing-keys:do_export_public_keys"
-do_rootfs[depends] += "signing-keys:do_export_public_keys"
+do_package_index[depends] += "signing-keys:do_deploy"
+do_rootfs[depends] += "signing-keys:do_populate_sysroot"
diff --git a/meta/classes/sign_rpm.bbclass b/meta/classes/sign_rpm.bbclass
index 8bcabee..d3e2b38 100644
--- a/meta/classes/sign_rpm.bbclass
+++ b/meta/classes/sign_rpm.bbclass
@@ -28,8 +28,11 @@ python () {
 raise_sanity_error("You need to define %s in the config" % var, d)
 
 # Set the expected location of the public key
-d.setVar('RPM_GPG_PUBKEY', os.path.join(d.getVar('STAGING_ETCDIR_NATIVE', 
False),
-'RPM-GPG-PUBKEY'))
+d.setVar('RPM_GPG_PUBKEY', os.path.join(d.getVar('STAGING_DIR_TARGET', 
False),
+d.getVar('sysconfdir', False),
+'pki',
+'rpm-gpg',
+'RPM-GPG-KEY-${DISTRO_VERSION}'))
 }
 
 python sign_rpm () {
@@ -45,5 +48,5 @@ python sign_rpm () {
 signer.sign_rpms(rpms)
 }
 
-do_package_index[depends] += "signing-keys:do_export_public_keys"
-do_rootfs[depends] += "signing-keys:do_export_public_keys"
+do_package_index[depends] += "signing-keys:do_deploy"
+do_rootfs[depends] += "signing-keys:do_populate_sysroot"
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 26f6466..340f104 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -145,16 +145,6 @@ class RpmIndexer(Indexer):
 if signer:
 for repomd in repomd_files:
 signer.detach_sign(repomd)
-# Copy pubkey(s) to repo
-distro_version = self.d.getVar('DISTRO_VERSION', True) or "oe.0"
-if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
-shutil.copy2(self.d.getVar('RPM_GPG_PUBKEY', True),
- os.path.join(self.deploy_dir,
-  'RPM-GPG-KEY-%s' % distro_version))
-if self.d.getVar('PACKAGE_FEED_SIGN', True) == '1':
-shutil.copy2(self.d.getVar('PACKAGE_FEED_GPG_PUBKEY', True),
- os.path.join(self.deploy_dir,
-  'REPODATA-GPG-KEY-%s' % distro_version))
 
 
 class OpkgIndexer(Indexer):
diff --git a/meta/recipes-core/meta/signing-keys.bb 
b/meta/recipes-core/meta/signing-keys.bb
index d7aa79d..2f190c3 100644
--- a/meta/recipes-core/meta/signing-keys.bb
+++ b/meta/recipes-core/meta/signing-keys.bb
@@ -3,25 +3,21 @@
 
 DESCRIPTION = "Make public keys of the signing keys available"
 LICENSE = "MIT"
-PACKAGES = ""
-
-do_fetch[noexec] = "1"
-do_unpack[noexec] = "1"
-do_patch[noexec] = "1"
-do_configure[noexec] = "1"
-do_compile[noexec] = "1"
-do_install[noexec] = "1"
-do_package[noexec] = "1"
-do_packagedata[noexec] = "1"
-do_package_write_ipk[noexec] = "1"
-do_package_write_rpm[noexec] = "1"
-do_package_write_deb[noexec] = "1"
-do_populate_sysroot[noexec] = "1"
+LIC_FILES_CHKSUM = 
"file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \
+
file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+
+i

[OE-core] [PATCH 2/5] toolchain-scripts.bbclass: Use PYTHONPATH instead of PYTHONHOME

[Randy Witt]

In the extensible sdk it was originally intended that the native sstate
would always be setscened as part of the sdk installation. However, the
soon to come "minimal" sdk won't do that.

A side effect of that is that pointing PYTHONHOME at the native sysroot
within the "bitbake workspace" won't work. For now only set PYTHONPATH
instead and continue using the python that comes from buildtools.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/toolchain-scripts.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/toolchain-scripts.bbclass 
b/meta/classes/toolchain-scripts.bbclass
index ab4feb0..13e7390 100644
--- a/meta/classes/toolchain-scripts.bbclass
+++ b/meta/classes/toolchain-scripts.bbclass
@@ -31,7 +31,7 @@ toolchain_create_sdk_env_script () {
echo "export OECORE_NATIVE_SYSROOT=\"$sdkpathnative\"" >> $script
echo 'export OECORE_TARGET_SYSROOT="$SDKTARGETSYSROOT"' >> $script
echo "export OECORE_ACLOCAL_OPTS=\"-I 
$sdkpathnative/usr/share/aclocal\"" >> $script
-   echo "export PYTHONHOME=$sdkpathnative$prefix" >> $script
+   echo "export PYTHONPATH=$sdkpathnative$prefix" >> $script
echo 'unset command_not_found_handle' >> $script
 
toolchain_shared_env_script
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/5] copy_buildsystem.py: Pass the nativelsb argument to gen-lockedsig-cache

[Randy Witt]

If the nativelsb argument is not used, then create_locked_sstate_cache()
can get collisions when moving the files from the input_sstate_cache
to the output_sstate_cache.

The specific case where this was encountered was when a "universal"
nativelsb directory already existed in the input_sstate_cache.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/lib/oe/copy_buildsystem.py | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index fb51b51..0627461 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -154,4 +154,11 @@ def create_locked_sstate_cache(lockedsigs, 
input_sstate_cache, output_sstate_cac
 if fixedlsbstring:
 nativedir = output_sstate_cache + '/' + nativelsbstring
 if os.path.isdir(nativedir):
-os.rename(nativedir, output_sstate_cache + '/' + fixedlsbstring)
+destdir = os.path.join(output_sstate_cache, fixedlsbstring)
+bb.utils.mkdirhier(destdir)
+
+dirlist = os.listdir(nativedir)
+for i in dirlist:
+src = os.path.join(nativedir, i)
+dest = os.path.join(destdir, i)
+os.rename(src, dest)
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/5] extsdk bug fixes and minimal type addition

[Randy Witt]

The following changes since commit b33e440cc6fbd703e8045d94b806790343e72eb6:

  libical: Work around hardcoded paths in pkgconfig file (2016-02-04 13:09:56 
+)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib 
fcef7183f0ac9ab38715bb0c0a80109696d79b2d
  
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=fcef7183f0ac9ab38715bb0c0a80109696d79b2d

Randy Witt (5):
  copy_buildsystem.py: Pass the nativelsb argument to
gen-lockedsig-cache
  toolchain-scripts.bbclass: Use PYTHONPATH instead of PYTHONHOME
  populate_sdk_ext: Don't set sdk_update_targets in the config
  populate_sdk_ext: Add support for a "minimal" type
  populate_sdk_ext: Make populate_sdk_ext depend on sdk_extra_conf

 meta/classes/populate_sdk_ext.bbclass  | 21 +++--
 meta/classes/toolchain-scripts.bbclass |  2 +-
 meta/lib/oe/copy_buildsystem.py|  9 -
 3 files changed, 20 insertions(+), 12 deletions(-)

-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/5] populate_sdk_ext: Don't set sdk_update_targets in the config

[Randy Witt]

sdk_update_targets isn't used by any code, so there is no reason to set
it.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index da7fdfb..ba30023 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -99,7 +99,6 @@ python copy_buildsystem () {
 config.set('General', 'core_meta_subdir', core_meta_subdir)
 config.add_section('SDK')
 config.set('SDK', 'sdk_targets', d.getVar('SDK_TARGETS', True))
-config.set('SDK', 'sdk_update_targets', d.getVar('SDK_INSTALL_TARGETS', 
True))
 updateurl = d.getVar('SDK_UPDATE_URL', True)
 if updateurl:
 config.set('SDK', 'updateserver', updateurl)
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 5/5] populate_sdk_ext: Make populate_sdk_ext depend on sdk_extra_conf

[Randy Witt]

If a user changes sdk_extra_conf, it should cause populate_sdk_ext to
run.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index caf8b6e..704c003 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -361,8 +361,9 @@ do_populate_sdk_ext[depends] = 
"${@d.getVarFlag('do_populate_sdk', 'depends', Fa
 
 do_populate_sdk_ext[rdepends] += "${@' '.join([x + ':do_build' for x in 
d.getVar('SDK_TARGETS', True).split()])}"
 
-# Make sure codes change in copy_buildsystem can result in rebuilt
-do_populate_sdk_ext[vardeps] += "copy_buildsystem"
+# Make sure code changes can result in rebuild
+do_populate_sdk_ext[vardeps] += "copy_buildsystem \
+ sdk_ext_postinst"
 
 do_populate_sdk_ext[file-checksums] += 
"${COREBASE}/meta/files/toolchain-shar-relocate.sh:True \
 
${COREBASE}/meta/files/toolchain-shar-extract.sh:True \
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 4/5] populate_sdk_ext: Add support for a "minimal" type

[Randy Witt]

If the user sets the SDK_EXT_TYPE variable to "minimal" then the sdk won't
contain any sstate. The sstate can come from an sstate mirror and be
installed on demand as usual.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index ba30023..caf8b6e 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -18,6 +18,9 @@ SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 SDK_EXT = ""
 SDK_EXT_task-populate-sdk-ext = "-ext"
 
+# Options are full or minimal
+SDK_EXT_TYPE ?= "full"
+
 SDK_LOCAL_CONF_WHITELIST ?= ""
 SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
  BB_NUMBER_THREADS \
@@ -29,7 +32,7 @@ SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
 SDK_TARGETS ?= "${PN}"
-SDK_INSTALL_TARGETS = "${SDK_TARGETS} 
${@'meta-world-pkgdata:do_allpackagedata' if d.getVar('SDK_INCLUDE_PKGDATA', 
True) == '1' else ''}"
+SDK_INSTALL_TARGETS = "${@SDK_TARGETS if d.getVar('SDK_EXT_TYPE', True) != 
'minimal' else ''} ${@'meta-world-pkgdata:do_allpackagedata' if 
d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
 
 # The files from COREBASE that you want preserved in the COREBASE copied
@@ -203,10 +206,6 @@ python copy_buildsystem () {
 bb.utils.remove(sstate_out, True)
 # uninative.bbclass sets NATIVELSBSTRING to 'universal'
 fixedlsbstring = 'universal'
-oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_pruned,
-   d.getVar('SSTATE_DIR', 
True),
-   sstate_out, d,
-   fixedlsbstring)
 
 # Add packagedata if enabled
 if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
@@ -218,7 +217,9 @@ python copy_buildsystem () {
  d.getVar('STAGING_DIR_HOST', 
True) + '/world-pkgdata/locked-sigs-pkgdata.inc',
  lockedsigs_pruned,
  lockedsigs_copy)
-oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_copy,
+
+if d.getVar('SDK_EXT_TYPE', True) != 'minimal':
+oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_pruned,
d.getVar('SSTATE_DIR', 
True),
sstate_out, d,
fixedlsbstring)
@@ -302,7 +303,7 @@ sdk_ext_postinst() {
# For now this is where uninative.bbclass expects the tarball
mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname 
${oe_init_build_env_path}`
 
-   if [ "$prepare_buildsystem" != "no" ]; then
+   if [ "$prepare_buildsystem" != "no" -a -n 
"${@SDK_INSTALL_TARGETS.strip()}" ]; then
printf "Preparing build system...\n"
# dash which is /bin/sh on Ubuntu will not preserve the
# current working directory when first ran, nor will it set $1 
when
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_ext: Add SSTATE_MIRRORS to config blacklist

[Randy Witt]

SSTATE_MIRRORS used by the builder may not be the same as what the
installer of the sdk will use. Therefore blacklist SSTATE_MIRRORS from
the builder configuration. Note: the actual SSTATE_MIRRORS for the sdk
can be added using sdk_extra_conf.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index a115127..da7fdfb 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -19,7 +19,12 @@ SDK_EXT = ""
 SDK_EXT_task-populate-sdk-ext = "-ext"
 
 SDK_LOCAL_CONF_WHITELIST ?= ""
-SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION BB_NUMBER_THREADS PARALLEL_MAKE 
PRSERV_HOST"
+SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
+ BB_NUMBER_THREADS \
+ PARALLEL_MAKE \
+ PRSERV_HOST \
+ SSTATE_MIRRORS \
+"
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] pixz: Add 1.0.6

[Randy Witt]

On 01/08/2016 10:22 AM, Richard Purdie wrote:

xz gives better compression results than bzip/gz but is often slower.
Using parallel compression mitigates this somewhat and is particularly
useful for the SDK.


xz as of 5.2.0 allows for multi-threaded compression.

http://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/3] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

[Randy Witt]

The nativesdk and crosssdk targets were only getting pulled into the
extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA.
However the nativesdk and crosssdk targets are not needed for the
extensible sdk to be used.

This patch therefore removes them since it reduces the size of the
extensible sdks that include sstate by ~300MB.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 5c99bd0..ef014b8 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -173,6 +173,12 @@ python copy_buildsystem () {
 # Filter the locked signatures file to just the sstate tasks we are 
interested in
 allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 
'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 
'do_package_write_deb', 'do_package_qa', 'do_deploy']
 excluded_targets = d.getVar('SDK_TARGETS', True)
+
+# Since buildtools is included in the sdk, the nativesdk items and
+# crosssdk end up in the tasks in locked-sigs.inc. However, the nativesdk
+# and crosssdk items aren't needed for the extensible sdk to function so
+# remove them from the list.
+excluded_targets += 'nativesdk crosssdk'
 lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
 oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
  excluded_targets.split(),
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] copy_buildsystem.py: Allow substring matches for excluded_targets

[Randy Witt]

This is so that the excluded_targets passed to prune_lockedsigs can
take a partial form such as nativesdk to filter based on partial matches.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/lib/oe/copy_buildsystem.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 15af4eb..9d8f3a6 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -83,8 +83,11 @@ def prune_lockedsigs(allowed_tasks, excluded_targets, 
lockedsigs, pruned_output)
 for line in infile:
 if invalue:
 if line.endswith('\\\n'):
-splitval = line.strip().split(':')
-if splitval[1] in allowed_tasks and not splitval[0] in 
excluded_targets:
+target, task = line.strip().split(':')[:2]
+badmatch = ([x for x in excluded_targets.split()
+ if x in target])
+
+if task in allowed_tasks and not badmatch:
 f.write(line)
 else:
 f.write(line)
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/2] Extensible SDK size reduction

[Randy Witt]

This removes some unneeded items from the extensible sdk to reduce size by
~300MB.

The following changes since commit f8185ff9f8ac697733b41f023cb3a4b98b2f823f:

  bitbake: ast: Add filename/lineno to mapped functions (2016-01-06 15:27:35 
+)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/extsdksize
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/extsdksize

Randy Witt (2):
  copy_buildsystem.py: Allow substring matches for excluded_targets
  populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

 meta/classes/populate_sdk_ext.bbclass | 6 ++
 meta/lib/oe/copy_buildsystem.py   | 7 +--
 2 files changed, 11 insertions(+), 2 deletions(-)

-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

[Randy Witt]

The nativesdk and crosssdk targets were only getting pulled into the
extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA.
However the nativesdk and crosssdk targets are not needed for the
extensible sdk to be used.

This patch therefore removes them since it reduces the size of the
extensible sdks that include sstate by ~300MB.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index c30181a..60467b9 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -173,6 +173,12 @@ python copy_buildsystem () {
 # Filter the locked signatures file to just the sstate tasks we are 
interested in
 allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 
'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 
'do_package_write_deb', 'do_package_qa', 'do_deploy']
 excluded_targets = d.getVar('SDK_TARGETS', True)
+
+# Since buildtools is included in the sdk, the nativesdk items and
+# crosssdk end up in the tasks in locked-sigs.inc. However, the nativesdk
+# and crosssdk items aren't needed for the extensible sdk to function so
+# remove them from the list.
+excluded_targets += 'nativesdk crosssdk'
 lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
 oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
  excluded_targets,
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [jethro][master][PATCH 1/1] uninative.bbclass: Choose the correct loader based on BUILD_ARCH

[Randy Witt]

On 12/10/2015 02:01 PM, Burton, Ross wrote:

On 9 December 2015 at 20:50, Randy Witt <randy.e.w...@linux.intel.com>
wrote:


-UNINATIVE_LOADER = "${STAGING_DIR_NATIVE}/lib/ld-linux-x86-64.so.2"
+UNINATIVE_LOADER ?= "${@bb.utils.contains('BUILD_ARCH', 'x86_64',
'${STAGING_DIR_NATIVE}/lib/ld-linux-x86-64.so.2',
'${STAGING_DIR_NATIVE}/lib/ld-linux.so.2', d)}"



Would it be neater, and more flexible when other arches are different
(specifically I'm thinking of aarch64 here) to do:

UNINATIVE_LOADER = "${STAGING_DIR_NATIVE}/lib/ld-linux.so.2"
UNINATIVE_LOADER_x86-64 = "'${STAGING_DIR_NATIVE}/lib/ld-linux-x86-64.so.2"


Yes, I like that better. I'll check the override and test it again.


At least I think that overload is correct...

Ross



--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [jethro][master][PATCH 0/1] Fix for 32-bit uninative

[Randy Witt]

The following changes since commit c093fd8bd2fbf070728f5401ce3e6cf951fa820d:

  linux-yocto/4.1: Fix kernel oops on qemuarm boot (2015-12-08 17:22:31 +)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/uninative-32bit-fix
  
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/uninative-32bit-fix

Randy Witt (1):
  uninative.bbclass: Choose the correct loader based on BUILD_ARCH

 meta/classes/uninative.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [jethro][master][PATCH 1/1] uninative.bbclass: Choose the correct loader based on BUILD_ARCH

[Randy Witt]

Previously UNINATIVE_LOADER was always ld-linux-x86-64.so.2. That is
incorrect when the host is 32-bit.

This change also changes to using ?= so the user can override
UNINATIVE_LOADER if so desired.

[YOCTO #8124]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/classes/uninative.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 51391db..0cd27db 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -1,6 +1,6 @@
 NATIVELSBSTRING = "universal"
 
-UNINATIVE_LOADER = "${STAGING_DIR_NATIVE}/lib/ld-linux-x86-64.so.2"
+UNINATIVE_LOADER ?= "${@bb.utils.contains('BUILD_ARCH', 'x86_64', 
'${STAGING_DIR_NATIVE}/lib/ld-linux-x86-64.so.2', 
'${STAGING_DIR_NATIVE}/lib/ld-linux.so.2', d)}"
 
 addhandler uninative_eventhandler
 uninative_eventhandler[eventmask] = "bb.event.BuildStarted"
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] RFC: Reference updater filesystem

[Randy Witt]

On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez <
mariano.lo...@linux.intel.com> wrote:

> There has been interest in an image based software updater in Yocto
> Project. The proposed solution for a image based updater is to use Stefano
> Babic's software updater (http://sbabic.github.io/swupdate). This
> software do a binary copy, so it is needed to have at least two partitions,
> these partitions would be the rootfs and the maintenance partition. The
> rootfs it's the main partition used to boot during the normal device
> operation, on the other hand, the maintenance will be used to update the
> main partition.
>
> To update the system, the user has to connect to device and boot in the
> maintenance partition; once in the maintenance partition the software
> updater will copy the new image in the rootfs partition. A final reboot
> into the rootfs it is necessary to complete the upgrade.
>
> As mentioned before the the software will copy an image to the partition,
> so everything in that partition will be wiped out, including custom
> configurations. To avoid the loss of configuration I explore three
> different solutions:
> 1. Use a separate partition for the configuration.
>   a. The pro of this method is the partition is not touched during the
> update.
>   b. The con of this method is the configuration is not directly in rootfs
> (example: /etc).
>
> Configuration files can be anywhere a package decides to install them. So
having a single partition would be difficult. If you could, you would most
likely be forced to have an initramfs to make sure /etc was mounted before
init runs.


> 2. Do the backup during the update.
>   a. The pro is the configuration is directly in rootfs.
>   b. The con is If the update fail most likely the configuration would be
> lost.
>
> Why would the configuration be lost if the update fails? Couldn't it just
be stored on the thumbdrive?


> 3. Have an OverlayFS for the rootfs or the partition that have the
> configuration.
>   a. The pro is the configuration is  "directly" in rootfs.
>   b. The con is there is need to provide a custom init to guarantee the
> Overlay is mounted before the boot process.
>
> With the above information I'm proposing to use a separate partition for
> the configuration; this is because is more reliable and doesn't require big
> changes in the current architecture.
>
> So, the idea is to have 4 partitions in the media:
> 1. boot. This is the usual boot partition
> 2. data. This will hold the configuration files. Not modified by updates.
> 3. maintenance. This partition will be used to update rootfs.
> 4. rootfs. Partition used for normal operation.
>
> Mariano
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] scripts: analyze_patch: Analyze patch and suggest tests.

[Randy Witt]

On 11/23/2015 07:55 AM, Leonardo Sandoval wrote:

Hi Daniel,

Some comments inline. BTW, is the script's output intended to be used as input
of another one? Currently, the output is human-friendly (which is good) but at
the end the purpose is to use the output and run this set of scripts/tests
automatically, trying to detect possible issues. Also, I briefly tried the sfood
library, and took quite a lot of time compared to deps.txts. I wonder if we can
fasten somehow this sfood process.

What is a "long time"? I don't like deps.txt at all because it will most likely 
become stale rather quickly.




On 11/20/2015 08:22 AM, Daniel Istrate wrote:

[YOCTO #8647] - Script to analyse what areas a patch changes
and what tests need running

This script will analyze files modified by a commit and
attempt to suggest what tests to run.
It is somewhat dependent of the 'snakefood' tool to determine
the import relationship between files. If this pkg is missing
a backup file will be used instead: deps.txt

Usage: analyze_patch --revision 
Example: analyze_patch --revision 351c69a

Revision: 351c69a

Modified files:
meta/recipes-support/sqlite/files/0001-using-the-dynamic-library.patch
meta/recipes-support/sqlite/sqlite3_3.8.10.2.bb
meta/recipes-support/sqlite/sqlite3_3.9.0.bb

Suggested tests:
Rebuild: sqlite3

Signed-off-by: Daniel Istrate 
---
  scripts/analyze_patch | 300 
  scripts/deps.txt  | 628 ++
  2 files changed, 928 insertions(+)
  create mode 100755 scripts/analyze_patch
  create mode 100644 scripts/deps.txt

diff --git a/scripts/analyze_patch b/scripts/analyze_patch
new file mode 100755
index 000..1b6609b
--- /dev/null
+++ b/scripts/analyze_patch
@@ -0,0 +1,300 @@
+#!/usr/bin/env python
+

THE GPL license should be included on this part.

+# Description: This will analyze the files modified by a commit and attempt
to suggest what tests to run.
+#  It uses an adjacent file 'deps.txt' that was generated by
'snakefood' tool:
+#  sfood  | grep -v /usr/lib/python | grep -v
/usr/share/pyshared/ |
+#  grep -v None | grep ".py')," | grep ".py\'))" > deps.txt
+#  This file has to be edited further: remove the  and
replace  "', '" with '/'.
+#  In the end you should have something like this: a matrix with
2 columns, which represent
+#  elements from the left import elements from the right.
+# (('bitbake/lib/bb/build.py'), ('bitbake/lib/bb/event.py'))
+# (('bitbake/lib/bb/build.py'), ('bitbake/lib/bb/msg.py'))
+#
+# How to use it: analyze_patch --revision 
+#analyze_patch -r 2d1071e
+# Notes: if revision is not provided, will default to HEAD
+# Author: Daniel Istrate 
+
+import argparse
+import os
+import glob
+import ast
+import re
+from subprocess import PIPE, Popen, STDOUT
+
+parser = argparse.ArgumentParser()
+parser.add_argument('-r', '--revision', help=' changes to analyze.')

Another option would be to define which method to use (sfood or deps.tx) in case
we have more than one method

+args = parser.parse_args()
+
+revision = args.revision or 'HEAD'

This default can be include on the same line were the argument was added.

+
+this_file_path = os.path.dirname(os.path.abspath(__file__))
+poky_dir = os.path.abspath(os.path.join(this_file_path, os.pardir))
+# Move into the the poky dir
+os.chdir(poky_dir)
+
+# Where are the available tests located
+bb_test_dir = 'bitbake/lib/bb/tests/'
+selftest_dir = 'meta/lib/oeqa/selftest/'
+runtime_dir = 'meta/lib/oeqa/runtime/'
+
+RETOK = 0
+
+
+def run_cmd(cmd, ignore_err=False):
+"""
+Run cmd in bash shell
+:param   cmd: command to be ran.
+:param   ignore_err: redirect stderr to /dev/null, otherwise redirect
stderr to stdout
+:return: tuple containing exit code & output of the cmd
+"""
+DEVNULL = open(os.devnull, 'w')
+
+if ignore_err:
+proc = Popen(args=cmd, stdout=PIPE, stderr=DEVNULL, shell=True,
executable='/bin/bash')
+else:
+proc = Popen(args=cmd, stdout=PIPE, stderr=STDOUT, shell=True,
executable='/bin/bash')
+retval = proc.communicate()[0]
+retcode = proc.returncode
+
+return retcode, retval
+
+
+def get_modified_files(rev=revision):
+"""
+Determine modified files by a commit
+:param   rev: the revision number to determine modified files for.
+:return: a list of modified files
+"""
+cmd = 'git diff-tree --no-commit-id --name-only -r ' + rev
+retcode, retval = run_cmd(cmd)
+
+if retcode != RETOK:
+return retcode, retval
+
+retlist = retval.split('\n')
+retlist.remove('')
+
+return retlist
+
+
+def get_list_of_testuites(test_dir):
+"""
+Looks for .py files in test_dir
+:param test_dir: directory containing (hopefully) test suites
+:return: list of test suites
+"""

Re: [OE-core] [PATCH 1/1] glibc: Allow 64 bit atomics for x86

[Randy Witt]

On Thu, Oct 29, 2015 at 3:20 PM, Richard Purdie <
richard.pur...@linuxfoundation.org> wrote:

> On Thu, 2015-10-29 at 13:43 -0700, Juro Bystricky wrote:
> > This patch fixes [YOCTO#8140].
> >
> > The fix consist of allowing 64bit atomic ops for x86.
> > This should be safe for i586 and newer CPUs.
> > It also makes the synchronization more efficient.
> >
> > Signed-off-by: Juro Bystricky 
> > ---
> >  .../glibc/glibc/use_64bit_atomics.patch| 24
> ++
> >  meta/recipes-core/glibc/glibc_2.22.bb  |  1 +
> >  2 files changed, 25 insertions(+)
> >  create mode 100644 meta/recipes-core/glibc/glibc/use_64bit_atomics.patch
>
> Since the patch is only changing nativesdk 32 bit x86 and we know that
> the 32 bit SDK is pretty broken at the moment I've merged this on the
> basis that it can't really make it any worse. There is pressure to move
> to the next rc candidate for 2.0.
>
> I would like to get to the bottom of the real issue here and I still
> suspect 32 bit x86 images are likely broken too :/.
>

With Juro's change I built core-image-minimal using a 32-bit buildtools on
a 32-bit host(vm) and that appears to boot fine using runqemu.


> Cheers,
>
> Richard
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 4/5] mtools_4.0.18.bb: Use create_wrapper() for mcopy

[Randy Witt]

mcopy uses the IBM850 codepage from gconv. The default install location
for gconv will not match the actual install location since it can be
pulled from sstate. This patch overrides the default location when
running by adding GCONV_PATH to the environment for mcopy.

[YOCTO #7629]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 meta/recipes-devtools/mtools/mtools_4.0.18.bb | 5 +
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-devtools/mtools/mtools_4.0.18.bb 
b/meta/recipes-devtools/mtools/mtools_4.0.18.bb
index 52decfd..24c9d49 100644
--- a/meta/recipes-devtools/mtools/mtools_4.0.18.bb
+++ b/meta/recipes-devtools/mtools/mtools_4.0.18.bb
@@ -45,3 +45,8 @@ do_install_prepend () {
 mkdir -p ${D}/${bindir}
 mkdir -p ${D}/${datadir}
 }
+
+do_install_append_class-native () {
+create_wrapper ${D}${bindir}/mcopy \
+GCONV_PATH=${libdir}/gconv
+}
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 5/5] wic/utils/oe/misc.py: Preserve PATH when running native tools

[Randy Witt]

Previously exec_native_cmd() would remove all items from PATH except for
the native sysroot. This can cause issues for the tools that are created
using create_wrapper().

Now instead of wiping out the PATH, run a sanity check to check if the
command is in the native sysroot.

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 20 
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 7370d93..c6d2e5f 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -82,6 +82,12 @@ def exec_cmd(cmd_and_args, as_shell=False, catch=3):
 
 return out
 
+def cmd_in_path(cmd, path):
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+return bb.utils.which(path, cmd) != "" or False
 
 def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
 """
@@ -92,15 +98,21 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
 Always need to execute native commands as_shell
 """
 native_paths = \
-"export PATH=%s/sbin:%s/usr/sbin:%s/usr/bin" % \
+"%s/sbin:%s/usr/sbin:%s/usr/bin" % \
 (native_sysroot, native_sysroot, native_sysroot)
-native_cmd_and_args = "%s;%s" % (native_paths, cmd_and_args)
+native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
+   (native_paths, cmd_and_args)
 msger.debug("exec_native_cmd: %s" % cmd_and_args)
 
-args = cmd_and_args.split()
+# The reason -1 is used is because there may be "export" commands.
+args = cmd_and_args.split(';')[-1].split()
 msger.debug(args)
 
-ret, out = _exec_cmd(native_cmd_and_args, True, catch)
+# If the command isn't in the native sysroot say we failed.
+if cmd_in_path(args[0], native_paths):
+ret, out = _exec_cmd(native_cmd_and_args, True, catch)
+else:
+ret = 127
 
 if ret == 127: # shell command-not-found
 prog = args[0]
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/5] Fix live images when using the extensible sdk

[Randy Witt]

The following changes since commit 505a82673ac2487df5ea343a6422c2fc47018831:

  build-appliance-image: Update to jethro head revision (2015-10-21 23:13:11 
+0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/codepage
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/codepage

Randy Witt (3):
  Add 850 codepage to uninative-tarball
  mtools_4.0.18.bb: Use create_wrapper() for mcopy
  wic/utils/oe/misc.py: Preserve PATH when running native tools

Richard Purdie (2):
  scripts/oe-pkgdata-util: Fix variable name in error handling
  populate_sdk_base: Ensure PKGDATA_DIR exists

 meta/classes/populate_sdk_base.bbclass|  2 +-
 meta/recipes-core/meta/uninative-tarball.bb   |  1 +
 meta/recipes-devtools/mtools/mtools_4.0.18.bb |  5 +
 scripts/lib/wic/utils/oe/misc.py  | 20 
 scripts/oe-pkgdata-util   |  2 +-
 5 files changed, 24 insertions(+), 6 deletions(-)

-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/5] Add 850 codepage to uninative-tarball

[Randy Witt]

---
 meta/recipes-core/meta/uninative-tarball.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/meta/uninative-tarball.bb 
b/meta/recipes-core/meta/uninative-tarball.bb
index 41f7927..21f3bd9 100644
--- a/meta/recipes-core/meta/uninative-tarball.bb
+++ b/meta/recipes-core/meta/uninative-tarball.bb
@@ -7,6 +7,7 @@ TOOLCHAIN_TARGET_TASK = ""
 
 TOOLCHAIN_HOST_TASK = "\
 nativesdk-glibc \
+nativesdk-glibc-gconv-ibm850 \
 nativesdk-patchelf \
 "
 
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/5] scripts/oe-pkgdata-util: Fix variable name in error handling

[Randy Witt]

From: Richard Purdie 

Fix:
logger.error('Unable to find pkgdata directory %s' % pkgdata_dir)
NameError: global name 'pkgdata_dir' is not defined

Signed-off-by: Richard Purdie 
---
 scripts/oe-pkgdata-util | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
index 116cfae..cb19cc4 100755
--- a/scripts/oe-pkgdata-util
+++ b/scripts/oe-pkgdata-util
@@ -496,7 +496,7 @@ def main():
 sys.exit(1)
 
 if not os.path.exists(args.pkgdata_dir):
-logger.error('Unable to find pkgdata directory %s' % pkgdata_dir)
+logger.error('Unable to find pkgdata directory %s' % args.pkgdata_dir)
 sys.exit(1)
 
 ret = args.func(args)
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/5] populate_sdk_base: Ensure PKGDATA_DIR exists

[Randy Witt]

From: Richard Purdie 

Signed-off-by: Richard Purdie 
---
 meta/classes/populate_sdk_base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_base.bbclass 
b/meta/classes/populate_sdk_base.bbclass
index 36876e8..0a0ba91 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -188,7 +188,7 @@ populate_sdk_log_check() {
done
 }
 
-do_populate_sdk[dirs] = "${TOPDIR}"
+do_populate_sdk[dirs] = "${TOPDIR} ${PKGDATA_DIR}"
 do_populate_sdk[depends] += "${@' '.join([x + ':do_populate_sysroot' for x in 
d.getVar('SDK_DEPENDS', True).split()])}  ${@d.getVarFlag('do_rootfs', 
'depends', False)}"
 do_populate_sdk[rdepends] = "${@' '.join([x + ':do_populate_sysroot' for x in 
d.getVar('SDK_RDEPENDS', True).split()])}"
 do_populate_sdk[recrdeptask] += "do_packagedata do_package_write_rpm 
do_package_write_ipk do_package_write_deb"
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] runqemu-internal: Make sure two serial ports always exist

[Randy Witt]

Since inittab for qemu images now always tries to start getty on a
second serial device, make sure that device exists.

Otherwise the following message will be spammed:
INIT: Id "S1" respawning too fast: disabled for 5 minutes

[YOCTO #8374]

Signed-off-by: Randy Witt <randy.e.w...@linux.intel.com>
---
 scripts/runqemu-internal | 32 
 1 file changed, 32 insertions(+)

diff --git a/scripts/runqemu-internal b/scripts/runqemu-internal
index d6b1102..078545e 100755
--- a/scripts/runqemu-internal
+++ b/scripts/runqemu-internal
@@ -700,6 +700,38 @@ if [ "x$SERIALSTDIO" = "x1" ]; then
 stty intr ^]
 fi
 
+
+# Preserve the multiplexing behavior for the monitor that would be there based
+# on whether nographic is used.
+if echo "$QEMUOPTIONS $SERIALOPTS $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT" | 
grep -- "-nographic"; then
+FIRST_SERIAL_OPT="-serial mon:stdio"
+else
+FIRST_SERIAL_OPT="-serial mon:vc"
+fi
+
+# qemuarm64 uses virtio for any additional serial ports so the normal mechanism
+# of using -serial will not work
+if [ "$MACHINE" = "qemuarm64" ]; then
+SECOND_SERIAL_OPT="$SCRIPT_QEMU_EXTRA_OPT -device virtio-serial-device 
-chardev null,id=virtcon -device virtconsole,chardev=virtcon"
+else
+SECOND_SERIAL_OPT="-serial null"
+fi
+
+# We always want a ttyS1. Since qemu by default adds a serial port when
+# nodefaults is not specified, it seems that all that would be needed is to
+# make sure a "-serial" is there. However, it appears that when "-serial" is
+# specified, it ignores the default serial port that is normally added.
+# So here we make sure to add two -serial if there are none. And only one
+# if there is one -serial already.
+NUM_SERIAL_OPTS=`echo $QEMUOPTIONS $SERIALOPTS $SCRIPT_QEMU_OPT 
$SCRIPT_QEMU_EXTRA_OPT | sed -e 's/ /\n/g' | grep --count -- -serial`
+
+if [ "$NUM_SERIAL_OPTS" = "0" ]; then
+SCRIPT_QEMU_EXTRA_OPT="$SCRIPT_QEMU_EXTRA_OPT $FIRST_SERIAL_OPT 
$SECOND_SERIAL_OPT"
+elif [ "$NUM_SERIAL_OPTS" = "1" ]; then
+SCRIPT_QEMU_EXTRA_OPT="$SCRIPT_QEMU_EXTRA_OPT $SECOND_SERIAL_OPT"
+fi
+
+
 echo "Running $QEMU..."
 # -no-reboot is a mandatory option - see bug #100
 if [ "$FSTYPE" = "vmdk" -o "$FSTYPE" = "hddimg" -o "$FSTYPE" = "hdddirect" ]; 
then
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [oe] Extensible SDK install errors

[Randy Witt]

On Tue, Aug 25, 2015 at 6:11 PM, Adam Lee adam.yh@gmail.com wrote:

 Thanks for your comments. I was not running the installer as root.
 I tried again and specified the install path in my home directory.
 This time it seems to have gone further, but still failed with this:

 error: Can't install packagegroup-core-boot-1.0-r17.0@qemuarm: no
 package provides systemd

 Perhaps I need to disable systemd from distro features?


You could do that, however there is a patch on the list to fix this kind of
problem.

http://patchwork.openembedded.org/patch/101213/

It may fix your problem until the next release comes out. I'm sorry you've
had this trouble. It was just something that was missed in the first
version of the extensible sdk.


 I will also check my user's permission to solve the permission error.

 On Mon, Aug 24, 2015 at 4:11 AM Paul Eggleton 
 paul.eggle...@linux.intel.com wrote:

 Hi Adam,

 On Friday 21 August 2015 15:06:25 Adam Lee wrote:
  Hello, I built the Extensible SDK on Fido (bitbake core-image-minimal -c
  populate_sdk_ext).
 
  During the installation I get this permission error:
 
  $ ./poky-glibc-x86_64-core-image-minimal-armv5e-toolchain-ext-1.8.sh
 
   Enter target directory for SDK (default: /opt/poky/1.8):
   You are about to install the SDK to /opt/poky/1.8. Proceed[Y/n]?
   Extracting SDK...done
   Setting it up...
   Extracting buildtools...
   ./poky-glibc-x86_64-core-image-minimal-armv5e-toolchain-ext-1.8.sh:
 line
   148: /opt/poky/1.8/environment-setup-armv5e-poky-linux-gnueabi:
 Permission
   denied
   ./poky-glibc-x86_64-core-image-minimal-armv5e-toolchain-ext-1.8.sh:
 line
   151: /opt/poky/1.8/environment-setup-armv5e-poky-linux-gnueabi:
 Permission
   denied
   ./poky-glibc-x86_64-core-image-minimal-armv5e-toolchain-ext-1.8.sh:
 line
   155: /opt/poky/1.8/environment-setup-armv5e-poky-linux-gnueabi:
 Permission
   denied
   mv: cannot move ‘x86_64-nativesdk-libc.tar.bz2’ to
   ‘/opt/poky/1.8/layers/poky/x86_64-nativesdk-libc.tar.bz2’: Permission
   denied Preparing build system...
   sh: 1: cannot create preparing_build_system.log: Permission denied
   SDK preparation failed: see /opt/poky/1.8/preparing_build_system.log
 
   It looks like `/opt/poky/1.8/layers/poky` directory belongs to the root
  user:
 
  $ ls -l /opt/poky/1.8/layers/
 
   total 36
   drwxrwxr-x 9 root root 4096 Jun 16 10:25 meta-gnome
   drwxrwxr-x 9 root root 4096 Jun 15 10:44 meta-multimedia
   drwxrwxr-x 11 root root 4096 Jun 15 10:44 meta-networking
   drwxrwxr-x 20 root root 4096 Jun 15 10:44 meta-oe
   drwxrwxr-x 7 root root 4096 Jun 15 10:44 meta-python
   drwxrwxr-x 5 root root 4096 Jun 15 10:44 meta-ruby
   drwxrwxr-x 5 root root 4096 Jun 15 10:44 meta-systemd
   drwxrwxr-x 11 root root 4096 Jun 15 10:44 meta-xfce
   drwxrwxr-x 13 root root 4096 Aug 21 14:46 poky
 
   I may have missed something obvious. Has anyone seen this?

 Basically you shouldn't be installing the extensible SDK as root. We
 ought to
 be telling you about this up front and we should have a more reasonable
 default path - these are open bugs and patches are in review on the list
 to fix
 them.

 Cheers,
 Paul

 --

 Paul Eggleton
 Intel Open Source Technology Centre


 --
 ___
 Openembedded-core mailing list
 Openembedded-core@lists.openembedded.org
 http://lists.openembedded.org/mailman/listinfo/openembedded-core


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] runqemu: Add a tcpserial option

[Randy Witt]

The option was added so that the qemurunner could start a second tcp
serial port without adding machine conditional logic to qemurunner.

The issue that made this necessary was that when virt is passed to
qemu-system-aarch64, the normal mechanism for specifying a tcp serial
port does not work. This is because the hardware for the virt machine
is hardcoded in the device tree blob and the addition devices must be
virtio devices.

So runqemu can specify virtio for qemuarm64 whereas it seems all other
qemu machines work with the -serial tcp* option.

Signed-off-by: Randy Witt randy.e.w...@linux.intel.com
---
 meta/lib/oeqa/utils/qemurunner.py | 4 ++--
 scripts/runqemu   | 4 
 scripts/runqemu-internal  | 8 
 3 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py 
b/meta/lib/oeqa/utils/qemurunner.py
index e976fd0..33f3185 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -92,7 +92,7 @@ class QemuRunner:
 # Set this flag so that Qemu doesn't do any grabs as SDL grabs interact
 # badly with screensavers.
 os.environ[QEMU_DONT_GRAB] = 1
-self.qemuparams = 'bootparams=console=tty1 console=ttyS0,115200n8 
qemuparams=-serial tcp:127.0.0.1:{} -serial 
tcp:127.0.0.1:{}'.format(threadport, self.serverport)
+self.qemuparams = 'bootparams=console=tty1 console=ttyS0,115200n8 
qemuparams=-serial tcp:127.0.0.1:{}'.format(threadport)
 if qemuparams:
 self.qemuparams = self.qemuparams[:-1] +   + qemuparams +   + 
'\'
 
@@ -102,7 +102,7 @@ class QemuRunner:
 fcntl.fcntl(o, fcntl.F_SETFL, fl | os.O_NONBLOCK)
 return os.read(o.fileno(), 100)
 
-launch_cmd = 'runqemu %s %s %s' % (self.machine, self.rootfs, 
self.qemuparams)
+launch_cmd = 'runqemu tcpserial=%s %s %s %s' % (self.serverport, 
self.machine, self.rootfs, self.qemuparams)
 # FIXME: We pass in stdin=subprocess.PIPE here to work around stty
 # blocking at the end of the runqemu script when using this within
 # oe-selftest (this makes stty error out immediately). There ought
diff --git a/scripts/runqemu b/scripts/runqemu
index 09c507d..8271160 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -68,6 +68,7 @@ SCRIPT_QEMU_OPT=
 SCRIPT_QEMU_EXTRA_OPT=
 SCRIPT_KERNEL_OPT=
 SERIALSTDIO=
+TCPSERIAL_PORTNUM=
 KVM_ENABLED=no
 KVM_ACTIVE=no
 
@@ -150,6 +151,9 @@ while true; do
 SCRIPT_KERNEL_OPT=$SCRIPT_KERNEL_OPT console=ttyS0
 SERIALSTDIO=1
 ;;
+tcpserial=*)
+TCPSERIAL_PORTNUM=${arg##tcpserial=}
+;;
biosdir=*)
 CUSTOMBIOSDIR=${arg##biosdir=}
;;
diff --git a/scripts/runqemu-internal b/scripts/runqemu-internal
index d743322..b317358 100755
--- a/scripts/runqemu-internal
+++ b/scripts/runqemu-internal
@@ -620,6 +620,14 @@ if [ x$QEMUOPTIONS = x ]; then
 return 1
 fi
 
+if [ $TCPSERIAL_PORTNUM !=  ]; then
+if [ $MACHINE = qemuarm64 ]; then
+QEMUOPTIONS=$QEMUOPTIONS -device virtio-serial-device -chardev 
socket,id=virtcon,port=$TCPSERIAL_PORTNUM,host=127.0.0.1 -device 
virtconsole,chardev=virtcon
+else
+QEMUOPTIONS=$QEMUOPTIONS -serial tcp:127.0.0.1:$TCPSERIAL_PORTNUM
+fi
+fi
+
 PATH=$OECORE_NATIVE_SYSROOT/usr/bin:$PATH
 
 QEMUBIN=`which $QEMU 2 /dev/null`
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH V2 1/3] qemurunner.py: Move some class variables that should only be local

[Randy Witt]

The bootlog and qemusock variables were set in the class as part of the
create_socket() routine. However those variables are never used outside
of the same function and thus serve no purpose as class variables.

This initializes those variables near where they are used.

Signed-off-by: Randy Witt randy.e.w...@linux.intel.com
---
 meta/lib/oeqa/utils/qemurunner.py | 19 +--
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py 
b/meta/lib/oeqa/utils/qemurunner.py
index 0458447..c5bb13c 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -43,9 +43,6 @@ class QemuRunner:
 
 def create_socket(self):
 
-self.bootlog = ''
-self.qemusock = None
-
 try:
 self.server_socket = socket.socket(socket.AF_INET, 
socket.SOCK_STREAM)
 self.server_socket.setblocking(0)
@@ -145,22 +142,24 @@ class QemuRunner:
 socklist = [self.server_socket]
 reachedlogin = False
 stopread = False
+qemusock = None
+bootlog = ''
 while time.time()  endtime and not stopread:
 sread, swrite, serror = select.select(socklist, [], [], 5)
 for sock in sread:
 if sock is self.server_socket:
-self.qemusock, addr = self.server_socket.accept()
-self.qemusock.setblocking(0)
-socklist.append(self.qemusock)
+qemusock, addr = self.server_socket.accept()
+qemusock.setblocking(0)
+socklist.append(qemusock)
 socklist.remove(self.server_socket)
 logger.info(Connection from %s:%s % addr)
 else:
 data = sock.recv(1024)
 if data:
 self.log(data)
-self.bootlog += data
-if re.search(.* login:, self.bootlog):
-self.server_socket = self.qemusock
+bootlog += data
+if re.search(.* login:, bootlog):
+self.server_socket = qemusock
 stopread = True
 reachedlogin = True
 logger.info(Reached login banner)
@@ -171,7 +170,7 @@ class QemuRunner:
 
 if not reachedlogin:
 logger.info(Target didn't reached login boot in %d seconds % 
self.boottime)
-lines = \n.join(self.bootlog.splitlines()[-25:])
+lines = \n.join(bootlog.splitlines()[-25:])
 logger.info(Last 25 lines of text:\n%s % lines)
 logger.info(Check full boot log: %s % self.logfile)
 self.stop()
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH V2 2/3] qemurunner: Make create_socket() return data and use exceptions

[Randy Witt]

So that create_socket() can be called more than once to create sockets,
it now returns the socket and port rather than setting class variables
directly.

create_socket() now only uses exceptions for errors, not the return
value from the function.

Signed-off-by: Randy Witt randy.e.w...@linux.intel.com
---
 meta/lib/oeqa/utils/qemurunner.py | 29 +++--
 1 file changed, 15 insertions(+), 14 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py 
b/meta/lib/oeqa/utils/qemurunner.py
index c5bb13c..c8d6899 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -42,20 +42,18 @@ class QemuRunner:
 self.runqemutime = 60
 
 def create_socket(self):
-
 try:
-self.server_socket = socket.socket(socket.AF_INET, 
socket.SOCK_STREAM)
-self.server_socket.setblocking(0)
-self.server_socket.bind((127.0.0.1,0))
-self.server_socket.listen(2)
-self.serverport = self.server_socket.getsockname()[1]
-logger.info(Created listening socket for qemu serial console on: 
127.0.0.1:%s % self.serverport)
-return True
-except socket.error, msg:
-self.server_socket.close()
-logger.error(Failed to create listening socket: %s % msg[1])
-return False
-
+sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+sock.setblocking(0)
+sock.bind((127.0.0.1,0))
+sock.listen(2)
+port = sock.getsockname()[1]
+logger.info(Created listening socket for qemu serial console on: 
127.0.0.1:%s % port)
+return (sock, port)
+
+except socket.error:
+sock.close()
+raise
 
 def log(self, msg):
 if self.logfile:
@@ -82,7 +80,10 @@ class QemuRunner:
 else:
 os.environ[DEPLOY_DIR_IMAGE] = self.deploy_dir_image
 
-if not self.create_socket():
+try:
+self.server_socket, self.serverport = self.create_socket()
+except socket.error, msg:
+logger.error(Failed to create listening socket: %s % msg[1])
 return False
 
 # Set this flag so that Qemu doesn't do any grabs as SDL grabs interact
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH V2 3/3] qemurunner: Use two serial ports and log console with a thread

[Randy Witt]

qemu can freeze and stop responding if the socket buffer connected to a tcp
serial connection fills up. This happens of course when the reader of
the serial data doesn't actually read it.

This happened in the qemurunner code, because after checking for the
login: sentinel, data was never again read from the serial connection.

This patch solves the potential freeze by adding a thread to continuously
read the data from the console and log it. So it also will give a full log
of the console, rather than just up to the login prompt.

To simplify this patch, another serial port was also added to use for the
sole purpose of watching for the sentinel as well as being the interactive
serial port. This will also prevent the possibility of lots of debug
data on the console preventing the sentinel value from being seen due to
interleaved text.

Signed-off-by: Randy Witt randy.e.w...@linux.intel.com
---
 meta/conf/machine/qemuarm.conf|   2 +-
 meta/conf/machine/qemuarm64.conf  |   2 +-
 meta/conf/machine/qemumips.conf   |   2 +-
 meta/conf/machine/qemumips64.conf |   2 +-
 meta/conf/machine/qemuppc.conf|   2 +-
 meta/conf/machine/qemux86-64.conf |   2 +-
 meta/conf/machine/qemux86.conf|   2 +-
 meta/lib/oeqa/utils/qemurunner.py | 128 +-
 8 files changed, 133 insertions(+), 9 deletions(-)

diff --git a/meta/conf/machine/qemuarm.conf b/meta/conf/machine/qemuarm.conf
index d07084b..cdad03f 100644
--- a/meta/conf/machine/qemuarm.conf
+++ b/meta/conf/machine/qemuarm.conf
@@ -8,5 +8,5 @@ require conf/machine/include/tune-arm926ejs.inc
 
 KERNEL_IMAGETYPE = zImage
 
-SERIAL_CONSOLE = 115200 ttyAMA0
+SERIAL_CONSOLES = 115200;ttyAMA0 115200;ttyAMA1
 
diff --git a/meta/conf/machine/qemuarm64.conf b/meta/conf/machine/qemuarm64.conf
index 20bcfba..7bbdad7 100644
--- a/meta/conf/machine/qemuarm64.conf
+++ b/meta/conf/machine/qemuarm64.conf
@@ -9,4 +9,4 @@ MACHINE_FEATURES = 
 
 KERNEL_IMAGETYPE = Image
 
-SERIAL_CONSOLE = 38400 ttyAMA0
+SERIAL_CONSOLES = 38400;ttyAMA0 38400;ttyAMA1
diff --git a/meta/conf/machine/qemumips.conf b/meta/conf/machine/qemumips.conf
index d9d2421..fbf8137 100644
--- a/meta/conf/machine/qemumips.conf
+++ b/meta/conf/machine/qemumips.conf
@@ -8,6 +8,6 @@ require conf/machine/include/tune-mips32r2.inc
 KERNEL_IMAGETYPE = vmlinux
 KERNEL_ALT_IMAGETYPE = vmlinux.bin
 
-SERIAL_CONSOLE = 115200 ttyS0
+SERIAL_CONSOLES = 115200;ttyS0 115200;ttyS1
 
 MACHINE_EXTRA_RRECOMMENDS =  kernel-modules
diff --git a/meta/conf/machine/qemumips64.conf 
b/meta/conf/machine/qemumips64.conf
index b2c7998..8c3f1fe 100644
--- a/meta/conf/machine/qemumips64.conf
+++ b/meta/conf/machine/qemumips64.conf
@@ -8,6 +8,6 @@ require conf/machine/include/tune-mips64.inc
 KERNEL_IMAGETYPE = vmlinux
 KERNEL_ALT_IMAGETYPE = vmlinux.bin
 
-SERIAL_CONSOLE = 115200 ttyS0
+SERIAL_CONSOLES = 115200;ttyS0 115200;ttyS1
 
 MACHINE_EXTRA_RRECOMMENDS =  kernel-modules
diff --git a/meta/conf/machine/qemuppc.conf b/meta/conf/machine/qemuppc.conf
index 93145a1..85cbbf7 100644
--- a/meta/conf/machine/qemuppc.conf
+++ b/meta/conf/machine/qemuppc.conf
@@ -7,5 +7,5 @@ require conf/machine/include/tune-ppc7400.inc
 
 KERNEL_IMAGETYPE = vmlinux
 
-SERIAL_CONSOLE = 115200 ttyS0
+SERIAL_CONSOLES = 115200;ttyS0 115200;ttyS1
 
diff --git a/meta/conf/machine/qemux86-64.conf 
b/meta/conf/machine/qemux86-64.conf
index 837f9f4..a4fd43c 100644
--- a/meta/conf/machine/qemux86-64.conf
+++ b/meta/conf/machine/qemux86-64.conf
@@ -13,7 +13,7 @@ require conf/machine/include/tune-core2.inc
 
 KERNEL_IMAGETYPE = bzImage
 
-SERIAL_CONSOLE = 115200 ttyS0
+SERIAL_CONSOLES = 115200;ttyS0 115200;ttyS1
 
 XSERVER = xserver-xorg \
${@bb.utils.contains('DISTRO_FEATURES', 'opengl', 
'mesa-driver-swrast', '', d)} \
diff --git a/meta/conf/machine/qemux86.conf b/meta/conf/machine/qemux86.conf
index 3562276..96cea66 100644
--- a/meta/conf/machine/qemux86.conf
+++ b/meta/conf/machine/qemux86.conf
@@ -12,7 +12,7 @@ require conf/machine/include/tune-i586.inc
 
 KERNEL_IMAGETYPE = bzImage
 
-SERIAL_CONSOLE = 115200 ttyS0
+SERIAL_CONSOLES = 115200;ttyS0 115200;ttyS1
 
 XSERVER = xserver-xorg \
${@bb.utils.contains('DISTRO_FEATURES', 'opengl', 
'mesa-driver-swrast', '', d)} \
diff --git a/meta/lib/oeqa/utils/qemurunner.py 
b/meta/lib/oeqa/utils/qemurunner.py
index c8d6899..e976fd0 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -13,6 +13,7 @@ import re
 import socket
 import select
 import errno
+import threading
 
 import logging
 logger = logging.getLogger(BitBake.QemuRunner)
@@ -38,6 +39,7 @@ class QemuRunner:
 self.logfile = logfile
 self.boottime = boottime
 self.logged = False
+self.thread = None
 
 self.runqemutime = 60
 
@@ -81,6 +83,7 @@ class QemuRunner:
 os.environ[DEPLOY_DIR_IMAGE] = self.deploy_dir_image
 
 try:
+threadsock, threadport = self.create_socket()
 self.server_socket

[OE-core] [PATCH V2 0/3] Try to fix qemu freezing due to full socket buffers

[Randy Witt]

Fix copy and pasted error message that had no meaning to the actual timeout.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/3] toolchain-shar-extract.sh: explain why we cannot use sudo in extensible SDK

[Randy Witt]

On 08/19/2015 06:32 AM, brendan.le.f...@intel.com wrote:

From: Brendan Le Foll brendan.le.f...@intel.com

Signed-off-by: Brendan Le Foll brendan.le.f...@intel.com
---
  meta/files/toolchain-shar-extract.sh | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/files/toolchain-shar-extract.sh 
b/meta/files/toolchain-shar-extract.sh
index c4c2a70..d8e714a 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -135,7 +135,7 @@ mkdir -p $target_sdk_dir /dev/null 21
  # if don't have the right to access dir, gain by sudo
  if [ ! -x $target_sdk_dir -o ! -w $target_sdk_dir -o ! -r $target_sdk_dir ]; 
then
if [ $SDK_EXTENSIBLE = 1 ]; then
-   echo Unable to access \$target_sdk_dir\.
+   echo Unable to access \$target_sdk_dir\, can\'t install 
extensible SDK as root.


This is a bit unclear since it makes it sound like the user was trying to 
install as root, which would have already failed if that were the case. Perhaps 
wording it as: Unable to access $dir, will not attempt to install using sudo, 
because the extensible sdk can't be installed as root.



exit 1
fi




--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/3] qemurunner.py: Move some class variables that should only be local

[Randy Witt]

The bootlog and qemusock variables were set in the class as part of the
create_socket() routine. However those variables are never used outside
of the same function and thus serve no purpose as class variables.

This initializes those variables near where they are used.

Signed-off-by: Randy Witt randy.e.w...@linux.intel.com
---
 meta/lib/oeqa/utils/qemurunner.py | 19 +--
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py 
b/meta/lib/oeqa/utils/qemurunner.py
index 0458447..c5bb13c 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -43,9 +43,6 @@ class QemuRunner:
 
 def create_socket(self):
 
-self.bootlog = ''
-self.qemusock = None
-
 try:
 self.server_socket = socket.socket(socket.AF_INET, 
socket.SOCK_STREAM)
 self.server_socket.setblocking(0)
@@ -145,22 +142,24 @@ class QemuRunner:
 socklist = [self.server_socket]
 reachedlogin = False
 stopread = False
+qemusock = None
+bootlog = ''
 while time.time()  endtime and not stopread:
 sread, swrite, serror = select.select(socklist, [], [], 5)
 for sock in sread:
 if sock is self.server_socket:
-self.qemusock, addr = self.server_socket.accept()
-self.qemusock.setblocking(0)
-socklist.append(self.qemusock)
+qemusock, addr = self.server_socket.accept()
+qemusock.setblocking(0)
+socklist.append(qemusock)
 socklist.remove(self.server_socket)
 logger.info(Connection from %s:%s % addr)
 else:
 data = sock.recv(1024)
 if data:
 self.log(data)
-self.bootlog += data
-if re.search(.* login:, self.bootlog):
-self.server_socket = self.qemusock
+bootlog += data
+if re.search(.* login:, bootlog):
+self.server_socket = qemusock
 stopread = True
 reachedlogin = True
 logger.info(Reached login banner)
@@ -171,7 +170,7 @@ class QemuRunner:
 
 if not reachedlogin:
 logger.info(Target didn't reached login boot in %d seconds % 
self.boottime)
-lines = \n.join(self.bootlog.splitlines()[-25:])
+lines = \n.join(bootlog.splitlines()[-25:])
 logger.info(Last 25 lines of text:\n%s % lines)
 logger.info(Check full boot log: %s % self.logfile)
 self.stop()
-- 
2.4.3

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] Try to fix qemu freezing due to full socket buffers

[Randy Witt]

The following changes since commit a3d228a484cf471e999b6537987e2897d8a7bd4e:

  bitbake: main: Handle RuntimeError exception in list_extension_modules 
(2015-08-20 22:04:43 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/logthread

for you to fetch changes up to c0f8b3c28c4410031dc763e714bc2ad0f87171d6:

  qemurunner: Use two serial ports and log console with a thread (2015-08-20 
14:16:29 -0700)


Randy Witt (3):
  qemurunner.py: Move some class variables that should only be local
  qemurunner: Make create_socket() return data and use exceptions
  qemurunner: Use two serial ports and log console with a thread

 meta/conf/machine/qemuarm.conf|   2 +-
 meta/conf/machine/qemuarm64.conf  |   2 +-
 meta/conf/machine/qemumips.conf   |   2 +-
 meta/conf/machine/qemumips64.conf |   2 +-
 meta/conf/machine/qemuppc.conf|   2 +-
 meta/conf/machine/qemux86-64.conf |   2 +-
 meta/conf/machine/qemux86.conf|   2 +-
 meta/lib/oeqa/utils/qemurunner.py | 174 --
 8 files changed, 155 insertions(+), 33 deletions(-)
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core