Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

2017-11-15 Thread Patrick Ohly 
On Tue, 2017-11-14 at 11:42 -0600, Leonardo Sandoval wrote:
> On Tue, 14 Nov 2017 17:09:28 +0100
> Patrick Ohly  wrote:
> > Is oe-selftest-internal even going to be in the default PATH? It
> > probably shouldn't be, once it truly becomes an implementation
> > detail.
> 
> where can we placed it besides the scripts folder?

scripts/lib/ perhaps, then call it from oe-selftest with an absolute
path?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

2017-11-14 Thread Leonardo Sandoval 
On Tue, 14 Nov 2017 17:09:28 +0100
Patrick Ohly  wrote:

> On Tue, 2017-11-14 at 10:09 -0600, Leonardo Sandoval wrote:
> > On Tue, 14 Nov 2017 09:24:30 +0100
> > Patrick Ohly  wrote:
> >   
> > > On Mon, 2017-11-13 at 10:17 -0800,
> > > leonardo.sandoval.gonza...@linux.intel.com wrote:  
> > > > From: Leonardo Sandoval  > > > om>  
> > > > 
> > > > The main idea is to isolate the oe-selftest execution so neither
> > > > the
> > > > current build dir nor the configuration data is touch/polluted.
> > > > This
> > > > approach uses a wrapper script (which is the one presented on
> > > > this
> > > > commit) which creates a unique directory and inside it copies all
> > > > scripts and metadata, re-initializes the enviroment (re-sources
> > > > oe-
> > > > init-build-env) and finally launches the oe-selftest-internal
> > > > (which
> > > > used to be oe-selftest) command, passing command arguments to the
> > > > latter.    
> > > 
> > > This mode of operation may or may not be desirable. Can it be made
> > > optional?  
> > 
> > good idea. However, you can call the oe-selftest-internal for the
> > this purpose.  
> 
> While doable, it feels wrong to rename something as "internal" and then
> still expect people to call it directly.
> 
> A command line parameter for the new oe-selftest which controls this
> behavior and just calls oe-selftest-internal directly when requested
> feels cleaner and more discoverable.
> 
> Is oe-selftest-internal even going to be in the default PATH? It
> probably shouldn't be, once it truly becomes an implementation detail.

where can we placed it besides the scripts folder?

> 
> > > In refkit CI testing, several selftests run tests on build
> > > artifacts
> > > (primarily the images) produced during the previous build and only
> > > build them if needed, i.e. they run "bitbake some-image" and that
> > > is
> > > usually fast because the image already exists. Reusing sstate and
> > > download dir also is important for speed.  
> > 
> > oe-selftest creates a new build/conf folder, with the same conf files
> > as the ones present in your current build/conf, so this means that
> > you can set there DL_DIR and SSTATE_DIR (for example) on your main
> > local.conf and these will be used by oe-selftest, effectively reusing
> > these folders.  
> 
> Instead of leaving it to the developer to figure this out, should the
> oe-selftest have --[no-]reuse-download and --[no]-reuse-sstate options?

sounds good. By default, we will let reuse download/sstate.

Good input. I will send a v3.




> 
> > >   
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

2017-11-14 Thread Patrick Ohly 
On Tue, 2017-11-14 at 10:09 -0600, Leonardo Sandoval wrote:
> On Tue, 14 Nov 2017 09:24:30 +0100
> Patrick Ohly  wrote:
> 
> > On Mon, 2017-11-13 at 10:17 -0800,
> > leonardo.sandoval.gonza...@linux.intel.com wrote:
> > > From: Leonardo Sandoval  > > om>
> > > 
> > > The main idea is to isolate the oe-selftest execution so neither
> > > the
> > > current build dir nor the configuration data is touch/polluted.
> > > This
> > > approach uses a wrapper script (which is the one presented on
> > > this
> > > commit) which creates a unique directory and inside it copies all
> > > scripts and metadata, re-initializes the enviroment (re-sources
> > > oe-
> > > init-build-env) and finally launches the oe-selftest-internal
> > > (which
> > > used to be oe-selftest) command, passing command arguments to the
> > > latter.  
> > 
> > This mode of operation may or may not be desirable. Can it be made
> > optional?
> 
> good idea. However, you can call the oe-selftest-internal for the
> this purpose.

While doable, it feels wrong to rename something as "internal" and then
still expect people to call it directly.

A command line parameter for the new oe-selftest which controls this
behavior and just calls oe-selftest-internal directly when requested
feels cleaner and more discoverable.

Is oe-selftest-internal even going to be in the default PATH? It
probably shouldn't be, once it truly becomes an implementation detail.

> > In refkit CI testing, several selftests run tests on build
> > artifacts
> > (primarily the images) produced during the previous build and only
> > build them if needed, i.e. they run "bitbake some-image" and that
> > is
> > usually fast because the image already exists. Reusing sstate and
> > download dir also is important for speed.
> 
> oe-selftest creates a new build/conf folder, with the same conf files
> as the ones present in your current build/conf, so this means that
> you can set there DL_DIR and SSTATE_DIR (for example) on your main
> local.conf and these will be used by oe-selftest, effectively reusing
> these folders.

Instead of leaving it to the developer to figure this out, should the
oe-selftest have --[no-]reuse-download and --[no]-reuse-sstate options?

> > 
-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

2017-11-14 Thread Leonardo Sandoval 
On Tue, 14 Nov 2017 09:24:30 +0100
Patrick Ohly  wrote:

> On Mon, 2017-11-13 at 10:17 -0800,
> leonardo.sandoval.gonza...@linux.intel.com wrote:
> > From: Leonardo Sandoval 
> > 
> > The main idea is to isolate the oe-selftest execution so neither the
> > current build dir nor the configuration data is touch/polluted. This
> > approach uses a wrapper script (which is the one presented on this
> > commit) which creates a unique directory and inside it copies all
> > scripts and metadata, re-initializes the enviroment (re-sources oe-
> > init-build-env) and finally launches the oe-selftest-internal (which
> > used to be oe-selftest) command, passing command arguments to the
> > latter.  
> 
> This mode of operation may or may not be desirable. Can it be made
> optional?

good idea. However, you can call the oe-selftest-internal for the this purpose.

> 
> In refkit CI testing, several selftests run tests on build artifacts
> (primarily the images) produced during the previous build and only
> build them if needed, i.e. they run "bitbake some-image" and that is
> usually fast because the image already exists. Reusing sstate and
> download dir also is important for speed.

oe-selftest creates a new build/conf folder, with the same conf files as the 
ones present in your current build/conf, so this means that you can set there 
DL_DIR and SSTATE_DIR (for example) on your main local.conf and these will be 
used by oe-selftest, effectively reusing these folders.

> 
> Forcing all tests to run in a clean environment would make the overall
> CI run slower.

Agree. better to start with a 'hot cache' scenario, and this can be 
accomplished the way I mentioned before.



> 
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

2017-11-13 Thread leonardo . sandoval . gonzalez 
From: Leonardo Sandoval 

The main idea is to isolate the oe-selftest execution so neither the current
build dir nor the configuration data is touch/polluted. This approach uses
a wrapper script (which is the one presented on this commit) which creates
a unique directory and inside it copies all scripts and metadata, re-initializes
the enviroment (re-sources oe-init-build-env) and finally launches
the oe-selftest-internal (which used to be oe-selftest) command, passing command
arguments to the latter.

The new build directory created when re-initializing the enviroment has the
same configuration data (local.conf, auto.conf, site.conf) as the
main build directory. The latter means that one can set DL_DIR and SSTATE_DIR
into /conf/site.conf and the oe-selftest execution will use 
this.

[YOCTO #11429]

Signed-off-by: Leonardo Sandoval 
---
 scripts/oe-selftest  | 108 ---
 scripts/oe-selftest-internal |  75 ++
 2 files changed, 135 insertions(+), 48 deletions(-)
 create mode 100755 scripts/oe-selftest-internal

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index 1bf860a415..0505f943e4 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -1,5 +1,7 @@
-#!/usr/bin/env python3
+#!/bin/sh
 
+# scripts/oe-selftest: calls oe-selftest-internal in a isolated environment
+#
 # Copyright (c) 2013-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
@@ -14,62 +16,72 @@
 # You should have received a copy of the GNU General Public License along
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+# Before anything else, check oe-core environment
+if [ -z "$BUILDDIR" ]; then
+echo "Please initialize the OE-Core environment through oe-init-build-env 
script"
+exit 1
+fi
+
+# Variable definitions
+ORIGBUILDDIR="$BUILDDIR"
+OESELFTESTSCRIPTDIR="$(which oe-selftest)"
+SCRIPTSDIR="$(dirname $OESELFTESTSCRIPTDIR)"
+OECOREDIR="$(dirname $SCRIPTSDIR)"
 
-# DESCRIPTION
-# This script runs tests defined in meta/lib/oeqa/selftest/
-# It's purpose is to automate the testing of different bitbake tools.
-# To use it you just need to source your build environment setup script and
-# add the meta-selftest layer to your BBLAYERS.
-# Call the script as: "oe-selftest -a" to run all the tests in 
meta/lib/oeqa/selftest/
-# Call the script as: "oe-selftest -r .." to run just a 
single test
-# E.g: "oe-selftest -r bblayers.BitbakeLayers" will run just the BitbakeLayers 
class from meta/lib/oeqa/selftest/bblayers.py
+# oe-selftest-$$ related
+OESELFTESTDIR="$ORIGBUILDDIR/oe-selftest-$$"
+OESELFTESTBUILDDIR="$OESELFTESTDIR/build"
 
+# Return immediately in case no test execution
+ADDTESTS="$(echo "$@" | grep -i '\-a')"
+SOMETESTS="$(echo "$@" | grep -i '\-r')"
+if [ -z "$ADDTESTS" -a -z "$SOMETESTS" ]; then
+if [ -z "$@" ]; then
+   oe-selftest-internal -h
+else
+   oe-selftest-internal "$@"
+fi
+exit 0
+fi
 
+# Create directories (build, build/conf and layers)
+mkdir -p $OESELFTESTDIR $OESELFTESTBUILDDIR/conf $OESELFTESTLAYERSDIR
 
-import os
-import sys
-import argparse
-import logging
+# Populate the new build/conf, bblayers.conf will be touch latter to reflect 
new layer structure
+cp $ORIGBUILDDIR/conf/* $OESELFTESTBUILDDIR/conf
 
-scripts_path = os.path.dirname(os.path.realpath(__file__))
-lib_path = scripts_path + '/lib'
-sys.path = sys.path + [lib_path]
-import argparse_oe
-import scriptutils
-import scriptpath
-scriptpath.add_oe_lib_path()
-scriptpath.add_bitbake_lib_path()
+# Copy OE-Core meta directories
+cp -a $OECOREDIR/meta-selftest  $OESELFTESTDIR
+cp -a $OECOREDIR/meta-skeleton  $OESELFTESTDIR
+cp -a $OECOREDIR/meta-yocto-bsp $OESELFTESTDIR
 
-from oeqa.utils import load_test_components
-from oeqa.core.exception import OEQAPreRun
+# Copy non-metadata files
+cp -a $OECOREDIR/bitbake$OESELFTESTDIR
+cp -a $SCRIPTSDIR   $OESELFTESTDIR
+cp $OECOREDIR/oe-init-build-env $OESELFTESTDIR
+cp $OECOREDIR/.templateconf $OESELFTESTDIR
 
-logger = scriptutils.logger_create('oe-selftest', stream=sys.stdout)
+# Copy all layers define in BBLAYERS and construct the new bblayers.conf at 
the same time
+BBLAYERS="$(bitbake -e | grep ^BBLAYERS= | sed -e s/BBLAYERS=// -e s/\"//g)"
+for src in $BBLAYERS; do
+cp -a $src $OESELFTESTDIR
+# rename the new layer into the new bblayers.conf
+tgt="$OESELFTESTDIR/$(basename $src)"
+sed -e "s|$src|$tgt|" -i $OESELFTESTBUILDDIR/conf/bblayers.conf
+done
 
-def main():
-description = "Script that runs unit tests against bitbake and other Yocto 
related tools. The goal is to validate tools functionality and metadata 
integrity. Refer to https://wiki.yoctoproject.org/wiki/Oe-selftest for more 
information."
-parser =